Hello and thanks for maintaining autopkgtest!

I wanted to give the QEMU/KVM testbed a try. Hence I tried to create
a VM image by using autopkgtest-build-qemu. Its man page states:

[...]
|      Note that you need to call this as root.
[...]

And indeed the command

  $ autopkgtest-build-qemu unstable ~/var/cache/autopkgtest/sid.img

fails, when issued by a regular user, as it cannot even find parted
in the search PATH.

I guess this is because of vmdb2, which requires superuser privileges.
But why?
Is there any hope to improve vmdb2 or to use another tool, in order
to create a KVM testbed without requiring superuser privileges?

Control: tag -1 + confirmed help

Most disk images creation tools do it by mounting an empty image as a
loop device and then debootstrapping into it, which requires root.

There may be ways of creating an image without root, but to be honest
this is very low on my priority list.  I will take patches though.

Hello Francesco,

Francesco Poli (wintermute) [2019-11-08 23:59 +0100]:

Antonio already replied to this part.

A much better approach to this would be what autopkgtest-buildvm-ubuntu-cloud
does, and use a cloud image. Debian has these
(https://cloud.debian.org/images/cloud/), we use them for e. g. Cockpit CI as
well. There aren't any for testing, so you would need to start with stable and
upgrade. Perhaps we can talk the cloud image maintainers into creating images
for testing regularly.

Then adjusting this to an autopkgtest-buildvm-debian-cloud script, or even
generalizing/unifying the two, should not be too hard. Certainly much simpler
than trying to teach vmdb2 to run as normal user.

Martin

I think there's a place for both options. Starting from a cloud image
definitely seems better for a lowest-common-denominator, "I just want some
sort of testing" requirement set, and I would certainly prefer support
for new architectures to start from "first, make the cloud image work"
rather than always having to know how to build an autopkgtest-specific
bootable image from first principles.

One approach to this is to take a generic cloud image and modify it to
provide the serial ports that autopkgtest-virt-qemu currently needs,
like autopkgtest-buildvm-ubuntu-cloud does.

Another approach that I've been experimenting with is to make use of
the fact that cloud images already include cloud-init and a ssh server,
and use cloud-init to bring up the ssh server with a known public key
accepted. This seems less fragile than injecting commands on serial ports.

What building our own images *does* give us is total control. This is
partly bad, because we have to know all the tricks for making images
bootable on assorted architectures, but also partly good, because it lets
us test the different sorts of image that we want to support (EFI vs BIOS
boot, GPT vs MBR, systemd vs sysv-rc, merged-/usr vs unmerged-/usr).
This is particularly important when there are irreversible conversions
involved (you can go from unmerged /usr to merged /usr, but you can't
go from merged /usr to unmerged /usr).

I think the future I'd prefer to live in is that we *mostly* test on
cloud-derived images, but we also have the ability to build our own
autopkgtest-specific images for at least x86_64, and we can use those
to test the configurations that are not used on cloud images.

If that's how things work, then having autopkgtest-build-qemu be runnable
as non-root becomes a lot less important, because casual users with no
particularly special requirements will be using the cloud images anyway.

Another thing that has occurred to me when working on autopkgtest is
that it seems wasteful that we have so many wrappers around debootstrap,
which requires root (or playing with namespaces like mmdebstrap does,
or some other special action) and is not very fast. I would prefer to
be able to debootstrap a minbase tarball *once*, and then use that same
tarball as a basis for a bootable qemu image, lxc, lxd and Docker root
filesystems, and so on.

    smcv

As a stopgap solution for this, the latest version in git will try to use
fakemachine(1) if you are not uid 0. fakemachine is currently only available
on amd64 (it has to make non-portable assumptions while building the
initramfs that it will use for the VM).

I think the long-term solution is to bootstrap from an official cloud
image or by running -build-qemu as root, then build any subsequent,
more minimal images by booting a pre-existing VM image and running
vmdb2 inside it (or by running whatever tool we use instead of vmdb2,
if the implementation of -build-qemu changes). This is how I do it in
<https://salsa.debian.org/smcv/vectis>.

    smcv

Hi Francesco,

since you reported #944485 against mmdebstrap you already know that the answer
is "Yes" but I just stumbled across this bug and wanted to share one possible
solution with the readers here as well.

How to do it is described in the mmdebstrap man page under "Use as replacement
for autopkgtest-build-qemu and vmdb2" and the TLDR is:

    $ mmdebstrap --variant=important --include=linux-image-amd64 \
        --customize-hook='chroot "$1" passwd --delete root' \
        --customize-hook='chroot "$1" useradd --home-dir /home/user --create-home user' \
        --customize-hook='chroot "$1" passwd --delete user' \
        --customize-hook='echo host > "$1/etc/hostname"' \
        --customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
        --customize-hook=/usr/share/autopkgtest/setup-commands/setup-testbed \
        unstable debian-unstable.tar
    $ cat << END > extlinux.conf
    > default linux
    > timeout 0
    >
    > label linux
    > kernel /vmlinuz
    > append initrd=/initrd.img root=/dev/vda1 console=ttyS0
    END
    $ guestfish -N debian-unstable.img=disk:8G -- \
        part-disk /dev/sda mbr : \
        part-set-bootable /dev/sda 1 true : \
        mkfs ext2 /dev/sda1 : mount /dev/sda1 / : \
        tar-in debian-unstable.tar / xattrs:true : \
        extlinux / : copy-in extlinux.conf / : \
        sync : umount / : shutdown
    $ qemu-img convert -O qcow2 debian-unstable.img debian-unstable.qcow2

Unfortunately, extlinux is only available on amd64 and i386. To support more
architectures we have to boot using grub. Because this becomes a bit more
complicated I put the whole thing into a script:

https://salsa.debian.org/debian/mmdebstrap/-/blob/master/mmdebstrap-autopkgtest-build-qemu

This now works on amd64, arm64, armhf, i386 and ppc64el. But it only works for
building *native* qemu images because guestfish cannot handle foreign
architecture emulation. I see five options that allow avoiding guestfish for
building foreign architecture bootable images without being root.

Option 1 is to run guestfish inside another qemu instance. This is extremely
slow though. Since it's also easiest to set-up I won't go further into this
option here.

Option 2 is to do the partition setup and bootloader installation as an
initramfs hook script:

main script: http://paste.debian.net/1235312/
initramfs-hook: http://paste.debian.net/1235313/
initramfs-script: http://paste.debian.net/1235314/

Option 3 is to use debian-installer and use pre-seeding to make the whole
process unattended. I managed to get this working for amd64, i386, arm64 and
ppc64el. Here is how to do it for ppc64el:

    $ qemu-img create -f qcow2 disk.qcow 4G
    $ curl https://deb.debian.org/debian/dists/bullseye/main/installer-ppc64el/current/images/netboot/debian-installer/ppc64el/initrd.gz > initrd.gz
    $ curl https://deb.debian.org/debian/dists/bullseye/main/installer-ppc64el/current/images/netboot/debian-installer/ppc64el/vmlinux > linux
    $ qemu-system-ppc64le -no-reboot -machine pseries,graphics=off -serial stdio -display none -monitor none -m 1G -initrd initrd.gz -kernel linux -append 'console=hvc0 auto-install/enable=true debconf/priority=critical preseed/url=http://www.debian.org/releases/stable/example-preseed.txt netcfg/get_hostname=hostname netcfg/get_domain=domain passwd/root-password=r00tme passwd/root-password-again=r00tme passwd/user-fullname=user passwd/username=user passwd/user-password=insecure passwd/user-password-again=insecure pkgsel/run_tasksel=false popularity-contest:popularity-contest/participate=false grub-installer/bootdev=/dev/sda1' -hda disk.qcow

Option 4 is to use u-boot-qemu like this (in case of riscv64):

    $ mmdebstrap --arch=riscv64 --variant=apt \
        --include=linux-image-riscv64,u-boot-menu,systemd-sysv
        --customize-hook='printf "U_BOOT_PARAMETERS=\"rw noquiet root=/dev/vda1\"\nU_BOOT_FDT_DIR=\"noexist\"\n" > "$1"/etc/default/u-boot' \
        --customize-hook='chroot "$1" u-boot-update' \
        unstable chroot.tar \
        "deb http://deb.debian.org/debian-ports/ sid main" \
        "deb http://deb.debian.org/debian-ports/ unreleased main"
    $ genext2fs --block-size 1024 --size-in-blocks 2097152 --bytes-per-inode 16384 --tarball chroot.tar root.img
    $ dd if=root.img of=disk.img seek=1 bs=4194304
    $ dd if=/dev/zero bs=512 count=1 >> disk.img
    $ /sbin/parted -s disk.img "mklabel msdos"
    $ printf debi | dd of=disk.img seek=440 bs=1 conv=notrunc
    $ /sbin/parted -s disk.img "mkpart primary ext4 4MiB 2052MiB"

Option 5 is the same as above but will work for architectures without
u-boot-qemu support. Without a bootloader, the kernel and initrd can be passed
to qemu manually via -kernel and -initrd arguments.

While the last two options seem to be fragile, they have the advantage of
producing bit-by-bit reproducible output. The last option is also the only
option that works for all our architectures including mips*.

So now we have seven different option of creating a (native or foreign)
bootable Debian image without being root. Locally I have scripts for all of
these and I'm tempted to put any of these into a script so that it gets easier
to create images for autopkgtest-virt-qemu without superuser privileges.

On the other hand, I'm really not keen on adding yet another tool to this long
list:

https://wiki.debian.org/SystemBuildTools#General_tools

Thanks!

cheers, josch

Hi,

Quoting Johannes Schauer Marin Rodrigues (2022-05-12 22:16:16)

I've pondered more about this because I also want this tool for sbuild-qemu
(putting ckk into CC). I now have a plan and a proof-of concept implementation
for all of the following:

guestfish (option 0 from above):

 - fastest when building native qemu images with grub

initramfs (option 2 from above)

 - same output as the last option but doesn't require guestfish. Installing
   guestfish requires more than 1.7 GB of disk space (574 packages) including
   systemd, dbus, poppler, python3, libllvm, linux-image, pocketsphinx and many
   other heavy packages that should not be necessary on a CI machine, for
   example
 - can create foreign architecture grub images which guestfish cannot as
   guestfish will only always emulate the native architecture

debian-installer (option 3 from above)

 - useful because this creates a system that is as close to a system as users
   would install it as possible -- but it takes longer to create than the
   others

kernel (option 5 from above)

 - works for all architectures (the others only work on i386, amd64, armhf,
   arm64 or ppc64el due to grub only being available on those platforms)
 - is bit-by-bit reproducible
 - but doesn't install a bootloader and has the kernel and initramfs on the
   outside, so the image cannot be upgraded and has to be re-created instead

I am not considering option 1 from above because it is too slow and it feels
wrong to solve the problem just by adding another layer. I'm not considering
option 4 from above because it's mostly useful for riscv64 which is not a
release architecture. Once it is, we can easily add it on later.

I am offering to either rewrite autopkgtest-build-qemu with a --method option
defaulting either to "guestfish" if a native image is requested, "initramfs" if
a foreign architecture is requested and "kernel" if the image is neither i386,
amd64, armhf, arm64 or ppc64el. If the user requires a system as close to a
real system as possible, they can manually use --method=debian-installer.

Why can existing tools not do the job:

 - vmdb2 needs root and this will not change:
https://gitlab.com/larswirzenius/vmdb2/-/issues/62
 - debos will only run on amd64 and not on the other arches
 - mkosi needs root
 - dqib only implements the "kernel" option

Would you appreciate a MR implementing this in autopkgtest-build-qemu?

Thanks!

cheers, josch

This is extremely comprehensive, but I'm concerned that if we have this
many code paths for building a qemu VM, at least one of them will be
broken at any given time. We already have more virtualization backends
than maintainer bandwidth or routine testing (see my recent efforts to
get all our tests passing, and I haven't even got as far as qemu yet).

I appreciate your offer to add more code, but if we add all these code
paths and document them as something that should work, I'm concerned that
we'll end up in a situation where we can't remove any of them because
someone might be relying on them, but we don't actually know that they
*work* either, and nobody among the autopkgtest maintainers has the time
or knowledge to fix them when they don't. (This could be mitigated by
outsourcing as much as possible to some other system builder, like we
already do to some extent with vmdb2.)

As I believe I've already said in an earlier message to this bug report,
one way to get autopkgtest out of the business of being yet another
system-building tool is to start from a "cloud" VM image produced by
someone else (ideally, "have a qemu-runnable cloud image" would be one
of the jobs for the porting team for each architecture). One approach to
this is to take a generic cloud image, boot it with qemu and modify it
to provide the serial ports that autopkgtest-virt-qemu currently needs,
as autopkgtest-buildvm-ubuntu-cloud does.

However, the serial-port-based command execution is really quite
fragile: there are arbitrary timeouts that cause testbed startup to
fail randomly while under load (which is a large part of why I haven't
been able to get it systematically passing tests), and it's vulnerable
to architectures having different conventions for how the virtualized
"hardware" is represented in the guest user-space. I've tried to reduce
that problem by using the virtio hypervisor serial consoles /dev/hvc*, but
as you can see from the ppc64le special case in lib/autopkgtest_qemu.py,
even that is not completely predictable, because on ppc64le the virtio
serial consoles start from hvc1.

I think a less fragile approach than the serial ports would be to use
ssh, which in practice needs to work anyway, because it's how porters
typically interact with their test hardware. My goal in refactoring
part of autopkgtest-virt-qemu into lib/autopkgtest_qemu.py was to get an
autopkgtest-virt-ssh setup script that runs and manages a qemu VM, using
cloud-init to provision a trusted public key. Unfortunately, getting that
tested and working has been delayed by trying to clear enough technical
debt to make the tests pass on the backends we already have...

I think the proliferation of system-building tools that you have noted in
the past is partly because until recently they have always needed root,
partly because none of the existing ones is particularly comprehensive,
and partly because of the number of fiddly details involved in getting
the correct bootloader for each architecture and poking it into place.
If we had something analogous to grub-cloud (install on and boot from the
first disk, no attempt to support dual-booting or special configurations,
hard-code a serial console, etc.) as part of the minimal requirements
for supporting an architecture, ideally with a metapackage available on
all architectures that pulls in the correct grub-cloud-equivalent, then I
think that would help a lot - but unfortunately grub-cloud is amd64-only.

It also occurs to me that thanks to debootstrap and its clones (including
mmdebstrap), making a non-bootable Debian rootfs tree (in some transport
format like a tarball) is a solved problem; but one of the reasons people
have quality-of-implementation preferences between system-building tools
is that all of them try to implement the entire procedure, starting from
nothing and ending up with a bootable image. Might it help to re-scope
the problem to: start from a non-bootable rootfs tarball as produced by
debootstrap/mmdebstrap, and do whatever architecture-specific things
are necessary to turn it into a bootable disk image with a kernel,
a bootloader, whatever architecture-specific boot partitions are required,
and an expandable root partition populated from the tarball?

I agree this doesn't seem like a great trade-off.

I think this has potential. I see it only works for certain architectures,
but we only know how to run certain architectures in qemu and give them the
necessary serial ports *anyway*, so...

Does this work on all architectures, or on all release architectures?

I think something more analogous to the official cloud images is perhaps
a better basis for CI systems than preseeding the interactive d-i
interface so much that it becomes non-interactive, but if this has wider
architecture coverage than (for example) the initramfs approach then it
seems maybe useful.

If we had u-boot-qemu on more architectures, I'd like this one a lot more.

The major disadvantage of booting with an external kernel and initrd is
that it can't test new kernels, and can't fully test new versions of a
component that contributes to the initramfs, which might break the
expectations of a test that has the isolation-machine restriction.

As soon as you have one qemu-bootable image (perhaps from adapting a
cloud image made by someone else), you can boot into that to be root
and produce others. On amd64, you don't even need an image, because
fakemachine can reuse your running system's kernel and /usr (but that's
currently amd64-only).
it's not actually all that hard conceptually, but it does need a table
mapping Go architectures to various architecture-specific properties,
and I don't know how a Go programmer would represent that in a somewhat
future-proof API.

    smcv

Hi Simon,

if you are short on time, you might skip right to the end of this mail and
ignore my other comments as they are not of any high priority since at least
for me, this issue is solved with somehow integrating debvm or adding an
interface that allows using debvm output with autopkgtest.

Quoting Simon McVittie (2022-06-08 11:57:58)

those are very valid concerns. This makes sense.

My concern with using downloadable cloud images is that what I download from
some server is not verified by the archive GPG keys. I much rather run one of
the existing system-building tools which only downloads data verified by
debian-archive-keyring. But in another threat you already said that you'd like
the input to autopkgutils just being a tarball created by some other tool and
I'd be completely happy with that idea.

Is the argument here, that you are not happy with how autopkgtest-virt-qemu
operates in general and that's an additional reason (probably in connection
with the maintenance burden) why you don't want to add new features to it?

As far as I know, there is no bootloader that supports all our release
architectures, no?

Isn't that what many system-build-tools already do? They do take the
debootstrap output and then turn that into a bootable disk but they don't
re-invent debootstrap. In that sense, that is also what autopkgtest-build-qemu
does. It takes the output of debootstrap (well, it calls debootstrap itself but
could probably easily be modified to take an existing debootstrap tarball) and
then adds that to a system image.

No. I wasn't able to figure out some of them like mips* or s390x.

The d-i approach unfortunately is also very slow.

Yeah, me too. :)

True, but there are different trade-offs here, I think. While it would of
course be nice to have one qemu backend that is able to do everything, I only
see multiple different variations that each have their advantages and
disadvantages. While this method cannot test new kernels, it provides a wider
architecture support that was not possible with existing solutions.

Hm... it would still be nice to avoid having to double-layer this just because
some tools decided that they just require superuser privileges even though they
are not a technical necessity.

If we all had more time, so much could be done. :D

So this issue is about allowing autopkgtest-build-qemu to create a bootable
image for autopkgtest-virt-qemu without superuser privileges. I did a lot of
investigation into the problem and implemented some proof-of-concept solutions
(see my earlier mails to this bugreport) but I wasn't completely happy with any
of them because they were either slow, didn't cover all release arches, or were
not reproducible.

Now helmut has provided debvm which (as far as my needs go) covers all these
issues. It's like my option 5 but uses debugfs to extract the kernel and initrd
from the filesystem image. It thus has all the benefits of option 5 but ships
the result in a single file and even allows upgrading the kernel.

As far as I'm concerned, this issue can be closed once it somehow becomes
possible to run autopkgtest via debvm:

https://bugs.debian.org/1033291
https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/236
https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/237

Francesco, you initially reported this issue. Do you share my conclusion that
debvm solves the problem you raised about creating autopktest disk images
without superuser privileges?

Thanks!

cheers, josch

Hi Johannes,
thanks for following up on this bug report!

I really appreciate all the investigation and development work done by
you and others, but I must confess that I somewhat lost track of all
the details.    :-p

If I may share my thoughts (daydreams?!?) about this issue, I was hoping
to see a (relatively) simple command able to create a QEMU/KVM image
for autopkgtest without requiring superuser privileges. An image
that could be easily upgraded after creation (without the need to
re-create it from scratch). Bonus, if the image can then be used
also for interactive testing of packages and for package building.

Basically, I was hoping to see [sbuild-qemu-create] drop the requirement
of superuser privileges (thus dropping the behind-the-scenes use of
vmdb2 and switching to some other backend)...

[sbuild-qemu-create]: <https://anarc.at/blog/2022-04-27-sbuild-qemu/>

I am not sure I clearly understand whether this may happen or whether
this is actually going to happen soon...

Hi,

Quoting Francesco Poli (2023-06-26 20:20:47)

this is not a daydream and I think we have nearly all building blocks in place
to make all of this happen very soon! Here is a summary:

 1. You use `debvm-create --arch $foo` to create a filesystem image for any
    architecture supported by QEMU without superuser privileges. The
    debvm-create utility is a thin wrapper around mmdebstrap and passes all the
    right arguments to create this filesystem image.

 2. You use either autopkgtest-virt-qemu or autopkgtest-virt-ssh (depending on
    whether Simon prefers merge request !236 or !237) to run your autopkgtest
    through qemu with that disk image

 3. ???

 4. Profit!

If you want to upgrade the disk image you can do so by using `debvm-run` which
gives you a root shell for that disk image where you can just run "apt
upgrade".

Now lets get to your "bonus". Once either MR !236 or !237 is accepted I will
rewrite sbuild-qemu-create to not call autopkgtest-build-qemu (which uses vmdb2
which requires root) but debvm-create instead. Similarly, the sbuild-qemu
script will be adapted to call sbuild with the autopkgtest backend and the
right option (again depending on whether MR !236 or !237 get merged). And the
sbuild-qemu-update tool will automate the updating so that you don't have to
run debvm-run yourself.

Of course once either MR !236 or !237 get merged, you do not have to wait for
me to change sbuild-qemu. Even before I have done this work you can always
manually call sbuild with the autopkgtest backend and do your package builds in
the QEMU image created by debvm-create without superuser privileges.

So I don't think it's a daydream. Two merge requests exist that provide the
necessary functionality for debvm support and we hopefully only have to go
through a few more iterations with Simon to get either of them into a state
where they can get merged (or in case of !237, shipped by debvm itself).

So stay tuned! :D

cheers, josch

As somebody who uses QEMU images for a lot of things, I'm really glad to
hear this. Finally! Getting rid of root is a such a big win, in many ways.

For FWIW, as stated earlier I fully support this -- in the sense that
whatever changes are necessary or convenient, just go for it. e.g.: I
chose Python for those scripts, but if Perl is more convenient, just
drop the old stuff. Or if I can help, let me know.

I just have one question about debvm though (hence Helmut in CC), which
is beyond my depth: is that kernel direct boot something akin to EFI, or
BIOS, or of its own kind?

As I've managed to land on a use case where EFI boot seems to be needed
(PCI device passthrough), at least according to the various docs I found.

Best,
Christia

Having slept over this, I realize this isn't a big issue for sbuild-qemu
after all. It's only needed if sbuild-qemu wants to run build-time tests.

That would be a nice-to-have, but we also ship (or will ship) upstream's
extensive test suites as autopkgtests, so we have testing covered, and
that nice-to-have shouldn't be a blocker for anything.

Best,
Christian

On Mon, 26 Jun 2023 21:20:15 +0200 Johannes Schauer Marin Rodrigues wrote:

[...]
[...]
[...]
[...]


Hello!   :-)

I stayed tuned and I saw that mmdebstrap/1.4.0-1 ships a
mmdebstrap-autopkgtest-build-qemu that does not require superuser
privileges (according to its man page).

Do I understand correctly that this new script is one of the building
blocks that can make the above-summarized "daydream" come true?

Could you please give me a status update on the "daydream"?

Looking forward to hearing back from you.
Thanks for all the good stuff!

Hello again,
a gentle ping about the above-quoted daydream.

Has anyone received my message?
Could someone please reply?

Thanks a lot for your time and patience.

Hi Francesco,

Well.. thanks a lot for *your* patience! Sorry for letting you hang for
this long. 🙏🏼🙇🏽‍♂️ And thank you for the ping -- I often loose track of
projects that I'm involved in.

So... mmdebstrap-autopkgtest-build-qemu might be what you want *if* you
don't care about ppc64el or mips. The latter was (I think) never
supported my autopkgtest-qemu because there is no good option for a
bootloader. In case of ppc64el, there is ieee1275 booting but that is
not supported by mmdebstrap-autopkgtest-build-qemu.

Could you give mmdebstrap-autopkgtest-build-qemu a try and see if it
does what you want? I'm preparing another mmdebstrap release and if you
find any issues with that script I can incorporate fixes in the next
release.

Thanks!

cheers, josch