Hi, thanks a lot for your help with #1011003, much appreciated. With the `-cpu host` flag I now have a working workaround. Unfortunately I just stumbled across another strange problem which I'm again able to reproduce inside QEMU with qemu-user-mode emulation. Have a look at this log file: https://jenkins.debian.net/job/fakeroot-foreign-worker/645/consoleText And search for "apt-get remove --yes qemu-user-static binfmt-support qemu-user". This used to successfully remove qemu-user support from the virtual machine. As you can see from a few lines below, running mmdebstrap --architectures=arm64 [...] succeeds. Why? Is this because I'm now running qemu with "-cpu host"? Or is this due to another change in qemu-user from recently? How do I disable user-mode if removing the package isn't doing it? Do I *additionally* have to run: echo 0 > /proc/sys/fs/binfmt_misc/qemu-aarch64 or echo 0 > /proc/sys/fs/binfmt_misc/status I'm filing this bug because I expected package removal to also disable it, but as evident from the log, it doesn't. Why? Thanks! cheers, josch
31.05.2022 10:44, Johannes Schauer Marin Rodrigues wrote: That bug still needs to be fixed though. It has nothing to do with -cpu host. It is due to the fact that I switched from binfmt-support to systemd for binfmt registration (actually the two work in parallel). And systemd does not provide an official way to trigger binfmt (de)registration besides a reboot. And in systemd, the service (systemd-binfmt.service) is described as "early boot service". Maybe I can run /lib/systemd/systemd-binfmt explicitly but this does not look like a good thing to do. And I don't see any way to remove registered formats which are not there anymore. It is actually the same for *installing* qemu-user - without binfmt-support it will not register its binfmts until reboot. /mjt
31.05.2022 10:44, Johannes Schauer Marin Rodrigues wrote: That bug still needs to be fixed though. It has nothing to do with -cpu host. It is due to the fact that I switched from binfmt-support to systemd for binfmt registration (actually the two work in parallel). And systemd does not provide an official way to trigger binfmt (de)registration besides a reboot. And in systemd, the service (systemd-binfmt.service) is described as "early boot service". Maybe I can run /lib/systemd/systemd-binfmt explicitly but this does not look like a good thing to do. And I don't see any way to remove registered formats which are not there anymore. It is actually the same for *installing* qemu-user - without binfmt-support it will not register its binfmts until reboot. /mjt
Quoting Michael Tokarev (2022-05-31 11:50:29) emulation binary got removed. This must mean that the qemu-user binaries are loaded into memory where they remain until reboot? I have another funny observation: the test is actually flaky and doesn't always produce the same results. I ran the same test 100 times and in 10% of the cases, qemu-user-mode emulation still worked despite the packages being removed. In 90% of the cases, foreign-arch mmdebstrap failed as I expected it to happen. Given your explanation above, I don't understand why I do not get the same result every time I run this test. In my tests, I now replaced removing qemu-user-static and binfmt-support with running `echo 0 > /proc/sys/fs/binfmt_misc/qemu-aarch64`. As expected, this seems to work all of the time, so I guess I'll stick with that. Thanks! cheers, josch
31.05.2022 16:45, Johannes Schauer Marin Rodrigues wrote: .. There's nothing surprising in there. This is exactly what makes your chroots working without copying qemu binaries into each chroot. This is what the "F" (fix binary) binfmt_misc flag is for. See the kernel binfmt_misc documentation, https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html . Yes, the kernel keeps the binary open since the registration of each binfmt. This is because you explicitly request/install binfmt-support, so it and systemd-binfmt.service are working in parallel. The result is non-deterministic. /mjt
Hi, Quoting Michael Tokarev (2022-05-31 17:12:22) https://tracker.debian.org/news/1332902/accepted-systemd-2512-4-source-into-unstable/ Changes: systemd (251.2-4) unstable; urgency=medium . * Use try-restart in systemd-binfmt dpkg trigger Thanks to Jochen Sprickerhof for making me aware of this. cheers, josch