Hi,

  I recently upgraded a bullseye physical machine to bookworm.
This machine is running two virtual machines (libvirt/qemu).
I was using backports for qemu with bullseye.
I also use backports for qemu with bookworm.

  All was working without problems for several years with bullseye.
After the upgrade, one of the two VM shows instabilities :
- freeze, ie no reaction on the physical consol (but ping still answers)
- freeze (including no ping)
This occured quickly after the reboot or a few hours later.

  I decided to try to check my rootfs, so I tried to start with
debian-12.7.0-amd64-netinst.iso
I never succeded to go until rootfs mount. The VM crashed before
with, sometimes, temporary freezes during the first steps of the installer.
But, this time, with the crash, I got an error in libvirt log file:
kvm: block/block-gen.c:1719: blk_lock_medium: Assertion `!qemu_in_coroutine()' failed.
2024-10-01 11:45:34.894+0000: shutting down, reason=crashed

  I did not find anything really relevant on the net, but
https://gitlab.com/qemu-project/qemu/-/issues/2277
Not the same error, but "!qemu_in_coroutine()" was due to a qemu bug.

  So, I checked my config and saw I was using the lsilogic SCSI controler.
I do not remember why I setup this controler. I tried to switch to
the virtio-scsi SCSI controler (as I usually use).
  Changing the controler fixes my problems:
- the debian installer runs without any problem (no more temporary freeze,
  no more crashes)
- the VM is now running without problems (I will report if it freezes/crashes
  during the next hours)

  So, my current guess is that the lsilogic SCSI controler has a bug
in the recent qemu. If so, it should be fixed or the lsilogic SCSI controler
can be removed if not used anymore.

  Regards,
    Vincent

01.11.2024 16:55, Vincent Danjean:

Hi!  Thank you for the bug report!

Ok.  Please tell us the qemu command line you're using.
If possible it would be excellent to reduce it to something much smaller
than the one libvirt is constructing, since it's difficult to work with
libvirt in this context.  If anything, please try as small configuration
with libvirt as possible.

Am I right you're getting different errors from qemu while trying to
install debian-12.7.0.iso and with your VM?

Also, did you try qemu 7.2 before - I can guess you used one of its
releases wit bullseye-backports.  I'd be interesting to find out
which (at least major) versions of qemu did and did not have this
issue (or two issues).

There's 9.1 currently waiting acceptance/review for bookworm-backports,
too.

lsilogic controller supposed to work at least somewhat, - it is often
used to convert existing machines (virtual or not) to run under qemu, -
for example, it's typical for vmware to emulate similar lsilogic
controller, so the drivers in the guest are installed. You boot this
VM in qemu and install virtio drivers, after which you change lsilogic
controller to virtio.  So it might not be a production quality, but
with some minimal "workings" to perform driver installation.

And yes, this issues needs to be diagnosed and fixed.

Thanks,

/mjt

Le 01/11/2024 à 15:45, Michael Tokarev a écrit :
do lots of tests for now. But the mail relay should be moved to another
machine in a few weeks, so it will be possible to make tests.

I put the command lines used by libvirt below:
[A] the version that works for a long time
[B] the version with the Debian Installer added

I see that the machine model is quite old (pc-q35-2.8).

 > If possible it would be excellent to reduce it to something much smaller
 > than the one libvirt is constructing, since it's difficult to work with
 > libvirt in this context.  If anything, please try as small configuration
 > with libvirt as possible.

Do you have an example of a small command line for a VM with only the debian
installer (on a SCSI CD with lsilogic controler but I can adapt) that allows
me to have a console? I should be able to test that on the host.
I cannot duplicate the VM as it uses kerberos (I do not want to risk
duplicate keytab in parallel). Or the duplicated VM must be launched
without network.

With the VM, the VM fully froze, but I always saw the login prompt in the
libvirt console (opened with virt-manager). I.e., the qemu binary did not
crashed (the freeze was several hour long sometimes before I forcibely
reboot the VM).
With the debian installer, the console temporary frozed before I got
the language selection screen. And, a bit later, qemu crashed with
the provided log message.
dpkg.log.9.gz:2023-07-15 20:13:32 upgrade qemu-system-x86:amd64 1:5.2+dfsg-11+deb11u2 1:7.2+dfsg-7~bpo11+1
dpkg.log.7.gz:2024-03-23 15:35:41 upgrade qemu-system-x86:amd64 1:7.2+dfsg-7~bpo11+1 1:7.2+dfsg-7+deb12u2~bpo11+1
dpkg.log.7.gz:2024-03-23 16:29:10 upgrade qemu-system-x86:amd64 1:7.2+dfsg-7+deb12u2~bpo11+1 1:8.2.1+ds-1~bpo12+1
dpkg.log.1:2024-10-28 13:44:24 upgrade qemu-system-x86:amd64 1:8.2.1+ds-1~bpo12+1 1:9.0.2+ds-1~bpo12+1

So, looking at that, I rechecked my data and correct my initial bugreport.
The host was already in bookworm since March, 23th (were I did a bullseye
upgrade followed by a dist-upgrade to bookworm). The last week, I only
upgrade packages within bookworm (and backports for the kernel and qemu
related packages)
That also tells us that the following qemu version were working:
1:5.2+dfsg-11+deb11u2
1:7.2+dfsg-7~bpo11+1
1:7.2+dfsg-7+deb12u2~bpo11+1
1:8.2.1+ds-1~bpo12+1

But 1:9.0.2+ds-1~bpo12+1 is not working.



[A] /usr/bin/kvm \
-name guest=david,process=qemu:david,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-david/master-key.aes"}' \
-machine pc-q35-2.8,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off \
-accel kvm \
-cpu qemu64 \
-m 8192 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \
-overcommit mem-lock=off \
-smp 3,sockets=3,cores=1,threads=1 \
-uuid f05e6e71-7a70-45e3-b21b-8bef9851548e \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=38,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"ioh3420","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"ioh3420","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"ioh3420","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"i82801b11-bridge","id":"pci.4","bus":"pcie.0","addr":"0x1e"}' \
-device '{"driver":"pci-bridge","chassis_nr":5,"id":"pci.5","bus":"pci.4","addr":"0x0"}' \
-device '{"driver":"ioh3420","port":19,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"ioh3420","port":20,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"ioh3420","port":21,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"ioh3420","port":22,"chassis":9,"id":"pci.9","bus":"pcie.0","addr":"0x2.0x6"}' \
-device '{"driver":"ioh3420","port":23,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x2.0x7"}' \
-device '{"driver":"ich9-usb-ehci1","id":"usb","bus":"pcie.0","addr":"0x1d.0x7"}' \
-device '{"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pcie.0","multifunction":true,"addr":"0x1d"}' \
-device '{"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pcie.0","addr":"0x1d.0x1"}' \
-device '{"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pcie.0","addr":"0x1d.0x2"}' \
-device '{"driver":"lsi","id":"scsi0","bus":"pci.6","addr":"0x0"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.7","addr":"0x0"}' \
-blockdev '{"driver":"host_device","filename":"/dev/vg-ngs/kvm-david-disk1","aio":"native","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \
-device '{"driver":"scsi-hd","bus":"scsi0.0","scsi-id":0,"device_id":"drive-scsi0-0-0","drive":"libvirt-3-format","id":"scsi0-0-0","bootindex":1,"write-cache":"on"}' \
-device '{"driver":"scsi-cd","bus":"scsi0.0","scsi-id":1,"device_id":"drive-scsi0-0-1","id":"scsi0-0-1","bootindex":2}' \
-blockdev '{"driver":"host_device","filename":"/dev/vg-ngs/kvm-david-disk2","aio":"native","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"scsi-hd","bus":"scsi0.0","scsi-id":2,"device_id":"drive-scsi0-0-2","drive":"libvirt-1-format","id":"scsi0-0-2","write-cache":"on"}' \
-fsdev local,security_model=passthrough,id=fsdev-fs0,path=/srv/kvm,readonly \
-device '{"driver":"virtio-9p-pci","id":"fs0","fsdev":"fsdev-fs0","mount_tag":"kvm","bus":"pci.1","addr":"0x0"}' \
-netdev '{"type":"tap","fd":"39","vhost":true,"vhostfd":"41","id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:42:fa:1c","bus":"pci.3","addr":"0x0"}' \
-netdev '{"type":"tap","fd":"42","vhost":true,"vhostfd":"43","id":"hostnet1"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet1","id":"net1","mac":"52:54:00:97:f8:ac","bus":"pci.10","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=36,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev spicevmc,id=charchannel1,name=vdagent \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":2,"chardev":"charchannel1","id":"channel1","name":"com.redhat.spice.0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"spice"}' \
-spice port=5901,addr=127.0.0.1,disable-ticketing=on,seamless-migration=on \
-device '{"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"intel-hda","id":"sound0","bus":"pci.5","addr":"0x2"}' \
-device '{"driver":"hda-duplex","id":"sound0-codec0","bus":"sound0.0","cad":0,"audiodev":"audio1"}' \
-chardev spicevmc,id=charredir0,name=usbredir \
-device '{"driver":"usb-redir","chardev":"charredir0","id":"redir0","bus":"usb.0","port":"2"}' \
-chardev spicevmc,id=charredir1,name=usbredir \
-device '{"driver":"usb-redir","chardev":"charredir1","id":"redir1","bus":"usb.0","port":"3"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.8","addr":"0x0"}' \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.9","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-device '{"driver":"pvpanic"}' \
-msg timestamp=on
kvm: -fsdev local,security_model=passthrough,id=fsdev-fs0,path=/srv/kvm,readonly: warning: short-form boolean option 'readonly' deprecated
Please use readonly=on instead

[B] /usr/bin/kvm \
-name guest=david,process=qemu:david,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-4-david/master-key.aes"}' \
-machine pc-q35-2.8,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off \
-accel kvm \
-cpu qemu64 \
-m 8192 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \
-overcommit mem-lock=off \
-smp 3,sockets=3,cores=1,threads=1 \
-uuid f05e6e71-7a70-45e3-b21b-8bef9851548e \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=42,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"ioh3420","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"ioh3420","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"ioh3420","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"i82801b11-bridge","id":"pci.4","bus":"pcie.0","addr":"0x1e"}' \
-device '{"driver":"pci-bridge","chassis_nr":5,"id":"pci.5","bus":"pci.4","addr":"0x0"}' \
-device '{"driver":"ioh3420","port":19,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"ioh3420","port":20,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"ioh3420","port":21,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"ioh3420","port":22,"chassis":9,"id":"pci.9","bus":"pcie.0","addr":"0x2.0x6"}' \
-device '{"driver":"ioh3420","port":23,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x2.0x7"}' \
-device '{"driver":"ich9-usb-ehci1","id":"usb","bus":"pcie.0","addr":"0x1d.0x7"}' \
-device '{"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pcie.0","multifunction":true,"addr":"0x1d"}' \
-device '{"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pcie.0","addr":"0x1d.0x1"}' \
-device '{"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pcie.0","addr":"0x1d.0x2"}' \
-device '{"driver":"lsi","id":"scsi0","bus":"pci.6","addr":"0x0"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.7","addr":"0x0"}' \
-blockdev '{"driver":"host_device","filename":"/dev/vg-ngs/kvm-david-disk1","aio":"native","node-name":"libvirt-3-storage","cache":
{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvir
t-3-storage"}' \
-device '{"driver":"scsi-hd","bus":"scsi0.0","scsi-id":0,"device_id":"drive-scsi0-0-0","drive":"libvirt-3-format","id":"scsi0-0-0",
"bootindex":2,"write-cache":"on"}' \
-blockdev '{"driver":"file","filename":"/srv/kvm/debian-12.7.0-amd64-netinst.iso","node-name":"libvirt-2-storage","auto-read-only":
true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"scsi-cd","bus":"scsi0.0","scsi-id":1,"device_id":"drive-scsi0-0-1","drive":"libvirt-2-format","id":"scsi0-0-1","bootindex":1}' \
-blockdev '{"driver":"host_device","filename":"/dev/vg-ngs/kvm-david-disk2","aio":"native","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"scsi-hd","bus":"scsi0.0","scsi-id":2,"device_id":"drive-scsi0-0-2","drive":"libvirt-1-format","id":"scsi0-0-2","write-cache":"on"}' \
-fsdev local,security_model=passthrough,id=fsdev-fs0,path=/srv/kvm,readonly \
-device '{"driver":"virtio-9p-pci","id":"fs0","fsdev":"fsdev-fs0","mount_tag":"kvm","bus":"pci.1","addr":"0x0"}' \
-netdev '{"type":"tap","fd":"43","vhost":true,"vhostfd":"45","id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:42:fa:1c","bus":"pci.3","addr":"0x0"}' \
-netdev '{"type":"tap","fd":"46","vhost":true,"vhostfd":"47","id":"hostnet1"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet1","id":"net1","mac":"52:54:00:97:f8:ac","bus":"pci.10","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=41,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev spicevmc,id=charchannel1,name=vdagent \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":2,"chardev":"charchannel1","id":"channel1","name":"com.redhat.spice.0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"spice"}' \
-spice port=5901,addr=127.0.0.1,disable-ticketing=on,seamless-migration=on \
-device '{"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"intel-hda","id":"sound0","bus":"pci.5","addr":"0x2"}' \
-device '{"driver":"hda-duplex","id":"sound0-codec0","bus":"sound0.0","cad":0,"audiodev":"audio1"}' \
-chardev spicevmc,id=charredir0,name=usbredir \
-device '{"driver":"usb-redir","chardev":"charredir0","id":"redir0","bus":"usb.0","port":"2"}' \
-chardev spicevmc,id=charredir1,name=usbredir \
-device '{"driver":"usb-redir","chardev":"charredir1","id":"redir1","bus":"usb.0","port":"3"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.8","addr":"0x0"}' \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.9","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-device '{"driver":"pvpanic"}' \
-msg timestamp=on
kvm: -fsdev local,security_model=passthrough,id=fsdev-fs0,path=/srv/kvm,readonly: warning: short-form boolean option 'readonly' deprecated
Please use readonly=on instead
char device redirected to /dev/pts/4 (label charserial0)
kvm: block/block-gen.c:1692: blk_lock_medium: Assertion `!qemu_in_coroutine()' failed.
2024-11-01 13:18:52.254+0000: shutting down, reason=crashed

02.11.2024 01:10, Vincent Danjean:

Ok.

This one is deprecated now and will be removed soon.
This is one of the reasons I dislike libvirt - it casts things in
stone without the user realizing that.  It might be good, eg, for
a windows VM which might be registered with a given exact set of
hw, or it migth be bad (and usually it is actually bad).

This is what I used, - as simple as this:

  qemu-system-x86_64 -enable-kvm -m 1G -smp 4 -cpu host \
    -net none \
    -monitor stdio -vga std \
    -usb -device usb-tablet \
    -device lsi53c895a,id=scsi0 \
    -device scsi-hd,drive=drive0,bus=scsi0.0,channel=0,scsi-id=0,lun=0 \
    -drive file=test.qcow,format=qcow2,if=none,id=drive0 \
    -cdrom debian-12.7.0-amd64-DVD-1.iso

Everything is more-or-less self-explanatory.  This one disables the network,
and uses default vga backend which is available (usually from qemu-system-gui
package, if X or Wayland display is available, or you can use vnc).

This is exact command line I used for testing after reading your report -
I fired up debian installer (full version, so I don't need to download
additional packages during install) - it just worked for me.

Also, I retried now with pc-q35-2.8 machine type, - it also works for
me.  So apparently it must be something else specified by libvirtd.
Our job is to find out what it is, and I'll try that give your command
lines from libvirt.  Or maybe there's something in the guest, some
sequence of actions, which helps to trigger the prob even with the
default qemu settins - which actually feels more likely, given the
"not in coroutine" message.

So it might be the same or two different issues.

Aha. I'm testing this last 9.0.2 one from bpo.
..

Ok.  Retrying with this info at hand now..  but as you can see,
these command lines are.. difficult to deal with ;)

/mjt