Dear Maintainer,


Recently I found that my 32 Windows Vista client halts during boot.

Moreover in logs I found (I am not sure if all of these are apropriate
here)


[63469.996020] qemu-system-x86[28061]: segfault at 560f63c29600 ip 0000560f63c29600 sp 00007ffc2509a078 error 15 likely on CPU 4 (core 8, socket 0)
[63469.996037] Code: 00 00 c8 1a ff 78 87 7f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 81 00 00 00 00 00 00 00 d0 52 1a 65 0f 56
[63469.996503] audit: type=1400 audit(1791052506.218:510): apparmor="DENIED" operation="mknod" class="file" profile="libvirt-31860a79-3e72-4fac-bda4-b199c1cbdb98" name="/core.28061" pid=28061 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=64055 ouid=64055
[63470.261264] audit: type=1400 audit(1791052506.482:511): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="libvirt-31860a79-3e72-4fac-bda4-b199c1cbdb98" pid=28172 comm="apparmor_parser"
[63498.412150] audit: type=1400 audit(1791052534.634:512): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-31860a79-3e72-4fac-bda4-b199c1cbdb98" pid=28183 comm="apparmor_parser"
[63498.412357] audit: type=1400 audit(1791052534.634:513): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-31860a79-3e72-4fac-bda4-b199c1cbdb98//passt" pid=28183 comm="apparmor_parser"
[63498.482560] audit: type=1400 audit(1791052534.706:514): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-31860a79-3e72-4fac-bda4-b199c1cbdb98" pid=28186 comm="apparmor_parser"
[63498.495081] audit: type=1400 audit(1791052534.718:515): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-31860a79-3e72-4fac-bda4-b199c1cbdb98//passt" pid=28186 comm="apparmor_parser"
[63498.558524] audit: type=1400 audit(1791052534.782:516): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-31860a79-3e72-4fac-bda4-b199c1cbdb98" pid=28190 comm="apparmor_parser"
[63498.583022] audit: type=1400 audit(1791052534.806:517): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-31860a79-3e72-4fac-bda4-b199c1cbdb98//passt" pid=28190 comm="apparmor_parser"
[63498.653836] audit: type=1400 audit(1791052534.874:518): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-31860a79-3e72-4fac-bda4-b199c1cbdb98" pid=28194 comm="apparmor_parser"
[63498.653982] audit: type=1400 audit(1791052534.874:519): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-31860a79-3e72-4fac-bda4-b199c1cbdb98//passt" pid=28194 comm="apparmor_parser"

I am not sure if this is latest version regression or not, as I did not
launch these clients for a while.

I can confirm that Vista  64bit / windows 7 32 and 64 bit /windows xp
works. Only vista 32 bit has problem.

What is more interesting - safe mode on these clients works.
KJ

Control: merge -1 1149859

This is never a good thing, - qemu crashed.

The rest messages from apparmor are not relevant, but they shows the process didn't
dump a core file, which might be useful.  Unfortunately I don't know an easy way to
run qemu under debugger when it is started by libvirtd, nor how to let it dump core.
Without either of that there's little I can do with remote hands.

Yes it would be useful to know when it started, but much more important is a way
to reproduce it.  I doubt it is vista-32bit specific though, it must be some
particular state or sequence of events which makes this issue to appear.  It is
a bug in qemu, and your particular guest does the "right thing" to trigger it.
It just happened to be windows vista guest.

Are there multiple virtual machines which trigger this issue?
Can you tell me more about the configuration of them, which options do you
use to run qemu (can be seen in libvirtd logfile), which version of vista
it is, and when the issue appears?  Maybe that will be enough for me to
reproduce it.

If I can't, we'll have to figure out how to debug it at your side, - either
run it under gdb (probably starting it in paused mode, attaching gdb and
continuing) or produce a core file (and analyze it with gdb).  I need a
stack trace of the crash, to understand where it does a bad thing.

Thanks,

/mjt

Michael Tokarev <mjt@tls.msk.ru> writes:

I am trying to make qemu core dump. When I tried to attach gdb to
running core I was not able to get crash.

Moreover  (but I am not sure, this needs to be checked) it seems that
crash not always happens. (but always hangs)
Yes. Two windows vista Business and one Home Basic (32 bit)

Do you think that plan:
1. run machine
2. attach debugger
3. dump stack (there are multpile threads) is reasonable?

below qemu commandline (from ps ax -Hwww) (sorry for length)

/usr/bin/qemu-system-x86_64 -name guest=azot,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-5-azot/master-key.aes"} -machine pc-i440fx-10.0,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off,acpi=on -accel kvm -cpu qemu64,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff -m size=1048576k -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":1073741824} -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 31860a79-3e72-4fac-bda4-b199c1cbdb98 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=33,server=on,wait=off -object {"qom-type":"monitor-qmp","id":"monitor","chardev":"charmonitor"} -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device {"driver":"ich9-usb-ehci1","id":"usb","bus":"pci.0","addr":"0x5.0x7"} -device {"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pci.0","multifunction":true,"addr":"0x5"} -device {"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pci.0","addr":"0x5.0x1"} -device {"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pci.0","addr":"0x5.0x2"} -device {"driver":"ahci","id":"sata0","bus":"pci.0","addr":"0x6"} -device {"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x7"} -device {"driver":"ide-cd","bus":"sata0.1","id":"sata0-0-1"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/virtualbox-vista-32.qcow2","node-name":"libvirt-4-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-4-format","read-only":true,"driver":"qcow2","file":"libvirt-4-storage","backing":null} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/virtualbox-azot.qcow2","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-3-format","read-only":true,"driver":"qcow2","file":"libvirt-3-storage","backing":"libvirt-4-format"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/azot-diff.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-3-format"} -device {"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x8","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1} -netdev {"type":"tap","fd":"34","vhost":true,"vhostfd":"36","id":"hostnet0"} -device {"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:55:58:40","bus":"pci.0","addr":"0x3"} -chardev pty,id=charserial0 -device {"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0} -chardev spicevmc,id=charchannel0,name=vdagent -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"com.redhat.spice.0"} -device {"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"} -audiodev {"id":"audio1","driver":"spice"} -spice port=5900,addr=127.0.0.1,disable-ticketing=on,image-compression=off,seamless-migration=on -device {"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pci.0","addr":"0x2"} -device {"driver":"AC97","id":"sound0","audiodev":"audio1","bus":"pci.0","addr":"0x4"} -chardev spicevmc,id=charredir0,name=usbredir -device {"driver":"usb-redir","chardev":"charredir0","id":"redir0","bus":"usb.0","port":"2"} -chardev spicevmc,id=charredir1,name=usbredir -device {"driver":"usb-redir","chardev":"charredir1","id":"redir1","bus":"usb.0","port":"3"} -device {"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.0","addr":"0x9"} -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

KJ

1. I downgraded to 1:10.2.2+ds-1 and there were no problem
2. Apart from these 32bit Vistas, there were some (but not all) XP
machines which were not able to boot.
When I find some time I will try to bisect as I did in
https://gitlab.com/qemu-project/qemu/-/work_items/1966#note_2681936884
that's probably all I can do.


KJ

Fetched gitlab  source  and make bisect:

%git bisect log
git bisect start
# status: waiting for both 'good' and 'bad' commits
# bad: [615ece3c406b262996e569f29ea6eed0d81c4d8d] Merge tag 'for-upstream' of https://gitlab.com/bonzini/qemu into staging
git bisect bad 615ece3c406b262996e569f29ea6eed0d81c4d8d
# status: waiting for 'good' commit(s), 'bad' commit known
# good: [f8ed81651e61d9c2166df6121ce2af0f44f06b3e] Update version for 10.2.2 release
git bisect good f8ed81651e61d9c2166df6121ce2af0f44f06b3e
# good: [698104725efad4b29079d857dfdebbd804e34c99] Update version for v10.2.0 release
git bisect good 698104725efad4b29079d857dfdebbd804e34c99
# good: [698104725efad4b29079d857dfdebbd804e34c99] Update version for v10.2.0 release
git bisect good 698104725efad4b29079d857dfdebbd804e34c99
# good: [4bf40dc8176a26136792bdc2d2606a535656fbcb] target/arm: Only set %kvm_target when KVM is enabled
git bisect good 4bf40dc8176a26136792bdc2d2606a535656fbcb
# good: [4bf40dc8176a26136792bdc2d2606a535656fbcb] target/arm: Only set %kvm_target when KVM is enabled
git bisect good 4bf40dc8176a26136792bdc2d2606a535656fbcb
# bad: [055952c0aa91ea7a00d135b73f78fc0b13442d6c] Merge tag 'pull-hex-20260812v2-1' of https://github.com/qualcomm/qemu into staging
git bisect bad 055952c0aa91ea7a00d135b73f78fc0b13442d6c
# bad: [64d98e0797c821da99894517215539dea2ade7cd] hw/riscv/sifive_u.c: use intc_phandle in plic creation
git bisect bad 64d98e0797c821da99894517215539dea2ade7cd
# bad: [608ad9307d5b4797d2254ee3a4bad2bc8c8bcbae] Merge tag 'pbouvier/pr/tests-20260615' of https://gitlab.com/p-b-o/qemu into staging
git bisect bad 608ad9307d5b4797d2254ee3a4bad2bc8c8bcbae
# good: [f89397fcb99bd921d83ccdf8ec50d8856396993c] MAINTAINERS: Fix F: in "WebAssembly"
git bisect good f89397fcb99bd921d83ccdf8ec50d8856396993c
# good: [51395f054a6e3a9eaa0ac52b8e18362d79c333ef] net/colo-compare: Use QEMU_LOCK_GUARD to simplify mutex handling
git bisect good 51395f054a6e3a9eaa0ac52b8e18362d79c333ef
# bad: [08c2415217b53ee66a8b7f705a21beeef19ffc6d] vhost-user.rst: Explicitly allow front-end to write to kick FDs
git bisect bad 08c2415217b53ee66a8b7f705a21beeef19ffc6d
# bad: [0641dfdac00ba76ffbe41c59291e5ad23c62c219] vhost-user-gpu: drop code duplication
git bisect bad 0641dfdac00ba76ffbe41c59291e5ad23c62c219
# good: [c579197d6be77be90607d5d59e8e7f68696dfa26] intel_iommu: Refactor PASID processing to use IOMMU_NO_PASID internally
git bisect good c579197d6be77be90607d5d59e8e7f68696dfa26
# good: [a50499350010ec0d238185f3c835f3fb39e58032] hw/acpi/aml-build.c: add aml_irq() representing the 3-byte IRQ descriptor
git bisect good a50499350010ec0d238185f3c835f3fb39e58032
# bad: [44c8f42d5824ab652ddeedb79851fa72d2af7be1] vhost-user: rework enabling vrings
git bisect bad 44c8f42d5824ab652ddeedb79851fa72d2af7be1
# bad: [d616781f2d63fa602b12020cedbb274d15797e41] hw/char/serial-isa.c: declare IRQ as shared in ACPI IRQ descriptor
git bisect bad d616781f2d63fa602b12020cedbb274d15797e41
# good: [6d40788bfd8f8b7b39d489bda1a61fa0a2ae9951] tests/acpi: allow DSDT acpi table changes
git bisect good 6d40788bfd8f8b7b39d489bda1a61fa0a2ae9951
# first 'bad' commit: [d616781f2d63fa602b12020cedbb274d15797e41] hw/char/serial-isa.c: declare IRQ as shared in ACPI IRQ descriptor

for the reference:

after fetching I make
./configure
and then
make
sudo make install repeatedly
During compiling I have to make some minor changes due to compiler warnings
like:

+#ifndef FUTEX_CMD_MASK
 #define FUTEX_CMD_MASK          ~(FUTEX_PRIVATE_FLAG | FUTEX_CLOCK_REALTIME)
-
+#endif

But I think these does not affect my tests.

As it is "serial" in name file I tried to remove "serial" device from
machines:
<serial type="pty">
  <source path="/dev/pts/7"/>
  <target type="isa-serial" port="0">
    <model name="isa-serial"/>
  </target>
  <alias name="serial0"/>
</serial>


And indee this makes a differences -
without serial Vistas started while with serial device hangs.
Also XP machines which hanged previously - after removing serial device
they started to work.

BUT there are XP machines which has serial device and still works - this
is kind of mystery for me.

KJ

This is what I'd do myself if I had a reproducer...

If you will ever need to do that again, there's much easier way:

   ./configure --target-list=x86_64-softmmu --disable-werror
     (this will request only one qemu binary and omit irrelevant parts)
   ninja qemu-system-x86_64
     (this will build just this binary and nothing else)

This is much, much faster than having to build everything each time.

Now, once you have this binary, you can point libvirt to it, by using
custom qemu binary in its settings, pointing to just built
/qemu/build/dir/qemu-system-x86_64 (whatever the full path is).

I don't recommend to install it just for this kind of testing.

Also, you can add --enable-debug to the ./configure line, this will
help with gdb and friends.

Yes, this is okay and I think with --disable-werror it wont even be
needed (but I'm not sure).

So this is just one serial port, not 3 or 4.  Interesting.  I think this
is this part in your command line:

  -chardev pty,id=charserial0
  -device
{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}

Ok, I don't have vista image here, lemme try to find one and see if
I can reproduce it myself.

Thank you for doing all this!  You took much harder route but you ended
in the right place.

/mjt