I have a hypervisor (sol) with 4 guests (mercury, venus, mars, and
jupiter).  The hypervisor and first three guests run Debian.  The 4th
guest runs Home Assistant.  The Debian guests have been running in
roughly their current configuration since 2016 (wheezy) and have been
upgraded since then for each Debian release.

I have been working to upgrade all of the Debian guests to trixie. I
followed the same process as usual: I upgraded each guest one at a time,
rebooted, and tested.  Once all guests were working, I upgraded the
hypervisor.  After rebooting the hypervisor, none of the guests were
running, and neither was the virtual network.  I started the network
manually, but I haven't managed to get any of the guests working.

The initial problem with the guests was that their machine type
pc-i440fx-2.1 was no longer supported.  I adjusted this to
pc-i440fx-2.4, and virsh accepted that change. Now, I can start the
guests, and they report as running, but they never do anything useful.

As far as I can tell, nothing is ever even written to the console.  The
guests definitely never come up far enough to respond to a ping or
accept an SSH connection.  If I try to cleanly shut down a guest, it
never stops.  I have to use destroy instead of shutdown.  When I attempt
to start a guest, I get a new kvm process that immediately uses 100% of
a CPU and stays like that for at least 30 minutes (which is as long as
I've been willing to wait).

Oddly, the jupiter guest (Home Assistant) boots without problems and
seems to be working fine.  So, I assume there must be something
wrong/different about the Debian guests that conflicts with the new
libvirt version in trixie, but can't figure out what that might be.

I compared the current state of each guest against my pre-upgrade backup
of /etc/libvirt, and none of the changes are surprising.  Besides
the change to the machine name, there are some minor changes to some XML
stanzas which libvirt seems to make on its own. (I removed them and they
came back.)  All of the other changes within /etc/livirt came with as
part of the upgrade to trixie.

The main difference I see is that jupiter uses <os firmware='efi'>, and
its only disk is a .qcow2 file.  The other guests do not use EFI and
their disks are raw block devices instead.

I hesitate to attach configuration here, because I'm not sure what will
be useful to you.  Let me me know what questions you have, and I can
attach configuration or run other diagnostics.

Thanks,

Ken

This is a very sensible way to go about upgrading an hypervisor host,
well done :)

The 2.1 machine types have been dropped from QEMU, so switching to
newer ones is the right thing to do. That said, I would probably pick
the most recent machine type (10.0) instead of the oldest one (2.4),
as that would give you the most runway. The 2.4 machine types are
almost certainly going to be absent from the QEMU version that will
end up in forky, and adopting them today pretty much guarantees that
you'll be forced to go through this process again in two years' time.

They're not using EFI "indirectly" through a legacy configuration
with a <loader> element either, right? And I assume you've confirmed
that the raw block devices are still present at the expected paths.

If libvirt is able to start the guests at all, that rules out most of
the common failure modes (e.g. failing to access the disk files or
rejecting an invalid configuration). It seems more likely than the
issue would be on the firmware/guest OS side, though of course it
could also be that libvirt is not setting things up in a way that
allows the firmware/guest OS to do anything useful.

You could start by collecting some debug logs[1] and checking that
there is nothing suspicious in the journal, such as AppArmor denials.

Another useful attempt would be to eschew the existing configuration
for one of the guest and import it from scratch using virt-install.
Assuming the needs are not too specific, that should result in a
working configuration, which we could then compare with the previous
one to try and identify the issue.


[1] https://libvirt.org/kbase/debuglogs.html

I enabled debug logs. I don't see any AppArmor errors. However, I ran
across this in the logs when starting the mars guest:

   error : virStorageFileBackendFileRead:106 : Failed to open file
   '/dev/vg_ssd/lv_mars_root': Permission denied error :
   virStorageFileBackendFileRead:106 : Failed to open file
   '/dev/vg_spin/lv_mars_data': Permission denied error :
   virStorageFileBackendFileRead:106 : Failed to open file
   '/dev/vg_ssd/lv_mars_fast': Permission denied

To work around that, I edited /etc/libvirt/qemu.conf, adjusted the
security driver, and restarted the service:

   #security_driver = "selinux"
   security_driver = "none"

That resolved the permission denied error, but made no difference to the
behavior of the guest once it started.  I still see nothing in the
console, and it never becomes available for use.

I also followed your advice and tried importing the existing filesystem
into a new guest. I had saved off the original command used to create
this guest back in 2016, so I adjusted it to use --import and --osinfo
rather than one of install options.

   virt-install \
          --import \
          --osinfo debiantrixie \
          --check path_in_use=off \
          --check mac_in_use=off \
          --connect qemu:///system \
          --virt-type kvm \
          --name mars.2 \
          --memory 22528 \
          --vcpus 4 \
          --graphics none \
          --machine "pc-i440fx-10.0" \
          --disk /dev/vg_ssd/lv_mars_root \
          --disk /dev/vg_spin/lv_mars_data \
          --disk /dev/vg_ssd/lv_mars_fast \
          --network network=virtual,mac=52:54:00:00:00:03

This behaves the same way as the original mars guest.  It starts, but
nothing is ever written to the console and it never becomes available.
There are slight differences in the generated XML.  There are different
PCI ids assigned to the disks, and virt-install chooses host-passthrough
for the CPU instead of a specific CPU type as in the old guest.
However, there's nothing else that looks particularly significant.

Besides the original permission denied errors that I mentioned above,
there aren't any messages that contain 'error', 'exception', 'fail',
'failure', etc. in the libvirt debug logs, regardless of whether I'm
booting the original mars guest or the imported mars.2 guest.

Just as confirmation, I grabbed the contents of kern.log and syslog from
the filesystem using virt-cat:

   virt-cat -d mars /var/log/kern.log > kern.log

There has been nothing written to those logs since the last time the
guest was shut down (when rebooting the hypervisor after upgrading it),
which I think tends to support my theory that the guest isn't really
booting up, even though virsh reports it as started.

I've also experimented with:

- using only the root --disk
- using --cpu host-passthrough,secure=off
- using --boot hd,uefi=off
- using --boot bootmenu.enable=on,bios.useserial=on

None of these made any difference in the final behavior of the VM.

However, the bootmenu change did yield some interesting results.  With
this option in use, the BIOS bootmenu flashes onto the screen about
every 10 seconds.  If I manage to hit ESC before it goes away, I can
capture this:

   SeaBIOS (version 1.16.3-debian-1.16.3-2)
   Machine UUID 4763922d-20fa-4efa-8261-621281bee979

   iPXE (https://ipxe.org) 00:02.0 C000 PCI2.10 PnP
   PMM+BEFC6C80+BEF06C80 C000

   Press ESC for boot menu.

   Select boot device:

   1. Virtio disk PCI:00:05.0
   2. Legacy option rom
   3. iPXE (PCI 00:02.0)

For option 1, "PCI:00:05.0" matches up with the slot assigned to the
root disk in its <address>, so at least that makes sense.

If I pick option 1, I get the following, which disappears in less than a
second:

   Booting from Hard Disk...
   GRUB loading..
   Welcome to GRUB!

After this is printed, the screen blanks, and I'm eventually sent back
to the BIOS boot menu again.  I suspect this tight loop explains the
100% CPU utilization I see in top, since utilization drops if I hit ESC
and sit at the BIOS menu without making a choice.

I haven't had any success in coaxing GRUB to tell me what's going on. I
manually changed the timeout in /boot/grub/grub.cfg, and that seems to
change the behavior of the BIOS menu loop. This makes me think that GRUB
renders the menu and waits for the timeout before selecting the default,
and then something crashes after that... but for some reason the menu
and whatever else happens after the crash aren't available to the
console.

Do you have any other suggestions about things that I should explore, or
things that might cause the old kernel or init to no longer work with
the new libvirt?

I haven't attached the libvirt debug logs, since they're fairly large.
Let me know if you want them.

Thanks,

Ken

I have a fix.  If I adjust /etc/default/grub to set one of these values
and then run update-grub, then the guest works:

   GRUB_TERMINAL="console"
   GRUB_TERMINAL="serial"
   GRUB_TERMINAL="console serial"

Initially, since I couldn't log into the guest, I prototyped this on the
hypervisor and then made changes manually in /boot/grub/grub.cfg for
each of the guests using virt-edit. I started with a bunch of other
changes (for instance to enable the console TTYs when booting the
kernel, etc.), but in the end this was the minimal necessary fix.

The GRUB_TERMINAL change has two effects in /boot/grub/grub.cfg:
1) it removes the conditional logic that sets up for "terminal_output
gfxterm", and 2) it removes conditional logic in the Debian theme
section, leaving just two menu color lines.

From manual testing, I'm pretty confident that the first change for
gfxterm is the important one, but I still don't understand why it's
necessary.  It's not clear whether the change in behavior is due to
libvirt, grub, or the kernel package.  But at least it works, which is
quite a relief. :)

There's an inline patch below showing the complete set of differences
when using GRUB_TERMINAL="console". Changes are similar when using the
other working values.

Ken

--- grub.orig.cfg 2025-08-17 21:36:02.347698273 +0000                                                                                                 +++ grub.works.cfg   2025-08-17 21:36:15.979841505 +0000                                                                                              @@ -55,30 +55,8 @@                                                                                                                                       fi                                                                                                                                                  }                                                                                                                                                                                                                                                                                                          -if [ x$feature_default_font_path = xy ] ; then                                                                                                       -   font=unicode                                                                                                                                      -else                                                                                                                                                 -insmod part_msdos                                                                                                                                    -insmod lvm                                                                                                                                           -insmod ext2                                                                                                                                          -set root='lvmid/fA9B2E-GJRf-H7XG-7HFA-1Wqq-3ZwA-kglN18/3KwGYi-I3t1-2jvP-ZsLj-fiDr-e21m-WMQ6dU'                                                       -if [ x$feature_platform_search_hint = xy ]; then                                                                                                     -  search --no-floppy --fs-uuid --set=root --hint='lvmid/fA9B2E-GJRf-H7XG-7HFA-1Wqq-3ZwA-kglN18/3KwGYi-I3t1-2jvP-ZsLj-fiDr-e21m-WMQ6dU'  537ffb3e-02da-4ed6-aa3b-8c84319f72a0
-else
-  search --no-floppy --fs-uuid --set=root 537ffb3e-02da-4ed6-aa3b-8c84319f72a0
-fi
-    font="/usr/share/grub/unicode.pf2"
-fi
-
-if loadfont $font ; then
-  set gfxmode=auto
-  load_video
-  insmod gfxterm
-  set locale_dir=$prefix/locale
-  set lang=en_US
-  insmod gettext
-fi
-terminal_output gfxterm
+terminal_input console
+terminal_output console
 if [ "${recordfail}" = 1 ] ; then
   set timeout=30
 else
@@ -94,23 +72,8 @@
 ### END /etc/grub.d/00_header ###

 ### BEGIN /etc/grub.d/05_debian_theme ###
-insmod part_msdos
-insmod lvm
-insmod ext2
-set root='lvmid/fA9B2E-GJRf-H7XG-7HFA-1Wqq-3ZwA-kglN18/3KwGYi-I3t1-2jvP-ZsLj-fiDr-e21m-WMQ6dU'
-if [ x$feature_platform_search_hint = xy ]; then
-  search --no-floppy --fs-uuid --set=root --hint='lvmid/fA9B2E-GJRf-H7XG-7HFA-1Wqq-3ZwA-kglN18/3KwGYi-I3t1-2jvP-ZsLj-fiDr-e21m-WMQ6dU'  537ffb3e-02da-4ed6-aa3b-8c84319f72a0
-else
-  search --no-floppy --fs-uuid --set=root 537ffb3e-02da-4ed6-aa3b-8c84319f72a0
-fi
-insmod png
-if background_image /usr/share/desktop-base/ceratopsian-theme/grub/grub-4x3.png; then
-  set color_normal=white/black
-  set color_highlight=black/white
-else
-  set menu_color_normal=cyan/blue
-  set menu_color_highlight=white/blue
-fi
+set menu_color_normal=cyan/blue
+set menu_color_highlight=white/blue
 ### END /etc/grub.d/05_debian_theme ###

 ### BEGIN /etc/grub.d/10_linux ###