#1050030 Potential regression: ovmf package update from Debian bookworm breaks Xen HVM boot

#1050030#5
Date:
2023-08-18 14:26:27 UTC
From:
To:
Dear Maintainer,

After upgrading from Debian Bullseye to Debian Bookworm, existing HVM
DomUs using "firmware = 'ovmf'" (specifically Windows Server 2022 and
Windows 10) can't boot anymore. Booting these systems leads to the
Windows error 0xc0000225. I wasn't able to fix this error. Booting an
installation .iso leads to the same error. Booting the installation
media with "firmware = 'bios'" leads to a normal boot.

This seems to be the effect of a change in ovmf sources, where a xen
specific platform was created in 2019:
https://lore.kernel.org/all/20190813113119.14804-1-anthony.perard@citrix.com/#t

With some help I found a workaround. I could successfully start the
Windows DomU with ovmf firmware after removing the current ovmf package
and installing a previous ovmf package from Debian repositories,
specifically version 2020.11-2+deb11u1. This strongly indicates that the
cause of this issue lies in the ovmf package.

My HVM config file:

type = "hvm"
memory = 6144
vcpus = 2
name = "kalliope"
firmware = 'ovmf'
firmware = '/usr/local/lib/ovmf/OVMF.fd'
vif = ['bridge=xenbr0,mac=XX:XX:XX:XX:XX:XX,ip=10.0.0.4']
disk = ['phy:/dev/vg0/matrix,hda,w']
device_model_version = 'qemu-xen'
hdtype = 'ahci'
pae = 1
acpi = 1
apic = 1
vga = 'stdvga'
videoram = 16
xen_platform_pci = 1
vendor_device = 'xenserver'
viridian = 1
on_crash = 'restart'
device_model_args_hvm = [
   # Debug OVMF
   '-chardev', 'file,id=debugcon,path=/var/log/xen/ovmf.log,',
   '-device', 'isa-debugcon,iobase=0x402,chardev=debugcon',
]
sdl = 0
serial = 'pty'
vnc = 1
vnclisten = "0.0.0.0"
vncpasswd = ""

As the Windows systems are not usable anymore, Xen is significantly
reduced in functionality after the upgrade.

There is a Debian bug report which could be related, I also described my
situation there: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978595

This (or at least a very similar) issue seems to be fixed in
Redhat/Fedora. A related bug report exists in
https://bugzilla.redhat.com/show_bug.cgi?id=2170930


Additional information:

I also tried building Ovmf following
https://lore.kernel.org/all/20190813113119.14804-1-anthony.perard@citrix.com/#t
, but I wasn't fully able to create a working system:

(1) Using the resulting OVMF.fd from the build process with "firmware
='/path/to/new/OVMF.fd' led consistently to a black screen in VNC or
Spice with the text "Guest has not initialized the display (yet)".

(2) Replacing the OVMF.fd in /var/lib/ovmf with the freshly built
OVMF.fd led to a slight improvement. I could then boot the Windows
Server installation .iso, but booting the Windows 10 installation .iso
lead to a crash where the TianoCore logo was visible, but nothing
happened. The two existing DomUs were still no>

However, I am not sure that I followed the procedure correctly, I might
very well have done something very wrong. Any pointers are welcome.

Thanks,

Paul

#1050030#12
Date:
2023-08-18 21:05:31 UTC
From:
To:
affects 1050030 src:xen
quit

I'm seeing a similar situation, though instead using FreeBSD/x86 in the
VM.

For FreeBSD the bootloader appears to operate normally, but something
fails quickly after loading the kernel:

Loading kernel...
/boot/kernel/kernel text=0x18aa98 text=0xdfd150 text=0x675154 data=0x140 data=0x1c38e8+0x43b718 0x8+0x18fe70+0x8+0x1ae449/
Loading configured modules...
/boot/entropy size=0x1000
/etc/hostid size=0x25
staging 0xe3e00000 (not copying) tramp 0xe351b000 PT4 0xe3512000
Start @ 0xffffffff8038b000 ...
EFI framebuffer information:
addr, size     0xf0000000, 0x1d5000
dimensions     800 x 600
stride         800
masks          0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000

I believe all these messages are from FreeBSD's bootloader.  The first
message from the kernel should be "---<<BOOT>>---", yet that message
never shows.  Xen shows the domain spinning on a single processor which
makes me believe the FreeBSD kernel has loaded, panic()ed and the
debugger is loaded (but there is no VGA console).


From reading the available information I suspect Tianocore/EDK2 may have
tried to move some functionality to a distinct build and neither setup
quite works.  Notably there is now a "OvmfPkg/OvmfXen.dsc" build
configuration.  The OVMF.fd for Qemu for Xen functionality may have been
moved /here/.  There might also be an attempt at functionality similar to
"ArmVirtPkg/ArmVirtXen.dsc" (Debian 978595) for x86.

#1050030#17
Date:
2023-08-24 00:37:46 UTC
From:
To:
Now confirmed reverting to 2020.11-2+deb11u1 takes care of the issues I'm
running into.  I've been able to build OvmfPkg/OvmfXen.dsc, but haven't
gotten it to do anything.  I'm suspecting the support for running
headless didn't get into OvmfXen.  I'm interacting with someone
knowledgeable, but nothing yet.

I suspect the "ovmf" package needs to be split.  I've gotten the
impression the build needed for normal `qemu` isn't going to be the same
as the build needed for xen-qemu.

I think what is really needed is for xen-utils-X.YY to Recommend a
virtual package "xen-domu-bootloader" which is then provided by tools
which can load VMs.  The current other in-service tool is grub-xen-host,
but it appears OvmfXen may also be able to provide the service.

I'm attaching two patches which should help organize the source package.
These leave all the "./edksetup.sh" lines identical.  Perhaps make use
of this to make the build cleaner?

#1050030#22
Date:
2024-07-05 13:03:30 UTC
From:
To:
Hello all,

As noted before, going back to 2020.11-2+deb11u1 allows booting HVM domains. However, for us it slowed down the boot from half a minute to two minutes, when using PCI passthrough.
The Xen community pointed out to me that this may be due to a patch for Xen not being built in that version (https://github.com/tianocore/edk2/commit/c05de360ec614f71716a201760b91ee055a5ff28).

I propose to either

- ship a Xen specific version of OVMF (as suggested here https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978595 and above and similar to Fedora https://packages.fedoraproject.org/pkgs/edk2/edk2-ovmf-xen/) or
- build Xen with OVMF included (the Xen build should supports this).

However, I myself could get neither package to compile this way from Debian sources.

We mitigated the problem for us, by building OVMF ourselves from  git://xenbits.xen.org/ovmf.git for "edk2-stable202305", the latest tag found there. It should actually be the same for https://github.com/tianocore/edk2.git, but I did not try.

On Debian 12 to make it work we

- removed "brotli" from BaseTools/Source/C/Makefile (otherwise the build will fail with warnings treated as errors)
- apt-get build-dep -y ovmf
# Build needs GCC 11 instead of version 12 shipped with Debian 12, because otherwise again the build will end with warnings treated as errors
- apt-get install -y gcc-11 g++-11 libgcc-11-dev
- update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 1
- update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-11 1
- cd BaseTools; make -C Source/C
- cd ..; OvmfPkg/build.sh -a X64 -b RELEASE -p OvmfPkg/OvmfXen.dsc

Then "Build/OvmfXen/RELEASE_GCC5/FV/OVMF.fd" must be placed at "/usr/share/ovmf/OVMF.fd", because the Debian package for Xen is compiled with "--with-system-ovmf=/usr/share/ovmf/OVMF.fd".

Best regards,
Tobias

#1050030#27
Date:
2024-12-02 21:08:29 UTC
From:
To:
Hello,

Having just upgraded a Xen host from bullseye to bookworm, I got to
experience this, too.  My UEFI HVM domU guests did not boot to OS, and
issue presentation was either attempting to boot from network with iPXE
[despite not having network boot enabled in domU config...], or with
network interfaces commented out, booting to the EDK2 UEFI shell.

This involved debugging the QEMU process startup to try to find what
was not honouring the domU boot configuration, finding what packages
were modified and split between bullseye and bookworm, identifying that
OVMF is a package dependency, and reverting it to package version
2020.11-2+deb11u2 from
http://snapshot.debian.org/package/edk2/2020.11-2%2Bdeb11u2/#ovmf_2020.11-2:2b:deb11u2

After reverting the package and rebooting the affected domU instances,
the guests proceeded to boot as expected.

I then came over to file a bug for this behaviour, and have found that
the bug already exists, from over a year ago...if I only know what the
problem was from the beginning...

David Comeau (SaturnNiGHTS)

#1050030#32
Date:
2024-12-18 19:00:50 UTC
From:
To:
Hi all,

I also encountered this problem.
This is my first time trying out Xen, could only get HVM to boot with Seabios, not with OVMF.

On Debian 12 Bookworm
Non-functional package version: 2022.11-6+deb12u1

As already mentioned here, reverting to package version: 2020.11-2+deb11u2 solved the problem, the HVMs boots as expected.

I also had success building upstream OVMF from source, branch 'stable/202408' , like so: OvmfPkg/build.sh -a X64 -b RELEASE -p OvmfPkg/OvmfXen.dsc
and placing it in /usr/share/ovmf/ on the Xen-host, the HVMs boots as expected.

We probably need something like `/usr/share/ovmf/XEN_OVMF.fd` built from OvmfXen, and also need to set the configure options for the xen-hypervisor package to `--with-system-ovmf=/usr/share/ovmf/XEN_OVMF.fd`.

On the other hand, /usr/share/doc/ovmf/README.Debian mentions that OVMF.fd(the unified image) is for backwards compatibility, the best way is to have a shared OVMF_CODE.fd and per-vm OVMF_VARS.fd, however I don't think you can use the vars with xl.

#1050030#37
Date:
2025-05-09 01:35:15 UTC
From:
To:
I can confirm that the 2020 package from Debian 11 works with Xen, and
that the 2022 package from Debian 12 is broken in interesting ways (I
could start a <4G VM, but >= 4G vms instantly crashed).

I tried to update the packaging to edk2 202502, but I ran into a bug
in OVMF: https://github.com/tianocore/edk2/issues/11046

I have prepared an update to the src:edk2 package based off of the
last working 202408 version of edk and posted the change to enable and
build ovmf-xen here: https://github.com/byteit101/edk2-debian. Note
that c2c08abb36224f5a6700f12e6a16c07ad33a3fa9 is the real change. I
also added on top c7b34aae3d3d873af6d57ff1d952021331c3450e which is
just a "build faster" commit for testing, and shouldn't be pulled by a
Debian developer.

With that tree built and installed, I was able to get all my HVM
machines working by specifying:

firmware='ovmf'
system_firmware="/usr/share/ovmf/XEN_OVMF.fd"

Presumably once a DD pulls this change, the Xen package will need the
ovmf build flags updated to point to the new path.

#1050030#42
Date:
2025-06-07 16:21:22 UTC
From:
To:
I have pushed a new branch based on 202502 at my repo
debian/xen-last here: https://github.com/byteit101/edk2-debian. Note
that c2c08abb36224f5a6700f12e6a16c07ad33a3fa9 is the real change. I
also added on top c7b34aae3d3d873af6d57ff1d952021331c3450e which is
just a "build faster" commit for testing, and shouldn't be pulled by a
Debian developer.

With that tree built and installed, I was able to get all my HVM
machines working by specifying:

bios='ovmf'
bios_path_override="/usr/share/ovmf/XEN_OVMF.fd"

#1050030#47
Date:
2025-11-06 00:32:20 UTC
From:
To:
found 1050030 2025.02-8
retitle 1050030 ovmf package update from Debian bookworm breaks Xen HVM boot
affects 1120146 src:xen
quit

Seems #1050030 is still there.  This really doesn't seem a "Potential
regression", but a genuine regression at this point.  Then just to make
things better it seems Qemu is now joining in the fun of breaking HVM.

#1050030#56
Date:
2025-11-06 07:43:23 UTC
From:
To:
FWIW, I didn't even know until today that ovmf can be used
to boot xen domains, - I thought it only works with regular
bios packages, not UEFI.  To me, using UEFI has always been
a no-go due to them being just too slow.

Sure it's not something that helps users facing this bug,
just a tiny bit of personal experience.

Thanks,

/mjt

#1050030#61
Date:
2025-11-19 12:08:12 UTC
From:
To:
On Thu, 6 Nov 2025 10:43:23 +0300 Michael Tokarev <mjt@tls.msk.ru> wrote:
 > FWIW, I didn't even know until today that ovmf can be used
 > to boot xen domains, - I thought it only works with regular
 > bios packages, not UEFI. To me, using UEFI has always been
 > a no-go due to them being just too slow.

I just ran into this bug because of the need for a Win11-DomU. Turns
out, if you want to create a WIn11-DomU, you need UEFI boot.

But even if you have UEFI boot, you also need a vTPM module for Win11,
which vanilla Xen doesn't provide (yet?). AFAIK, XenServer and XCP-ng
provide a vTPM module, and then there are customized solutions.

Paul

#1050030#66
Date:
2025-11-19 12:25:20 UTC
From:
To:
FWIW, you can run win11 without TPM or UEFI just fine (and it will
boot significantly faster).  Use IoT version of win11 for that.

Speaking of this missing dependency on ipxe roms, it will be added back
in the next trixie point release (13.3), a few months from now.

Until then, please install ipxe-qemu package manually - it's a rather
trivial work-around (though it isn't easy to find, unfortunately, due
to very difficult to find logs).

Thanks,

/mjt

#1050030#71
Date:
2026-05-30 19:55:04 UTC
From:
To:
On Fri, 18 Aug 2023 16:26:27 +0200 Paul Leiber <paul@onlineschubla.de> wrote:
 > Package: ovmf
 > Version: 2022.11-6
 > Severity: important
 >
 > Dear Maintainer,
 >
 > After upgrading from Debian Bullseye to Debian Bookworm, existing HVM
 > DomUs using "firmware = 'ovmf'" (specifically Windows Server 2022 and
 > Windows 10) can't boot anymore. Booting these systems leads to the
 > Windows error 0xc0000225. I wasn't able to fix this error. Booting an
 > installation .iso leads to the same error. Booting the installation
 > media with "firmware = 'bios'" leads to a normal boot.
 >
 > This seems to be the effect of a change in ovmf sources, where a xen
 > specific platform was created in 2019:
 > https://lore.kernel.org/all/20190813113119.14804-1-anthony.perard@citrix.com/#t
 >
 > With some help I found a workaround. I could successfully start the
 > Windows DomU with ovmf firmware after removing the current ovmf package
 > and installing a previous ovmf package from Debian repositories,
 > specifically version 2020.11-2+deb11u1. This strongly indicates that the
 > cause of this issue lies in the ovmf package.
 >

Just a short update: The regression still exists in Trixie. I also managed to build OVMF from source with the instructions from [1] and with arctic's input to this bug report:

OvmfPkg/build.sh -a X64 -b RELEASE -p OvmfPkg/OvmfXen.dsc

Best regards,

Paul

[1] https://github.com/tianocore/tianocore.github.io/wiki/How-to-Build-With-Stuart