Dear qemu maintainers,

I will soon maintain ipxe along with Harry Chen, planing importing new
ipxe version snapshots and enabling more architectures. I noticed that
qemu-system-* is depending the ROMs provided by ipxe. So I believe it will
be necessary for coordination between the two packages.

Currently the package ipxe-qemu provides a series of ROM files for the
emulated NICs. Without manually specifying romfile=, qemu-system-* for
whichever architecture will try to load the corresponding ROM file.
Without the ROM file, qemu will refuse to start, unless an empty romfile=
is manually specified. In such condition, of course the PXE functionality
will not work.

Previously, each ROM file provided by ipxe-qemu is combined with two ROMs,
one is for x86 legacy booting and the other is for x86-64 UEFI booting.
The former provides the networking driver for the emulated NIC and full
PXE stack, while the later only provides the driver on the lower layer,
letting the firmware provide the upper PXE stack. When booting a virtual
machine of other architectures, say an arm64 virtual machine, the ROMs
will also be picked up by qemu-system-aarch64 by default, but not
providing any functionalities, since they do not contain code for that
architecture.

The first problem is that currently the package qemu-system-x86 depends
on ipxe-qemu, but other qemu-system-* packages recommend it. According to
the same behavior that qemu for no matter which architecture requires a
ROM file by default, should the relationship between all the qemu-system-*
packages be made the same?

The second problem is that Debian is additionally providing edk2
firmwares for arm64, riscv64 and loong64, enabling UEFI booting for
qemu-system for these architectures. I've tested that if drivers for these
architectures are also compiled in iPXE and combined into the ROM files,
the emulated NIC can be recognized in the UEFI firmware settings interface
(in Device Manager -> Network Device List). We now may have two options
for enabling drivers for these architectures:

Opt1: Combine the drivers for all the architectures into one ROM like
before.

Pros:
- changes are only needed in ipxe and no changes are needed for qemu;
- it is actually tested and indeed works;
Cons:
- the size of the ROM files will grow beyond the current 512k pow2
  bucket, affecting live migration. (See #881263)

Opt2: Using separate ROM files for each architecture. The name of the
files remain unchanged but the ROMs for each architecture is put into
separate directories, and maybe into separate packages, and qemu-system-*
is configured to search different directories for different architectures.

Pros:
- it will be unlikely that the size of ROM files grows beyond the current
  512k pow2 bucket, even if new architectures are introduced or ipxe is
  updated in the future.
Cons:
- changes are needed for qemu, and further coordination is required.

We have uploaded a new version of ipxe into stable (approval from FTP
team is needed), in which drivers for arm64 are enabled and combined into
the ROM files, since none of the ROM files exceeds the current 512k pow2
bucket.

Looking forward to your advice on introducing support for new
architectures to the ROM files.

Cheers,

Miao Wang

Hi,

I have uploaded new ipxe to experimental (1.21.1+git20250207.bd90abf4+dfsg-1~exp1).
This version has adopted Opt1, i.e. merging drivers for all arches into one file.
The following rom files have doubled its sizes to 1MB [1]:

efi-e1000, efi-e1000e, efi-rtl8139, efi-virtio, efi-vmxnet3

I have raised the severity to important since freeze is approaching, and it would be
better if we could get a consensus before trixie.

[1]: https://salsa.debian.org/debian/ipxe/-/commit/8a84bc846f713632a8acc51ddf28d7f59d5470fa

14.02.2025 17:36, Chen Shengqi wrote:
[...]

This is completely unacceptable, unfortunately, since it breaks migration of
qemu guests between different qemu invocations.

I didn't have a chance to look at this due to unrelated stuff, will do hopefully
tomorrow.

/mjt

Actually, the size of the ROM files in bookworm and bullseye is below 256K.
Rom file padding is not introduced in bookworm. As a result, even before
this change, live migration is already broken between bookworm and trixie.

Looking forward to your idea on how we will deal with this.

Cheers,

Miao Wang

03.02.2025 17:51, Miao Wang wrote:

Thank you for this effort.  And please excuse me for long delay, - I had
very little time lately which I can spare.

IIRC, there are two separate ROM files, - pxe-foo.rom and efi-foo.rom,
which are not combined.

This is a historic disparity, so to say.  In both cases it's possible to
get away from rom dependency by specifying romfile=, but in non-x86 case
these roms actually does nothing due to the wrong architecture.

I don't see this depends/recommends disparity as a problem.

I never thought about providing network booting for other architectures,
and didn't even know it is possible.  Yes, this is definitely interesting.

I did try to eliminate dependency on the rom files for non-x86, but not
very hard, - because it needs to be done in migration-compatible way and
it becomes twisty.

This is definitely not an option, because this breaks migration.  The
size bucket is here for a reason, the size must stay within the current
constraints.

Current bucket is 256k.  If ipxe source package is unable to keep the
roms within this size for trixie, I'll have to resort to building them
from qemu source instead, which I want to avoid.  This is principal.
(Currently I remove roms/ipxe/ dir from qemu source when doing the +ds
repack, - I'll have to stop doing this.  But this opens a possibililty
to also compress the efi roms using compression utility from edk2, so
there's at least a possible good side here).

When/why we went past the 256k limit for the efi roms?

This is the way to go, given the size constraints we have.

But while this can be done in debian by specifying different subdirs for
the rom files, I think it needs to be coordinated with qemu upstream, so
everyone can benefit from this.  We can try it as a proof-of-concept, to
demonstrate that it works.

Thanks,

/mjt

Actually, in the current ipxe-qemu package, efi-foo.rom is actually
combined from bin-i386-pcbios/foo.rom and src/bin-x86_64-efi/foo.efirom,
and pxe-foo.rom is solely bin-i386-pcbios/foo.rom. Also, qemu itself is
unaware of whether it is booted using legacy BIOS or UEFI, so whether
to use efi-foo.rom or pxe-foo.rom is hard-coded in the device emulation
code. A simple search in the qemu code base shows:

$ grep -R 'romfile = "' hw/
hw/net/pcnet-pci.c:    k->romfile = "efi-pcnet.rom",
hw/net/e1000e.c:    c->romfile = "efi-e1000e.rom";
hw/net/ne2000-pci.c:    k->romfile = "efi-ne2k_pci.rom",
hw/net/e1000.c:    k->romfile = "efi-e1000.rom";
hw/net/rtl8139.c:    k->romfile = "efi-rtl8139.rom";
hw/net/eepro100.c:    k->romfile = "pxe-eepro100.rom";
hw/net/vmxnet3.c:    c->romfile = "efi-vmxnet3.rom";
hw/display/vmware_vga.c:    k->romfile = "vgabios-vmware.bin";
hw/display/vga-pci.c:    k->romfile = "vgabios-stdvga.bin";
hw/display/virtio-vga.c:    pcidev_k->romfile = "vgabios-virtio.bin";
hw/display/ati.c:    k->romfile = "vgabios-ati.bin";
hw/display/qxl.c:    k->romfile = "vgabios-qxl.bin";
hw/virtio/virtio-net-pci.c:    k->romfile = "efi-virtio.rom";

You can have a try with the ipxe-qemu in experimental, which provides
ROMs supporting other architectures. A demonstration is:

qemu-system-aarch64 \
  -cpu max -machine virt \
  -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd \
  -smp cpus=1,cores=1,sockets=1 -m 1G \
  -chardev stdio,mux=on,id=char0 -nographic \
  -monitor chardev:char0 -serial chardev:char0 \
  -netdev user,id=net0,tftp=/tmp/test,bootfile=__nonexistent__ \
  -device e1000,netdev=net0,id=net0

You can see the firmware is trying to boot from PXE. When specifying
an empty romfile= to -device e1000, the firmware will not recognize
the NIC and will directly boot into the UEFI Shell.

For riscv64 and loong64, although not tested, I believe PXE will also
work if the PXE stack is enabled in the edk2 build.

It seems that the generated ipxe images has already been compressed.

According to my knowledge, the size limit was not passed because ipxe
is not updated in the Debian repo since buster, util Sven Geuer tried
to import changes made on the Ubuntu side into unstable, when the limit
is raised to 512K, the same to Ubuntu. Note that the ROMs in stable
now contains only code for legacy boot and x86-64 UEFI boot. I propose
that if we would like to choose to combine the images for other
architectures, we can raise the limit to 1M.

Yes, it would be nice if we can use different subduers for this. I propose
that in the rules file of qemu, we can prepend a target-arch-specific path
to FIRMWAREPATH, e.g. /usr/share/qemu-aarch64, where we can put all the
ROMS for arm64, including NIC ROMs and firmwares. And in the end (maybe
in the next stable release, forky), we can finally remove the common
/usr/share/qemu directory.

Cheers,

Miao Wang

Hi all,

I am looping Sergio in, who seems to maintain ipxe in Ubuntu recently
(correct me if wrong), and these changes in Debian are backported
from Ubuntu by Sven.

IIUC, qemu live migration with ipxe enabled cannot be performed during
the upgrade to focal or jammy due to the change to rom size changes.
I would also like to learn how Ubuntu handled this.

Thanks,
Shengqi Chen

24.02.2025 11:14, Chen Shengqi wrote:

Thanks.

This "with ipxe enabled" part is always true, since the pxe roms are
loaded once you enable a network adapter in the guest.

As far as I can see, Ubuntu does not handle this case at all, making
the guests unmigratable between different ubuntu host releases.


Now, back to the current situation.

Compressing x86-only ROMs makes them fit in 256Mb just fine, and makes
qemu guests migratable between distant versions of debian (the ROMs
in ipxe-qemu package are not compressed, because EfiRom utility from
edk2 sources is not packaged in debian still, after so many years).
However, this makes it unmigratable on ubuntu again, but it seems like
ubuntu just does not care, so no big deal.

I think combining ROMs from all architectures into a single file is not
a good idea at all, - it makes the thing just too large and it it too
easy to overgrow it.  Where network booting is relevant for non-x86
anyway?  I wonder why upstream qemu never bothered with this so far,
but ubuntu enabled it already?

/mjt

Actually, currently shipped ROMs have already been compressed. Legacy
boot images (including ROMs and other images) are compressed using zbin
located in src/utils/zbin.c with lzma and self-decompressed when booted.
EFI boot images are not compressed, but EFI ROMs are compressed using
efirom, located in src/utils/efirom.c, which includes the eficompress.c,
which is actually copied from EDK2. As a result, all the ROM files in
the package are already compressed, which can be seen by running
strings against these files, from which no meaningful strings can be
found.

Combining ROMs for x86 legacy boot and x86-64 EFI boot is at least
necessary, since when qemu cannot tell whether the current booting
firmware is legacy or UEFI. However, the x86-only ROMs are exceeding
256 KB now. In ipxe_qemu_1.21.1+git20220113.fbbdc3926+dfsg-4_all.deb,
the provided ROMs are x86 only, the size of some of which without
padding is larger than 256 KB.

for i in efi-*.rom; do
  echo -n "$i: "; xxd $i | \
    grep -v "ffff ffff ffff ffff ffff ffff ffff ffff" | \
    xxd -r | wc --bytes
done

efi-e1000.rom: 343040
efi-e1000e.rom: 343040
efi-eepro100.rom: 217600
efi-ne2k_pci.rom: 216576
efi-pcnet.rom: 216576
efi-rtl8139.rom: 345600
efi-virtio.rom: 345600
efi-vmxnet3.rom: 337920

As a result, even if we do not combine the ROMs, they cannot stay within
256KB limit.

Since we must break the size limitation, the idea that combining ROMs for
all the architectures is not that bad. Current size of the combined ROMs
is padded to 1MB, and the ROM with the maximum size without padding is
efi-virtio.rom, the size of which is 659456B, which leaves us 55% of the
size for the growth. As a result, 1MB size will be enough for a certain
of time.

As shown by tests, arm64 network booting is functioning with the edk2
firmware provided by Debian and the ipxe NIC ROMs. For riscv64 and
loong64, the emulated NIC can be recognized by the firmware with the
ipxe NIC ROMs. I believe network booting will also be functional when
the PXE stack is enabled in the EDK2 firmware.

Cheers,

Miao Wang

26.02.2025 11:39, Miao Wang wrote:

I haven't noticed eficompress & efirom in ipxe until you pointed this out.
That's good, because I was making an MR for edk2 to provide EfiRom in a
new binary package, - which is not needed obviously.  Okay, this is one
issue less.

Yup, there's no question here.

I built a few ROMs from current ipxe sources (on bookworm though) -
they all fits within 256k when built the way used by qemu:

../../edk2/BaseTools/Source/C/bin/EfiRom \
   -f 0x8086 -i 0x100e -l 0x02 \
   -b bin/8086100e.rom \
   -ec bin-x86_64-efi/8086100e.efidrv \
   -o efic-e1000.rom


So it looks like the thing fits within 256k perfectly.
I'll rebuild it with the toolchain from trixie, and with
the compression from ipxe side (without using edk2/EfiRom).

I'd say we should provide x86-only (legacy+efi) boot roms for x86, and
rely on native virtio support in edk2 for aarch64, riscv64 and loong64.
Personally I see no reason for providing pxe roms for qemu in ipxe for
anything but x86.  Network booting for these architectures is younger
than virtio, so we don't have to support old compatible baggage there.
Just use virtio-net for all more recent platforms and be done with it.

Hence, since (so far, it looks like, at least when compressing the efi
image), the size does not actually changes, we'll be nicely backwards-
compatible with the previous debian releases.

There's an open question about the actual ROM sizes though - why yours
are larger than mine.  I'm investigating.

Also, there's an open question about the actual need to provide and
use x86 roms for non-x86 targets, - historically these were required
but has always been useless.  I raised this question in the upstream
qemu mailing list.  But given the history, I don't think it's worth
the effort breaking this now, - maybe only to reduce Depends/Recommends
line just a little bit.


Thanks,

/mjt

I have found out the reason why the ROMs is larger in the debian package.
It is because many useless objects are pulled into the linked efidrv file.
The efidrv file should be very simple, only containing the driver, without
iPXE boot stack. It however contains those symbols although not using them.
I proposed a patch which can prevent linking unnecessary objects in and
reduce the size of the images.

To allow a virtual machine to boot from network, to use virtio network
is a simple solution. However, the users have the freedom to choose
whatever NIC they want, since qemu offers the possibility. To support
these cases is not so complex -- just build ROMs from iPXE and we can
support more cases. There is no such standard to deprecate or forbid
other NICs on those targets, so I don't see this as a compatible baggage.
The 256KB limitation will however finally indeed become a compatible
baggage.

I also noticed that among the edk2 firmwares packaged in Debian for x86_64,
arm64, loong64 and riscv64, only that one for arm64 can boot from network.
I have no idea why the other three cannot boot from network. I wonder
if it is expected.

Currently, missing a romfile for a virtual NIC is seen a fatal error for
qemu. As a result, without a ROM file, qemu for other architectures
cannot start. That's why there should be an rom, no matter for which
architecture, to just allow qemu to start.

Based on my previous discovery, it seems that to control the ROM size
within 256KB is possible. If we would like to provide network boot
support for non-x86 targets and keep the 256K compatiblity, a choice
would be split the ROMs for each architecture into the corresponding
path and let qemu for different targets search different paths.

Cheers,

Miao Wang

27.02.2025 00:56, Miao Wang wrote:
...
additional pxe architectures are necessary, while you think they are.

So if you really think users needs this freedom, how about me providing
"standard" ipxe roms of fixed size for x86 only in qemu-system-data package,
exactly like qemu upstream does, and you can ship ipxe-qemu with more
architectures?  The only caveat is that you'll have to move the roms
back to /usr/lib/ipxe/qemu/ where they were before.  Qemu will look there
first, and fall back to /usr/share/qemu/ if they're not there.

This way, when using qemu-provided roms, we'll have migratability and the
same feature set as upstream qemu provides, and with ipxe-qemu installed,
users will have an ability to boot other architectures your way, if such
need arises, but migration compatibility is not provided.

I'll add Conflicts: ipxe-qemu (<< current), so that both wont be installable
at the same time, and on upgrade, ipxe-qemu is removed - this will also keep
migratability.

Yes, I'll have to ship (stop stripping) ipxe sources within qemu, and will
have to build ipxe there, so it's extra work for me, but I do care about
migration ability more than some artificial freedom of choice.

..

Neither me nor upstream qemu see any use for multi-arch roms (or for
multiple per-arch roms), for the reasons already stated multiple times:
for all current architectures besides x86, we've better alternatives
already supported natively by edk2.  ipxe roms are only needed for legacy
x86 guests.

/mjt

Since no one believes multi-arch ROMs is necessary, I will not bother to
include them in iPXE. I'll later update ipxe to provide an x86-only ROM,
combined from the ROMs for legacy pc and x86-64 EFI, within 256KB limit.

Cheers,

Miao Wang

03.03.2025 16:17, Miao Wang wrote:
...

That would be really great.  Because I'm having some.. issues building this
stuff from qemu sources (I have to stop stripping both edk2 and ipxe, or
get upstream to update ipxe - current ipxe in qemu does not support
compression in efirom utility).

Thank you!

/mjt

We believe that the bug you reported is fixed in the latest version of
ipxe, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1095073@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Shengqi Chen <harry@debian.org> (supplier of updated ipxe package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Tue, 18 Mar 2025 17:03:54 +0800
Source: ipxe
Architecture: source
Version: 1.21.1+git20250317.42a29d56+dfsg-1
Distribution: unstable
Urgency: medium
Maintainer: Miao Wang <shankerwangmiao@gmail.com>
Changed-By: Shengqi Chen <harry@debian.org>
Closes: 1095073
Changes:
 ipxe (1.21.1+git20250317.42a29d56+dfsg-1) unstable; urgency=medium
 .
   [ Miao Wang ]
   * d/control: add <!nocheck> back for qemu-system-misc
   * d/p/reduce_efirom_size.patch: remove unnecessary objects
   * d/p/fix-lkrn-long-version.patch: use upstream patch
   * d/tests: add pxe boot tests
   * d/rules: output size information during combining ROMs
   * efirom: remove non-x86 roms and remain within 256k (closes: #1095073)
   * d/rules: fix build failure on i386
 .
   [ Shengqi Chen ]
   * New upstream version 1.21.1+git20250317.42a29d56+dfsg
   * d/control: bump std-ver to 4.7.2 (no changes required)
   * d/patches: remove applied patch and refresh
Checksums-Sha1:
 22f62254adf4d33a99c8e3aa9edded16f8bde5f8 3316 ipxe_1.21.1+git20250317.42a29d56+dfsg-1.dsc
 ec1d7d252438e3f97edf4bc59d17803cad38954a 2871940 ipxe_1.21.1+git20250317.42a29d56+dfsg.orig.tar.xz
 6a11ef7411f3b2271db210e2cebe672e2169c01e 24508 ipxe_1.21.1+git20250317.42a29d56+dfsg-1.debian.tar.xz
 2c70905d67a658bd1fec33a8bba089e651ecfa9e 11340 ipxe_1.21.1+git20250317.42a29d56+dfsg-1_amd64.buildinfo
Checksums-Sha256:
 82c2db9e7e68923ca2f384cf57a085510c57664cc88e8ddd0672fb3091fb8b96 3316 ipxe_1.21.1+git20250317.42a29d56+dfsg-1.dsc
 149ea7a227321eea0cb236107a20897c6d637fb9351adfd46c37d09fb980905a 2871940 ipxe_1.21.1+git20250317.42a29d56+dfsg.orig.tar.xz
 0ba6af47e5cef41df370d4a2d6b3c92f12a2e944c727ae46f177608c18a8b79f 24508 ipxe_1.21.1+git20250317.42a29d56+dfsg-1.debian.tar.xz
 a07f1808b5e5d0dbcd84e28a5327f5605fdb7901dcc97c37ece7a2606b71951f 11340 ipxe_1.21.1+git20250317.42a29d56+dfsg-1_amd64.buildinfo
Files:
 7110b9c2b06ad06120ad24df529507bc 3316 admin optional ipxe_1.21.1+git20250317.42a29d56+dfsg-1.dsc
 d876d52bb4f4a277250eeb209888b8ee 2871940 admin optional ipxe_1.21.1+git20250317.42a29d56+dfsg.orig.tar.xz
 0d88b8e9397765d4d395fe02dd2669da 24508 admin optional ipxe_1.21.1+git20250317.42a29d56+dfsg-1.debian.tar.xz
 9c6790644d16e6966db582fd3ff030ac 11340 admin optional ipxe_1.21.1+git20250317.42a29d56+dfsg-1_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEE+Fg++qmpHzqjSzySzhGnsHs6uUYFAmfZOQsACgkQzhGnsHs6
uUbPRRAApyO4bZUj8m2aRzariX2xnf8PZHXRZowF8pb/MfD9c0ZK6R2zDicmZmk3
8qPl/q57fl/mXG7y6f0YpzCyYxON1zyvU2DoqOJHSILyiP83LSLAGnrGwiOj20FY
46pSx3zehLfiZFt3hRlcTbR8rZdmNEBKGi+wbCHBIxKa+Hy7/B1cK+zJbl6ADyLp
kNNbNkiFobDL3SSmAduLKVbIwO5Y4DNq9cOWdFoHsmTet6Yc9zu/6BkH2Jv+R5th
bSKRT/SRHad+6Zr20Aezsk8PeH2LBY/+DTuMoxLXhKdh8hQAkrCrlESFnM4i7Vkb
PYT+mBafsTO1aAiVOyYgcGg9Q9ccdF10EVkD61jJL6/wp/tO97Q1mE/UN9zH7JDc
NnXwXZ2di323QTsEHYTtFPPP27/LZ6PmAsRdU4Gy8C9c5sH751bC2mVwyT/GiZDx
txtF7vJddwNRj4PffJtkdd5KoFOYQVzmNaKdycQFCjB6yxb7WW6lcc+tiV2qBtZy
XRYZuL9gkFhu2hIfqtI6iYSLwMOGg3RJAd4xJ65hMq7COQ0Rg1lWgApK8wQge7ea
Ki5HHES/7VvHCT6qmIHOfrcLAlJV0d5Glx5J7nJJ95wnZD9AuRwlp6SmfmM5VSRC
ir4tAnqjvceCEjbHgkfNz8YkZQFCSiUiIEEKdaS+SDR32ExyxlU=
=/uoB
-----END PGP SIGNATURE-----