TL;DR:
- rom size change (coming on next ipxe merge for Debian) break migrations
- we need a way to allow incoming migrations to work but still be able
to update roms
- I suggest a machine type based mapping to compat roms

Details:

Hi,
I've seen that the upcoming ipxe updates to a more recent git checkout
will make the efi-*.rom files pass the 256k boundary they are
currently under.
We faced that in Ubuntu earlier [5] as we have https enabled as
additional feature which let us grow out of that boundary earlier.

This is inherently a Distribution problem as it depends on what roms
we bundle in a release. I have prepared an example case below to
illustrate the case and a proof of concept solution.
We will eventually need our own solution, as even Debian and Ubuntu
differ in some details. But I'd be happy to discuss and agree with you
on the way to do so, to share as much as possible when doing so.

Note: My example has two systems, one with Zesty one with Artful.
You'd encounter the same that I illustrate when updating ipxe in
Buster in migrations stretch to buster.

1. Set up two KVM systems and spawn a guest on one of them

    I usually use uvt, but you can use whatever you prefer (no uvt in Debian)
    $ uvt-simplestreams-libvirt sync --source
http://cloud-images.ubuntu.com/daily arch=amd64 label=daily
release=zesty
    $ uvt-kvm create --password=ubuntu guest1 release=zesty arch=amd64
label=daily

2. set up for live migration (mostly share ssh keys, maybe shared storage)

3. check that you can migrate between them as-is
   I use copy-storage-all to eliminate some constraints on the setup
   $ virsh migrate --live --copy-storage-all guest1 qemu+ssh://10.0.4.174/system
   and back
   $ virsh migrate --live --copy-storage-all guest1 qemu+ssh://10.0.4.2/system

Lets be clear those systems can have different rom version/sizes and
work fine on migrations.
But once you have a target with a newer rom breaking the former
power-of-2 size like the new ipxe-rom I installed from a ppa:
  - Old 241152 bytes efi-virtio.rom
  - New 300544 bytes efi-virtio.rom

Now the migration will hit the following issue:
  qemu-system-x86_64: Length mismatch:
0000:00:03.0/virtio-net-pci.rom: 0x40000 in != 0x80000: Invalid
argument
  qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram'

That is the former 256k not matching into the now 512k pow2 bucket of the rom.
I tried to downsize the roms but failed and I discussed some ways out [1].
The TL;DR is we could have padded it to a bigger size before, but
since we didn't we have an issue now.

Based on the suggestion in [1] I developed a POC.
This includes we would provide something like a ipxe-qemu-256k compat
deb which is co-installable and adds 256k- prefixed roms for the roms
(all or just a subset upon decision). So /usr/lib/ipxe/qemu/ would
look like [2].
Further qemu would hold a patch similar to my POC [3]

This makes it work in my test outlined above - even backward migration
is working.

My suggestion would be:
1. QEMU: Each of us develops a patch for the next qemu (2.11 I'd
think) = this bug
2. QEMU: adds a depdends to the new compat package (see 3)
3. IPXE: create a ?new (= rename the current)? package to carry the
old content as compat as long as needed?
4. IPXE: along all I'd ask to enable https in Debians ipxe, as size
change isn't stopping it anymore (we cross it anyway).
5. IPXE: I have a checker in ipxe for Ubuntu that I'd like to contrib
to catch those early (before they go into the archive) [4]

I'd file extra bugs for IPXE for 4,5 if we agree. And an ITP for 3 ?.
But you might have different opinions (or not care about migration
across releases) and I'm open to other solutions as well.
But if this is important for stretch->buster then I'd be happy if we
could agree on one "style" of approach.
Even with both of us needing different content in the 256k- roms as
well as needing to patch different machine types agreeing on one
approach will be better.

P.S: I targeted this to qemu as this is where alternative solutions
would have to be. I wanted to keep the discussion in one place before
cloning the bug and drifting off. So I CC'ed Bastian Blank here for
ipxe experience.

[1]: http://lists.nongnu.org/archive/html/qemu-devel/2017-11/msg00193.html
[2]: http://paste.ubuntu.com/25925091/
[3]: http://paste.ubuntu.com/25925120/
[4]: https://git.launchpad.net/~usd-import-team/ubuntu/+source/ipxe/commit/?id=6eb7e240412930d64878fcedd23969c929ae51b1
[5]: https://bugs.launchpad.net/ubuntu/+source/ipxe/+bug/1713490

09.11.2017 16:47, Christian Ehrhardt wrote:

I'm strongly against these changes in debian infrastructure.
Upstream will face the same problem once they import new ipxe,
and will have to deal with this somehow as well. If we will
follow this change, we will both stay compatible with upstream
and ourselves.  But adding this incompatibility to debian/ubuntu
and dealing with it inside debian/ubuntu makes us incompatible
with both upstream (and the world) and partially ourselves..

Especially if upstream choses to do it the other way, we'll
either have to support our solution (TOGETHER with upstream
solution) for some singnificant amount of time, or we'll become
incompatible with ourselves again.

Hmm...

/mjt

I see and agree with your point - IF - upstream would go along with
the same versions and handles it.
Looking back there are a few issues that make me doubt we can just
"rely on upstream" in this case:
1. ipxe and qemu are different projects nothing forces them to e.g.
match qemu 2.11 with a given ipxe version.
    The ones who define what goes together are the Distributions => we
     That is why I doubt there will be an upstream "fix".

2. rom sizes are build option dependent, thereby each
distribution/upstream will face the issue at a different point in
time.
    I was able to prove that e.g. with the debian build options it
will pass that size already as of today, but there is nothing in qemu
yet.

Due to (1)+(2) upstream qemu can't decide to which version to couple
such changes - they just can't know for all downstreams as they are
different.

3. Deadlines are different Buster vs Ubuntu 18.04 vs
other-Distro-releases we all will have to make the final "cutoff" at a
different time.
    That further limits the chance that upstream will fix it for all of us.

4. I got the impression this is nothing Upstream usually handles. Look
at the (not) overwhelming feedback to [1] and even those I got were:
    a) we solved that long ago via padding, you are doomed now
    b) we fixed it with a workaround ourself

Yes, I clearly see the maintenance pain this might bring - and I'd be
very happy if I could assume we could "just go with upstream on this".
But I doubt it will happen, so I wanted to make sure we have a way to
"get out of it" on our own.

Once the new ipxe is uploaded it is kind of "too late", size will
change and you'll see all the issues I outlined - that is why I was
calling for coordination up front.
With that in mind what should we do now?
- Wait until qemu 2.11 is out and see if there is anything happening
upstream (I didn't see any related action so far)?
- If there is no change upstream eventually, then consider my
suggestion (or any others if there are some)?
- Go back to upstream on my thread with these issues - especially your
POV that you'd expect upstream to handle it?

[1]: http://lists.nongnu.org/archive/html/qemu-devel/2017-11/msg00193.html

Hi,
I didn't hear back any more for a while on this bug, and since 2.11 is
ahead I wanted to ask for the last status/opinion on this.
I discussed with several people on this since my last post, but so far
none could:
- come up with a different solution
- come up with a way how upstream could solve it

That said without a different way to fix it - even with some refusal
to the suggested change left - I'd again ask to do this together in
Debian and Ubuntu. As I outlined before just the same applies to
Debian, which is why I asked to jointly do it to at least agree e.g.
in package names and overall strategy.

Unless there is an alternative solution coming up for myself (in
Ubuntu) this is rather simple - I can do it this way or declare
migrations of guests from older systems won't work (which is a no-go
IMHO).

cu,
Christian

It looks like we managed to keep the rom sizes in size for quite a long
time.  This is actually an upstream issue - if upstream can't keep the
roms size unchanged, they have to invent a work-around. And we shouldn't
enable more feature than upstream enables (IMHO anyway), to keep our
roms in shape.

Thanks,

/mjt