I'm attaching a new version with a grub.cfg which is tested and seems to work. I've also taken the liberty of filing this into a wishlist bug (keeping entire quoted text below for the bug). I needed to do a little hack which is to rmmod memdisk before searching for /boot/grub/grub.cfg or else it just loops infinitely finding the grub.cfg in the memdisk. I have a feeling that the same thing might apply to debian/grub-firmware-qemu_grub.cfg. Ian.
On Sat, 2014-08-23 at 21:38 +0100, Ian Campbell wrote: (Context for new CC: I'd like to provide a Debian package for dom0 which includes pvgrub2 binaries which can be referenced from your domain cfg files as the bootloader/kernel. The associated Debian wishlist bug is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=759018) [...] The net result of the above is that the dom0 core.img built by the package will search /boot/grub/{i386,x86_64}-xen/core.elf (for the arch matching itself only) and then the version without the /boot prefix (for systems with separate /boot). Colin mentioned at debconf that it would be good if these paths were standard across all distros -- probably this is best treated as something which upstream Xen specifies a standard on, so CCing xen-devel and Ian J who has previously been involved with these conversations. I'm not sure what the best way to promulgate the spec is -- I think a patch to add xen.git/docs/misc/pvgrub2.markdown would be sufficient (it would end up under http://xenbits.xen.org/docs/). Other alternatives for the paths might be things in the /boot/xen "namespace" rather than /boot/grub, I don't mind which so long as it is decided. Unless someone has a strong preference I'm inclined to go with the paths as they happen to be in the patch above. Ian.
there is the matter of establishing this as a boot protocol upstream, which I hope I can leave up to you to sort out. For the information of pkg-grub-devel: I have given Ian commit access to pkg-grub, on the basis that he is much more familiar with Xen than I am and can start maintaining the Xen-related bits himself. Welcome! This is what https://lists.gnu.org/archive/html/grub-devel/2013-12/msg00185.html and subthread was about, but that never reached a clear conclusion, although Andrey's patch seemed fairly reasonable to me. The rmmod hack is OK for now. Static seems reasonable until we need otherwise. This file is probably unnecessary. dh_install will create directories as needed in response to debian/grub-xen-host.install. This is something that could perhaps be simplified if the guest boot protocol were standardised. I think there is some justification for having it be in /boot/xen/ etc. instead, so that we don't have to deal upstream with variations such as distributions that configure with "--program-transform-name=s/grub/grub2/". You may need to arrange for this to be executed using normal mode rather than the rescue shell. See debian/build-efi-images for a possible source of inspiration. As discussed, grub-xen-host shouldn't be in REAL_PACKAGES; it should be in something new instead. Then you won't need to have dummy debian/stamps/configure-grub-xen-host* rules.
That was I think roughly what Ian J suggested late last year. My preference would be to have it be under /boot/xen/, if you're OK with that. The rationale is that some distributions configure GRUB with "--program-transform-name=s/grub/grub2/", resulting in /boot/grub2/, and I think life will be better if dom0 code doesn't have to deal with that. As for the basename, "core.elf" is probably a bit generic if it's under /boot/xen/. We just need something vaguely sensible there, though, and then the Debian grub-xen package and equivalents in other distributions can create it. There is also the question of whether the guest-side name should mention GRUB. One might argue that it shouldn't because all that matters is that it uses the Multiboot protocol. Then there is the question of who gets to own the architecture names ... As a strawman, how about: /boot/xen/pvboot-$arch.elf ... where $arch is the first bit of a GNU config triplet, modified such that we always use i386 for that family?
In order to support pvgrub (or other bootloader) from dom0 chainloading a pvgrub (or other) from within the domU filesystem we need a standard for where the stage 1 bootloader should look and what it should expect to find there. Add a document along those lines. Signed-off-by: Ian Campbell <ijc@hellion.org.uk> Cc: Ian Jackson <Ian.Jackson@eu.citrix.com> Cc: Colin Watson <cjwatson@debian.org> Cc: 759018@bugs.debian.org --- docs/misc/xenpv-bootloader.markdown | 58 +++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 docs/misc/xenpv-bootloader.markdown diff --git a/docs/misc/xenpv-bootloader.markdown b/docs/misc/xenpv-bootloader.markdown new file mode 100644 index 0000000..a5a0665 --- /dev/null +++ b/docs/misc/xenpv-bootloader.markdown @@ -0,0 +1,58 @@ +# Xen PV Bootloader Protocol + +## Introduction + +One method for booting a Xen PV guest is to use a PV bootloader, that +is, a bootloader which is itself a PV kernel but which behaves as a +bootloader (examples include the pvgrub-legacy and grub2 targeting +Xen). + +In many cases the user wishes to manage this PV bootloader from within +the guest, and therefore wishes to chainload something from the guest +filesystem, most likely via a stage 1 PV bootloader provided by dom0. + +The purpose of this document is to define the paths within the guest +filesystem where a stage 1 bootloader should look for the in-guest PV +bootloader to load and the protocol/format expected from the +to-be-chainloaded bootloader. + +## Protocol + +### x86 + +The bootloader binary should be an ELF file of the appropriate type +(32- or 64-bit). It should contain the standard Xen ELF notes allowing +it to be loaded by the Xen toolstack domain builder (TBD: Reference). + +### ARM + +TBD + +## Paths + +The second stage bootloader should be installed into the guest filesystem as: + + * `/boot/xen/pvboot-<ARCH>.elf` + +Where `<ARCH>` is the first element of the GNU triplet e.g. one of: + + * i386 (nb only i386, not i686 etc), corresponding to the Xen x86\_32(p) arch; + * x86\_64, corresponding to the Xen x86\_64 arch; + * arm, corresponding to the Xen arm32 arch; + * aarch64 corresponding to the Xen arm64 arch; + +It is allowable for `/boot` to be a separate filesystem from `/` and +therefore stage 1 bootloaders should search +`/boot/xen/pvboot-<ARCH>.elf` and `/xen/pvboot-<ARCH>.elf` (in that +order). The `xen` directory should be on the same filesystem as /boot +and therefore it is not necessary to search for /pvboot-<ARCH>.elf. + +On processors which have 32- and 64-bit variants (i.e. x86\_32/x86\_64 +or arm32/arm64) it is not in general possible under Xen for a +bootloader to boot a kernel of a different width from itself, and this +extends to chainloading from a stage one. Therefore it is permissible +to have both `/boot/xen/pvboot-i386.elf` and +`/boot/xen/pvboot-x86\_64.elf` present in a guest to be used by the +appropriate stage 1 (e.g. for systems with 32-bit userspace and an +optional 64-bit kernel). +
Reviewed-by: Don Slutz <dslutz@verizon.com>
-Don Slutz
want this to be ELF, so maybe we shouldn't hardcode that in the architecture-independent part of the name. If you think this might be a reasonable concern then you might want to adjust this. Otherwise LGTM. Reviewed-by: Colin Watson <cjwatson@debian.org> Thanks,
This crossed my mind too (it might even be the case on arm*, not sure yet). My thinking was that this spec would already need adjusting for this (to describe the non-ELF thing, if nothing else) so this it could be tweaked then too. Or I could e.g. omit all the vestigial ARM stuff and explicitly list the two x86 options, bypassing the whole question of future arches, who would then need to add themselves to this doc. Thanks.
While talking to Colin IRL a couple of nights ago I realised that this stuff is most likely more specific to x86 PV than I had originally thought. ARM for example is far more likely to follow the ARM VM spec[0] and use a UEFI firmware as the stage 1 and the EFI bootprotocol to find and execute the stage 2 (e.g. grub.efi from the ESP). And who knows what any future arches might do. So I intend to update the doc to be less arch agnostic and make it just cover x86. If some future arch wants something along these lines then they would have to patch this spec anyway. Ian. [0] http://lists.linaro.org/pipermail/cross-distro/2014-March/000668.html
In order to support pvgrub (or other bootloader) from dom0 chainloading a pvgrub (or other) from within the domU filesystem we need a standard for where the stage 1 bootloader should look and what it should expect to find there. Add a document along those lines. This is currently x86-specific since it is expected that other new architecures (including ARM) will either not need to implement this due to the existence of more generic specifications for that arch (e.g. the ARM VM spec) or will update this document in due course. Signed-off-by: Ian Campbell <ijc@hellion.org.uk> Cc: Ian Jackson <Ian.Jackson@eu.citrix.com> Cc: Colin Watson <cjwatson@debian.org> Cc: 759018@bugs.debian.org --- v2: Don't try and second guess future developments on non-x86 architectures, making the document more x86 centric. --- docs/misc/x86-xenpv-bootloader.markdown | 49 +++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 docs/misc/x86-xenpv-bootloader.markdown diff --git a/docs/misc/x86-xenpv-bootloader.markdown b/docs/misc/x86-xenpv-bootloader.markdown new file mode 100644 index 0000000..ec8854e --- /dev/null +++ b/docs/misc/x86-xenpv-bootloader.markdown @@ -0,0 +1,49 @@ +# Xen x86 PV Bootloader Protocol + +## Introduction + +One method for booting an x86 Xen PV guest is to use a PV bootloader, +that is, a bootloader which is itself a PV kernel but which behaves as +a bootloader (examples include the pvgrub-legacy and grub2 targeting +Xen). + +In many cases the user wishes to manage this PV bootloader from within +the guest, and therefore wishes to chainload something from the guest +filesystem, most likely via a stage 1 PV bootloader provided by dom0. + +The purpose of this document is to define the paths within the guest +filesystem where a stage 1 bootloader should look for the in-guest PV +bootloader to load and the protocol/format expected from the +to-be-chainloaded bootloader. + +## Protocol + +The bootloader binary should be an ELF file of the appropriate type +(32- or 64-bit). It should contain the standard Xen ELF notes allowing +it to be loaded by the Xen toolstack domain builder (TBD: Reference). + +## Path + +The second stage bootloader should be installed into the guest +filesystem as: + + * `/boot/xen/pvboot-<ARCH>.elf` + +Where `<ARCH>` is the first element of the GNU triplet e.g. one of: + + * i386 (nb only i386, not i686 etc), corresponding to the Xen + x86\_32(p) arch; + * x86\_64, corresponding to the Xen x86\_64 arch; + +It is allowable for `/boot` to be a separate filesystem from `/` and +therefore stage 1 bootloaders should search +`/boot/xen/pvboot-<ARCH>.elf` and `/xen/pvboot-<ARCH>.elf` (in that +order). The `xen` directory should be on the same filesystem as /boot +and therefore it is not necessary to search for /pvboot-<ARCH>.elf. + +It is not in general possible under Xen for a bootloader to boot a +kernel of a different width from itself, and this extends to +chainloading from a stage one. Therefore it is permissible to have +both `/boot/xen/pvboot-i386.elf` and `/boot/xen/pvboot-x86\_64.elf` +present in a guest to be used by the appropriate stage 1 (e.g. for +systems with 32-bit userspace and an optional 64-bit kernel).
Ack. Thanks. Is there a policy regarding using e.g. people/foo branches in the main grub.git or shall I go create myself a gitorious repo for WIP stuff? I've attached a little mini series which addresses the issues you raised and switches to the proposed /boot/xen naming scheme. I've also added some new patches. The first causes grub-xen to actually produce things under /boot/xen, I'm not entirely sure about it though -- it places /boot/xen/pvboot-x86_64.elf as required but grubdir (e.g. all the modules etc) are still under /boot/grub. I wasn't sure if it was better to actually move grubdir to /boot/xen too. The implementation is a bit unsatisfactory since grub-mkimage doesn't let you change only the output file without moving grubdir (AFAICT), so it generates and then moves it. Lastly I made grub-xen not conflict with the other grub-* package. grub-xen is a bit of a special case since you can have a filesystem which is bootable both as PV (grub-xen) or HVM (grub-pc or grub-efi). Switching to using "normal" made this go away in the end, since the bootstrap.cfg doesn't need to use /boot/grub/grub.cfg as it's next stage, so there is no conflict. Ian.
Given that it's basically me, I haven't really worried about policy, but I generally think people/FOO/BAR type branches are a good idea. This all LGTM. Please go ahead. I would be inclined to leave grubdir at the default. It already uses platform-specific subdirectories, and there are already situations on other platforms where the core image is placed somewhere else. Seems surprising. Can't you use the -o option? I don't think that does anything to grubdir. Or failing that the default is to output the image to stdout, so you could just redirect. Ah, but it looks like you're actually talking about grub-install here. Yeah. Moving it after the fact isn't the end of the world, but armed with a Xen-blessed spec I think we should fix grub-install upstream to automatically DTRT here. You might consider doing this via dh_installdir instead; but I'm a bit twitchy about packages directly shipping entries in /boot for some reason. Maybe mkdir -p in the postinst (later in grub-install) instead? Thanks,
About the same. Colin suggested the GNU config triplet names in his strawman and I couldn't think of a reason to change. Yes. Essentially you write kernel = /path/too/pvgrub-<32bit-name> or -<64bit-name> in your guest config. Yes, this sucks. AIUI in cloud interfaces etc you generally have to ask for a 32- or 64-bit domain explicitly too. I suppose that Would Be Nice(tm). I'm not sure but it might be that for a PVH guest (once grub is ported to that) we might be able to switch bitness at runtime and avoid this whole mess (which comes largely from the size of the p2m entries and the difficulties in switching, which is hidden from pvh guests) You mean if all guests happen to only use one bitness? You waste a roundtrip through a stunt domain which will do the exiting trick and restart, or you supply the right dom0 grub to start with I guess. I can't think of anything which would need to be changed in the spec now preemptively, FWIW. Ian.
Colin Watson writes ("Re: Bug#759018: [PATCH RFC] Provide prebuilt grub-xen binaries for host (dom0) use"):
...
The general scheme seems sound.
Ian Campbell:
Since this path is in /boot/xen and contains an executable which
expects to run in the Xen PV environment, it could also use Xen names
for the architectures. I don't know whether a GNU config triplet arch
name as you suggest is easier or harder than that.
I have a question about the spec's wording about bitness:
+It is not in general possible under Xen for a bootloader to boot a
+kernel of a different width from itself, and this extends to
+chainloading from a stage one. Therefore it is permissible to have
+both `/boot/xen/pvboot-i386.elf` and `/boot/xen/pvboot-x86\_64.elf`
+present in a guest to be used by the appropriate stage 1 (e.g. for
+systems with 32-bit userspace and an optional 64-bit kernel).
Is it therefore expected that the host admin will be told out of band
what bitness the guest would prefer ? And that then the host
toolstack will set up that bitness of guest, load its pvgrub for that
bitness, and hope that the guest has an appropriate-bitness core image
load in the canonical place ?
I wonder if we might, in the future, want this to be more automatic.
I guess we could have a feature in the host's 64-bit pvgrub which
would look for and load 64-bit guest pvgrub if it exists, and
alternatively check for 32-bit guest pvgrub and if found exit
signalling somehow to the host toolstack to restart the domain with
the other bitness.
But what if the host has both 64- and 32-bit pvgrub but in fact only
has one bitness of kernel ? Signalling this back to the host by
somehow hiding or renaming one of the bitnesses of guest pvgrub seems
unpleasant.
I mention this just in case there's a better way of organising this
which might depend on a refinement to the host/guest interface.
Thanks,
Ian.
There is no real technical obstacle to prevent PV domains switching width after starting. In the past, the toolstack has directly loaded the target running kernel, but that behaviour/assumption is no longer correct given these chainloading plans. As we no longer support 2-level PV guests, allowing a PV domain to switch between 3 and 4 levels becomes easier to manage from Xens point of view. From the top of my head, it would involve relaxing the restriction that 3 level PV guests can't pin L4 tables (but still enforce that a 3 level PV guest can't load an L4 pagetable), and provide a new hypercall which takes a desired with, new cr3 (referring to a pinned pagetable of the appropriate new width) and a new eip to jump to. ~Andrew
All I know is that when this stuff was first invented lots of people thought quite hard about this and it is trickier than you might think. and rewrite/remap the p2m and m2p tables on the fly. and arrange somehow that all the memory used by the guest is actually visible in the m2p mapping for that guest (since 32-bit guests don't see the whole host m2p). and a couple of other things which I can't quite remember right now. Anyway, you are welcome to try ;-) Ian.
We discussed this IRL (nearer to then than now, so I hope I recall the outcome correctly) and decided that we couldn't think of any change we could proactively make to the spec which would help the person who eventually comes to implement some sort of scheme for dynamically selecting the bitness of pvgrub to use. v2 of the spec was posted at http://lists.xen.org/archives/html/xen-devel/2014-08/msg02964.html does it look ok to you? Ian.
(16:52:37) ijc: Diziet: Did you (intend to) ack <1409343270-2375-1-git-send-email-ijc@hellion.org.uk>? (16:55:14) Diziet: ijc: I'm sure I intended to ack that, sorry. (16:55:42) ijc: Will you or can I take the above as one? (16:57:48) Diziet: ijc: Please do take it as an ack. So I have fabricated an Ack and applied.
Upstream have defined a specification for where guests ought to place their xenpv grub binaries in order to facilitate chainloading from a stage 1 grub loaded from dom0. http://xenbits.xen.org/docs/unstable-staging/misc/x86-xenpv-bootloader.html The spec calls for installation into /boot/xen/pvboot-i386.elf or /boot/xen/pvboot-x86_64.elf. Signed-off-by: Ian Campbell <ijc@hellion.org.uk> --- util/grub-install.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/util/grub-install.c b/util/grub-install.c index 70f514c..5f4d737 100644 --- a/util/grub-install.c +++ b/util/grub-install.c @@ -1979,6 +1979,14 @@ main (int argc, char *argv[]) } break; + case GRUB_INSTALL_PLATFORM_I386_XEN: + grub_install_copy_file (imgfile, "/boot/xen/pvboot-i386.elf", 1); + break; + + case GRUB_INSTALL_PLATFORM_X86_64_XEN: + grub_install_copy_file (imgfile, "/boot/xen/pvboot-x86_64.elf", 1); + break; + case GRUB_INSTALL_PLATFORM_MIPSEL_LOONGSON: case GRUB_INSTALL_PLATFORM_MIPSEL_QEMU_MIPS: case GRUB_INSTALL_PLATFORM_MIPS_QEMU_MIPS: @@ -1987,8 +1995,6 @@ main (int argc, char *argv[]) case GRUB_INSTALL_PLATFORM_MIPSEL_ARC: case GRUB_INSTALL_PLATFORM_ARM_UBOOT: case GRUB_INSTALL_PLATFORM_I386_QEMU: - case GRUB_INSTALL_PLATFORM_I386_XEN: - case GRUB_INSTALL_PLATFORM_X86_64_XEN: grub_util_warn ("%s", _("WARNING: no platform-specific install was performed")); break;
What ensures that /boot/xen exists? Should grub-install create it?
I expected it would just happen, but looking again only grub_install_copy_files has this behaviour, not grub_install_copy_file. Should I add grub_install_mkdir_p in the hunks above, or should I be thinking of patching grub_install_copy_file? I've been testing this via the Debian packaging, which contains the mkdir before it invokes grub-mkimage. Ian.
В Wed, 22 Oct 2014 14:37:56 +0100 Ian Campbell <ijc@hellion.org.uk> пишет: grub_install_copy_files is pretty misleading, it should really be named grub_install_platform_files as this is the only thing it does. All other usages of grub_install_copy_file assume existing directory, so I guess explicitly creating directory is fine. Please do not use hardcoded absolute path names. It should respect bootdir to facilitate offroot installation.
Upstream have defined a specification for where guests ought to place their xenpv grub binaries in order to facilitate chainloading from a stage 1 grub loaded from dom0. http://xenbits.xen.org/docs/unstable-staging/misc/x86-xenpv-bootloader.html The spec calls for installation into /boot/xen/pvboot-i386.elf or /boot/xen/pvboot-x86_64.elf. Signed-off-by: Ian Campbell <ijc@hellion.org.uk>--- v2: Respect bootdir, create /boot/xen as needed. --- util/grub-install.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/util/grub-install.c b/util/grub-install.c index 70f514c..7a7734e 100644 --- a/util/grub-install.c +++ b/util/grub-install.c @@ -1979,6 +1979,28 @@ main (int argc, char *argv[]) } break; + case GRUB_INSTALL_PLATFORM_I386_XEN: + { + char *path = grub_util_path_concat (2, bootdir, "xen"); + char *dst = grub_util_path_concat (2, path, "pvboot-i386.elf"); + grub_install_mkdir_p (path); + grub_install_copy_file (imgfile, dst, 1); + free (dst); + free (path); + } + break; + + case GRUB_INSTALL_PLATFORM_X86_64_XEN: + { + char *path = grub_util_path_concat (2, bootdir, "xen"); + char *dst = grub_util_path_concat (2, path, "pvboot-x86_64.elf"); + grub_install_mkdir_p (path); + grub_install_copy_file (imgfile, dst, 1); + free (dst); + free (path); + } + break; + case GRUB_INSTALL_PLATFORM_MIPSEL_LOONGSON: case GRUB_INSTALL_PLATFORM_MIPSEL_QEMU_MIPS: case GRUB_INSTALL_PLATFORM_MIPS_QEMU_MIPS: @@ -1987,8 +2009,6 @@ main (int argc, char *argv[]) case GRUB_INSTALL_PLATFORM_MIPSEL_ARC: case GRUB_INSTALL_PLATFORM_ARM_UBOOT: case GRUB_INSTALL_PLATFORM_I386_QEMU: - case GRUB_INSTALL_PLATFORM_I386_XEN: - case GRUB_INSTALL_PLATFORM_X86_64_XEN: grub_util_warn ("%s", _("WARNING: no platform-specific install was performed")); break; -- 2.1.0
A fair bit of time has passed, we are now quite close to the freeze and this series adds a new binary package (hence NEW queue etc) so rather than committing I've pushed to the people/ijc/xen branch. Is it still OK to go ahead? I've posted such a patch upstream and it's in the above branch. http://lists.gnu.org/archive/html/grub-devel/2014-10/msg00041.html Ian.
Please go ahead and push this to master. There's nothing else immediately urgently queued, and I think this would make a pretty compelling case for a freeze exception. If we push it on its own then it should be tractable.