#1069844 grub-efi-arm64: grub fails to load Xen hypervisor on arm64 EFI systems (AVA & QEMU) #1069844
- Package:
- grub-efi-arm64
- Source:
- grub-efi-arm64
- Submitter:
- Alex BennΓ©e
- Date:
- 2024-04-26 16:15:03 UTC
- Severity:
- normal
Attempting to boot Xen on an arm64 EFI system fails. Further attempts to
select another boot entry fails and the system needs to be rebooted
before you can load the kernel without Xen. This is likely due to boot
services having been unloaded.
Running on the Ampere AVA platform with additional diagnostics
(xen,xen_loader,efiemu,fdt,linux) gets the following:
loader/arm64/xen_boot.c:116:xen_loader: Xen Hypervisor cmdline : placeholder
no-real-mode edd=off @ 0x807ff98a6e0 size:33
loader/arm64/xen_boot.c:224:xen_loader: Module @ 0xf283b000 size:0x1f2c75b
loader/arm64/xen_boot.c:139:xen_loader: Module node name module@f283b000
loader/arm64/xen_boot.c:161:xen_loader: Module
loader/arm64/xen_boot.c:183:xen_loader: Module has no bootargs!
loader/arm64/xen_boot.c:224:xen_loader: Module @ 0xef17d000 size:0x3514a00
loader/arm64/xen_boot.c:139:xen_loader: Module node name module@ef17d000
loader/arm64/xen_boot.c:161:xen_loader: Module
loader/arm64/xen_boot.c:172:xen_loader: Module cmdline : placeholder
root=UUID=06e49159-b374-445c-bf5f-2bf93e3f4d6b ro @ 0x807ffa1a3a0 size:62
loader/arm64/xen_boot.c:224:xen_loader: Module @ 0xf4769000 size:0x3514a00
loader/arm64/xen_boot.c:139:xen_loader: Module node name module@f4769000
loader/arm64/xen_boot.c:161:xen_loader: Module
loader/arm64/xen_boot.c:172:xen_loader: Module cmdline : placeholder
root=UUID=06e49159-b374-445c-bf5f-2bf93e3f4d6b ro @ 0x807fd65a280 size:62
loader/efi/fdt.c:136:fdt: Installed/updated FDT configuration table @
0xef160000
error: cannot load image.
So it appears EFI boot services can successfully stat the various bits
and pieces but once it gets to the stage of
grub_arch_efi_linux_boot_image it fails. As debugging grub on the AVA
platform is hard I replicated the setup in QEMU. At which point I
obtained the following backtrace:
Thread 1 hit Breakpoint 6.2, grub_error (n=GRUB_ERR_BAD_OS, fmt=0x23bedbbbd "cannot load image") at ../../../grub-core/kern/err.c:41
41 grub_errno = n;
(grub gdb) bt
#0 grub_error (n=GRUB_ERR_BAD_OS, fmt=0x23bedbbbd "cannot load image") at ../../../grub-core/kern/err.c:41
#1 0x000000023bedabf0 in grub_arch_efi_linux_boot_image (addr=9561964544, size=1081352,
args=0x23bbb8b00 "placeholder dom0_mem=4G,max:4G loglvl=all guest_loglvl=all no-real-mode edd=off") at ../../../grub-core/loader/efi/linux.c:214
#2 0x000000023bff41bc in grub_loader_boot () at ../../../grub-core/commands/boot.c:211
#3 grub_loader_boot () at ../../../grub-core/commands/boot.c:190
#4 0x000000023bf42158 in grub_command_execute (name=0x23bf4e72c "boot", argc=0, argv=0x0 <_start>) at ../../../include/grub/command.h:126
#5 grub_menu_execute_entry (entry=entry@entry=0x23bd17660, auto_boot=auto_boot@entry=0) at ../../../grub-core/normal/menu.c:306
#6 0x000000023bf41e2c in show_menu (autobooted=<optimized out>, nested=<optimized out>, menu=<optimized out>) at ../../../grub-core/normal/menu.c:925
#7 grub_show_menu (menu=menu@entry=0x23bd1a940, nested=nested@entry=1, autoboot=autoboot@entry=0) at ../../../grub-core/normal/menu.c:940
#8 0x000000023bf408a8 in grub_normal_execute (config=<optimized out>, nested=nested@entry=1, batch=batch@entry=0) at ../../../grub-core/normal/main.c:291
#9 0x000000023bf32260 in grub_cmd_source (cmd=<optimized out>, argc=1, args=0x23bd1fcc8) at ../../../grub-core/commands/configfile.c:48
#10 grub_cmd_source (cmd=<optimized out>, argc=<optimized out>, args=0x23bd1fcc8) at ../../../grub-core/commands/configfile.c:30
#11 0x000000023bf48d0c in grub_script_execute_cmdline (cmd=<optimized out>) at ../../../grub-core/script/execute.c:1034
#12 0x000000023bf478c0 in grub_script_execute_cmd (cmd=cmd@entry=0x23bd190c8) at ../../../grub-core/script/execute.c:819
#13 0x000000023bf4874c in grub_script_execute_cmdlist (list=<optimized out>) at ../../../grub-core/script/execute.c:1079
#14 0x000000023bf478c0 in grub_script_execute_cmd (cmd=<optimized out>) at ../../../grub-core/script/execute.c:819
#15 0x000000023bf489b4 in grub_script_execute (script=<optimized out>) at ../../../grub-core/script/execute.c:1191
#16 0x000000023bf497fc in grub_normal_parse_line (line=line@entry=0x23bd20060 "configfile $prefix/grub.cfg", getline=getline@entry=0x23bf40430 <read_config_file_getline>,
getline_data=getline_data@entry=0x23bd20380) at ../../../grub-core/script/main.c:36
#17 0x000000023bf409a0 in read_config_file (config=0x23bd20780 "(hd0,gpt1)/EFI/debian/grub.cfg") at ../../../grub-core/normal/main.c:179
#18 grub_normal_execute (config=config@entry=0x23bd20780 "(hd0,gpt1)/EFI/debian/grub.cfg", nested=nested@entry=0, batch=batch@entry=0)
at ../../../grub-core/normal/main.c:277
#19 0x000000023bf40ca4 in grub_enter_normal_mode (config=config@entry=0x23bd20780 "(hd0,gpt1)/EFI/debian/grub.cfg") at ../../../grub-core/normal/main.c:304
#20 0x000000023bf40da0 in grub_try_normal_prefix (prefix=0x23bd209a0 "(hd0,gpt1)/EFI/debian") at ../../../grub-core/normal/main.c:356
#21 0x000000023bf40ea0 in grub_try_normal (variable=0x23bf4e492 "fw_path") at ../../../grub-core/normal/main.c:407
#22 grub_cmd_normal (cmd=<optimized out>, argc=0, argv=<optimized out>) at ../../../grub-core/normal/main.c:421
#23 grub_cmd_normal (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at ../../../grub-core/normal/main.c:412
#24 0x000000023c018fb8 in grub_command_execute (name=0x23c01ec6e ")", argc=0, argv=0x0 <_start>) at ../../../include/grub/command.h:126
#25 grub_load_normal_mode () at ../../../grub-core/kern/main.c:247
#26 grub_main () at ../../../grub-core/kern/main.c:339
#27 0x000000023c5c02c8 in ?? ()
#28 0x000000023c62a000 in ?? ()
#29 0xafafafaf6c617470 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
While there are different firmwares (QEMU is EDKII targeting -M virt) it
looks like the same error. However before writing off the firmware I did
build the upstream grub:
β git describe
grub-2.12-17-g8719cc204
π17:25:21 alex@gwenyn:grub.git on ξ master [?]
β git show HEAD
commit 8719cc2040368d43ab2de0b6e1b850b2c9cfc5b7 (HEAD -> master, origin/master, origin/HEAD)
Author: Daniel Kiper <daniel.kiper@oracle.com>
Date: Tue Apr 9 19:56:02 2024 +0200
windows: Add _stack_chk_guard/_stack_chk_fail symbols for Windows 64-bit target
Otherwise the GRUB cannot start due to missing symbols when stack
protector is enabled on EFI platforms.
Signed-off-by: Daniel Kiper <daniel.kiper@oracle.com>
Reviewed-by: Vladimir Serbinenko <phcoder@gmail.com>
And installed it in parallel with the distro grub. This was able to
start Xen using the same grub.cfg and get most of the way through the
Dom0 boot before that failed for unrelated issues. So it seems there is
a bug introduced by the debian customisation of the package or missing a
fix from the current state of upstream.
Continuing to debug on QEMU it seems there is an incompatibility with
the images and the peloader (which overrides the normal efi loader):
Thread 1 hit Breakpoint 3.2, grub_load_normal_mode () at ../../../grub-core/kern/main.c:241
241 in ../../../grub-core/kern/main.c
(grub gdb) hbreak do_load_image
Hardware assisted breakpoint 4 at 0x23bdf0e00: do_load_image. (2 locations)
(grub gdb) c
Continuing.
add symbol table from file "bli.module" at
.text_addr = 0x23ba772e0
.bss_addr = 0x0
.module_license_addr = 0x23ba7764a
.data_addr = 0x0
.rodata.str1.1_addr = 0x23ba77560
.rodata_addr = 0x23ba77550
add symbol table from file "xen_boot.module" at
.text_addr = 0x23bcef3c0
.bss_addr = 0x23bcf0370
.module_license_addr = 0x23bcf035e
.data_addr = 0x0
.rodata.str1.1_addr = 0x23bcefff8
Thread 1 hit Breakpoint 4.1, do_load_image (boot_policy=0 '\000', parent_image_handle=0x23e889f18, file_path=0x237d1bce0, source_buffer=0x239f00000, source_size=1081352,
image_handle=0x4766c498) at ../../../grub-core/loader/efi/peimage.c:745
warning: 745 ../../../grub-core/loader/efi/peimage.c: No such file or directory
(grub gdb) hbreak grub_error
Hardware assisted breakpoint 5 at 0x6db0: grub_error. (2 locations)
(grub gdb) c
Continuing.
Thread 1 hit Breakpoint 4.2, 0x000000023bdf0e4c in do_load_image (boot_policy=<optimized out>, parent_image_handle=<optimized out>, image_handle=<optimized out>,
source_size=<optimized out>, source_buffer=<optimized out>, file_path=<optimized out>) at ../../../grub-core/loader/efi/peimage.c:751
751 in ../../../grub-core/loader/efi/peimage.c
(grub gdb) c
Continuing.
Thread 1 hit Breakpoint 5.2, grub_error (n=GRUB_ERR_BAD_OS, fmt=0x23bdf1703 "section inside header") at ../../../grub-core/kern/err.c:41
warning: 41 ../../../grub-core/kern/err.c: No such file or directory
(grub gdb) bt
#0 grub_error (n=GRUB_ERR_BAD_OS, fmt=0x23bdf1703 "section inside header") at ../../../grub-core/kern/err.c:41
#1 0x000000023bdf0e34 in do_load_image (boot_policy=<optimized out>, parent_image_handle=<optimized out>, file_path=<optimized out>, source_buffer=<optimized out>,
source_size=<optimized out>, image_handle=<optimized out>) at ../../../grub-core/loader/efi/peimage.c:747
#2 0x000000023bedabdc in grub_arch_efi_linux_boot_image (addr=9561964544, size=1081352,
args=0x23bbb8b00 "placeholder dom0_mem=4G,max:4G loglvl=all guest_loglvl=all no-real-mode edd=off") at ../../../grub-core/loader/efi/linux.c:210
#3 0x000000023bff41bc in grub_loader_boot () at ../../../grub-core/commands/boot.c:211
#4 grub_loader_boot () at ../../../grub-core/commands/boot.c:190
#5 0x000000023bf42158 in grub_command_execute (name=0x23bf4e72c "boot", argc=0, argv=0x0 <_start>) at ../../../include/grub/command.h:126
#6 grub_menu_execute_entry (entry=entry@entry=0x23bd17660, auto_boot=auto_boot@entry=0) at ../../../grub-core/normal/menu.c:306
#7 0x000000023bf41e2c in show_menu (autobooted=<optimized out>, nested=<optimized out>, menu=<optimized out>) at ../../../grub-core/normal/menu.c:925
#8 grub_show_menu (menu=menu@entry=0x23bd1a940, nested=nested@entry=1, autoboot=autoboot@entry=0) at ../../../grub-core/normal/menu.c:940
#9 0x000000023bf408a8 in grub_normal_execute (config=<optimized out>, nested=nested@entry=1, batch=batch@entry=0) at ../../../grub-core/normal/main.c:291
#10 0x000000023bf32260 in grub_cmd_source (cmd=<optimized out>, argc=1, args=0x23bd1fcc8) at ../../../grub-core/commands/configfile.c:48
#11 grub_cmd_source (cmd=<optimized out>, argc=<optimized out>, args=0x23bd1fcc8) at ../../../grub-core/commands/configfile.c:30
#12 0x000000023bf48d0c in grub_script_execute_cmdline (cmd=<optimized out>) at ../../../grub-core/script/execute.c:1034
#13 0x000000023bf478c0 in grub_script_execute_cmd (cmd=cmd@entry=0x23bd190c8) at ../../../grub-core/script/execute.c:819
#14 0x000000023bf4874c in grub_script_execute_cmdlist (list=<optimized out>) at ../../../grub-core/script/execute.c:1079
#15 0x000000023bf478c0 in grub_script_execute_cmd (cmd=<optimized out>) at ../../../grub-core/script/execute.c:819
#16 0x000000023bf489b4 in grub_script_execute (script=<optimized out>) at ../../../grub-core/script/execute.c:1191
#17 0x000000023bf497fc in grub_normal_parse_line (line=line@entry=0x23bd20060 "configfile $prefix/grub.cfg", getline=getline@entry=0x23bf40430 <read_config_file_getline>,
getline_data=getline_data@entry=0x23bd20380) at ../../../grub-core/script/main.c:36
#18 0x000000023bf409a0 in read_config_file (config=0x23bd20780 "(hd0,gpt1)/EFI/debian/grub.cfg") at ../../../grub-core/normal/main.c:179
#19 grub_normal_execute (config=config@entry=0x23bd20780 "(hd0,gpt1)/EFI/debian/grub.cfg", nested=nested@entry=0, batch=batch@entry=0)
at ../../../grub-core/normal/main.c:277
#20 0x000000023bf40ca4 in grub_enter_normal_mode (config=config@entry=0x23bd20780 "(hd0,gpt1)/EFI/debian/grub.cfg") at ../../../grub-core/normal/main.c:304
#21 0x000000023bf40da0 in grub_try_normal_prefix (prefix=0x23bd209a0 "(hd0,gpt1)/EFI/debian") at ../../../grub-core/normal/main.c:356
#22 0x000000023bf40ea0 in grub_try_normal (variable=0x23bf4e492 "fw_path") at ../../../grub-core/normal/main.c:407
#23 grub_cmd_normal (cmd=<optimized out>, argc=0, argv=<optimized out>) at ../../../grub-core/normal/main.c:421
#24 grub_cmd_normal (cmd=<optimized out>, argc=<optimized out>, argv=<optimized out>) at ../../../grub-core/normal/main.c:412
#25 0x000000023c018fb8 in grub_command_execute (name=0x23c01ec6e ")", argc=0, argv=0x0 <_start>) at ../../../include/grub/command.h:126
#26 grub_load_normal_mode () at ../../../grub-core/kern/main.c:247
#27 grub_main () at ../../../grub-core/kern/main.c:339
#28 0x000000023c5c02c8 in ?? ()
#29 0x000000023c62a000 in ?? ()
#30 0xafafafaf6c617470 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Is it possible to override the peloader or does the Xen image need to be
prepared a certain way?
Well it needs to be a proper UEFI binary. The loader works for Linux and for chainloading the Windows boot loader, we haven't tested other binaries (except some small test binaries I suppose). Personally, I'm surprised Xen still exists in 2024, so don't expect me to be very helpful here. In the error case you can see though, that one of the section addresses in the Xen binary to be relocated points into the (PE) header of the binary, which obviously seems wrong. So go check your PE sections and check which one is wrong?
Julian Andres Klode <julian.klode@canonical.com> writes: I'm able to load the hypervisor directly (it appears as a Linux kernel ARM64 boot executable Image, little-endian, 4K pages). Also using the upstream grub its able to load the hypervisor with the direct EFI load_image boot service. There are still use cases for a "traditional" lowvisor type hypervisor that isn't entangled with a whole kernel. In our case there are automotive use cases which benefit from Xen's ability to partition and pin vCPUs to different domains. I don't know what Debian's policy is towards Xen but it did at least use to work. Is there any tooling for examining PE sections?
Alex BennΓ©e <alex.bennee@linaro.org> writes:
<snip>
Nothing really jumps out from objdump:
1:08:50 [root@debian-arm64:~] # objdump -h /boot/xen
/boot/xen: file format pei-aarch64-little
Sections:
Idx Name Size VMA LMA File off Algn
0 .reloc 00000000 0000000000000000 0000000000000000 00000000 2**0
ALLOC, LOAD, READONLY, DATA
1 .text 00107ea8 0000000000000160 0000000000000160 00000160 2**4
CONTENTS, ALLOC, LOAD, CODE
21:08:53 [root@debian-arm64:~] # objdump -h /boot/vmlinuz
/boot/vmlinuz: file format pei-aarch64-little
Sections:
Idx Name Size VMA LMA File off Algn
0 .text 018c0000 0000000000010000 0000000000010000 00010000 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .data 0090a200 00000000018d0000 00000000018d0000 018d0000 2**2
CONTENTS, ALLOC, LOAD, DATA
This looks suspicious. Yes it's 0 size but it's address is 0 which clearly points into the header, and we don't skip 0 size sections when loading the PE binary for later relocation, and we don't use any .reloc section. I suppose the header is smaller than 0x160 bytes and this is ok. My colleague Heinrich has written a nice PE analyser tool too: https://github.com/xypron/efi_analyzer
On Fri, 26 Apr 2024 at 16:48, Julian Andres Klode < julian.klode@canonical.com> wrote:
It does get skipped in the reloc code:
if (!info->reloc || !(info->reloc->size))
{
grub_dprintf ("linux", "no relocations\n");
return GRUB_EFI_SUCCESS;
}
Although the x86_64 build of Xen uses reloc sections (which Xen itself
deals with) we don't actually need them for Arm. However the way the PE
binary is built is a little funky:
make
UPD include/xen/compile.h
Xen 4.19-unstable
make[1]: Nothing to be done for 'include'.
make[1]: 'arch/arm/include/asm/asm-offsets.h' is up to date.
CC common/version.o
LD common/built_in.o
CC arch/arm/acpi/domain_build.o
INIT_O arch/arm/acpi/domain_build.init.o
LD arch/arm/acpi/built_in.o
CC arch/arm/domain_build.o
INIT_O arch/arm/domain_build.init.o
CC arch/arm/efi/boot.o
INIT_O arch/arm/efi/boot.init.o
LD arch/arm/efi/built_in.o
CC arch/arm/setup.o
LD arch/arm/built_in.o
LD prelink.o
ld -EL --no-warn-rwx-segments --fix-cortex-a53-843419 -T
arch/arm/xen.lds -N prelink.o \
./common/symbols-dummy.o -o ./.xen-syms.0
nm -pa --format=sysv ./.xen-syms.0 \
| ./tools/symbols --sysv --sort \
> ./.xen-syms.0.S
make -f ./Rules.mk obj=. ./.xen-syms.0.o
CC .xen-syms.0.o
ld -EL --no-warn-rwx-segments --fix-cortex-a53-843419 -T
arch/arm/xen.lds -N prelink.o \
./.xen-syms.0.o -o ./.xen-syms.1
nm -pa --format=sysv ./.xen-syms.1 \
| ./tools/symbols --sysv --sort \
> ./.xen-syms.1.S
make -f ./Rules.mk obj=. ./.xen-syms.1.o
CC .xen-syms.1.o
ld -EL --no-warn-rwx-segments --fix-cortex-a53-843419 -T
arch/arm/xen.lds -N prelink.o --build-id=sha1 \
./.xen-syms.1.o -o xen-syms
nm -pa --format=sysv xen-syms \
| ./tools/symbols --all-symbols --xensyms --sysv --sort \
> xen-syms.map
rm -f ./.xen-syms.[0-9]*
objcopy -O binary -S xen-syms xen
ln -sf xen xen.efi
I can't work out how to build without the reloc section and trying to
remove the .reloc section after the fact results in something where the
file-offset is ahead of the VMA position:
β objcopy --remove-section=.reloc -O binary -S xen-syms xen-test
π17:03:23 alex@gwenyn:xen.git/xen on ξ testing/new-attempt:new-attempt
[!?]
β objdump -h xen-test
xen-test: file format pei-aarch64-little
Sections:
Idx Name Size VMA LMA File off
Algn
0 .reloc 00000000 0000000000000000 0000000000000000 00000000
2**0
ALLOC, LOAD, READONLY, DATA
1 .text 00127ea8 0000000000000160 0000000000000160 00000160
2**4
CONTENTS, ALLOC, LOAD, CODE
π17:03:25 alex@gwenyn:xen.git/xen on ξ testing/new-attempt:new-attempt
[!?]
β objcopy --remove-section=.reloc xen-test
π17:03:37 alex@gwenyn:xen.git/xen on ξ testing/new-attempt:new-attempt
[!?]
β objdump -h xen-test
xen-test: file format pei-aarch64-little
Sections:
Idx Name Size VMA LMA File off
Algn
0 .text 00127ea8 0000000000000160 0000000000000160 000001b0
2**2
CONTENTS, ALLOC, LOAD, CODE
That seems pretty happy with the binary:
/home/alex/lsrc/xen.git/xen/xen.efi
Offset to PE: 0x40
Machine type: 0xaa64, ARM64 little endian
NumberOfSymbols should be 0.
Characteristics: 0x0206
* The file is executable.
* COFF line numbers were stripped from the file.
* Debugging information was removed.
LinkerVersion 2.20
BaseOfCode: 0x160
AddressOfEntryPoint: 0xe8c20
Image type: PE32+
Subsystem: EFI application
DLL Characteristics: 0x0000
ImageBase: 0x0
SectionAlignment: 0x1000
FileAlignment: 0x8
SizeOfImage: 0x175000
.reloc.address: 0x0
.reloc.size: 0x0
Number data tables: 6
Exports : 0x00000000 - 0x00000000
Imports : 0x00000000 - 0x00000000
Resources : 0x00000000 - 0x00000000
Exceptions : 0x00000000 - 0x00000000
Certificates : 0x00000000 - 0x00000000
Base Relocations : 0x00000000 - 0x00000000
Number of sections: 2
Section[0]: .reloc
Virtual size: 0x0
Virtual address: 0x0
Size of raw data: 0x0
Pointer to raw data: 0x0
End of raw data: 0x0
Characteristics: 0x42000040
* The section contains initialized data.
* The section can be discarded as needed.
* The section can be read.
* Align data on a 1 byte boundary.
Section[1]: .text
Virtual size: 0x174ea0
Virtual address: 0x160
Size of raw data: 0x127ea8
Pointer to raw data: 0x160
End of raw data: 0x128008
Characteristics: 0xe0000020
* The section contains executable code.
* The section can be executed as code.
* The section can be read.
* The section can be written to.
* Align data on a 16 byte boundary.
I'm going to continue to see if I can improve the way Xen builds its EFI
blob but I think the validation code should just skip zero length sections.