- Package:
- cdimage.debian.org
- Source:
- cdimage.debian.org
- Submitter:
- Jack Truong
- Date:
- 2015-04-01 17:57:11 UTC
- Severity:
- normal
Apologies if this is the wrong package. I'm using the jessie rc1 amd64 mini.iso and the EFI partition doesn't seem to have anything in it. It should have efi/boot/boot<arch>.efi for EFI firmware to load properly. It also doesn't seem to exist in the i386 image either. I'm using a Minnowboard MAX to test the image via USB booting.
Hi Jack, and thanks for your report. Jack Truong <jack.truong@uwaterloo.ca> (2015-01-26): This might be, but don't worry, we'll sort it out. mini.iso is produced during the build of the debian-installer package, and later installed on mirrors in e.g.: http://ftp.fr.debian.org/debian/dists/testing/main/installer-amd64/20141002/images/netboot/ http://ftp.fr.debian.org/debian/dists/testing/main/installer-amd64/20150107/images/netboot/ I've just grabbed mini.iso in both (respectively used for Beta 2 and RC 1), and both seem to have efi bits inside (at least according to a loop mount). I don't know much about EFI Partition etc. therefore I'm not sure how they're related to an ISO9660 image. :/ It'd be nice to start by telling us the full URL to the mini.iso you downloaded. Mraw, KiBi.
Not speaking of boot<arch>.efi but -- there was no useful grub.cfg on
the x86 mini.iso until the 20150107 release (see #762618). So 20150107
is certainly necessary for a useful EFI boot from mini.iso and
everything was present and correct when I was looking at #762618.
It's possible I suppose that the lack of grub.cfg might look a bit like
a lack of boot<arch>.efi under some circumstances.
(Feel free to skip this bit if you don't care about the details!)
The EFI System Partition (ESP) is on the ISO as /boot/grub/efi.img,
which is a file with a suitable FAT partition for EFI to read (it's also
referenced by various ISO headers, el-torito/*mumble* etc).
AFAICT the beta release (20141002) had everything needed apart from the
grub.cfg:
$ wget http://ftp.fr.debian.org/debian/dists/testing/main/installer-amd64/20141002/images/netboot/mini.iso
$ isoinfo -i mini.iso -RJ -x /boot/grub/efi.img > efi.img
$ mdir -i efi.img ::efi/boot
Volume in drive : has no label
Volume Serial Number is FF65-0EDB
Directory for ::/efi/boot
. <DIR> 2014-09-27 0:08
.. <DIR> 2014-09-27 0:08
bootx64 efi 392192 2014-09-27 0:08
3 files 392 192 bytes
10 240 bytes free
$
The remainder of the grub bits are then on the ISO (outside the ESP)
in /boot/grub/..., except as noted, that the grub.cfg in 20141002 was
empty.
$ isoinfo -i mini.iso -fRJ | grep /boot/grub
/boot/grub
/boot/grub/efi.img
/boot/grub/font.pf2
/boot/grub/grub.cfg
/boot/grub/x86_64-efi
/boot/grub/x86_64-efi/acpi.mod
[...lots more...]
Yes, please.
Ian.
Hi,
Summary:
There are two EFI System Partitions in mini.iso.
The one for booting from USB stick or qemu -hdb is indeed empty.
One could try whether it helps to copy the files from the FAT
image /boot/grub/efi.img in the ISO to MBR partition 2:
- Put mini.iso onto a USB stick /dev/sdb or other hard disk device
dd if=mini.iso of=/dev/sdb
- Mount the ISO image
mount -o loop mini.iso /mnt/iso
- Mount the FAT filesystem inside /mnt/iso/boot/grub/efi.img
mount -o loop /mnt/iso/boot/grub/efi.img /mnt/fat_iso
- Mount the FAT filesystem in MBR partition 2 of the USB stick
mount /dev/sdb2 /mnt/fat
- Copy from efi.img into partition 2 of the USB stick:
cp -a /mnt/fat_iso/* /mnt/fat
- Check for sucessful copy:
find /mnt/fat
should yield
/mnt/fat
/mnt/fat/efi
/mnt/fat/efi/boot
/mnt/fat/efi/boot/bootx64.efi
- Unmount what you mounted:
umount /mnt/fat_iso /mnt/fat /mnt/iso
- Try whether the stick boots via EFI.
---------------------------------------------------------------
Details:
Cyril Brulebois wrote:
mini.iso and debian-7*-amd64-netinst.iso differ in this aspect.
In Debian amd64 netinst, the EFI System Partition is a data file
inside the ISO filesystem which contains a FAT filesystem image.
In this FAT image there are the files which EFI will use for booting.
Debian has GRUB2 boot loader equipment in there.
The block address and size of the EFI partition is published by
an entry in the El Torito boot catalog, by a partition table entry
in the MBR, by a partition table entry in GPT, and by a (probably
useless) entry in Apple Partition Map.
Inspecting mini.iso
http://ftp.fr.debian.org/debian/dists/testing/main/installer-amd64/20150107/images/netboot/mini.iso
by xorriso-1.3.8 yields:
Volume id : 'ISOIMAGE'
El Torito catalog : 59 1
El Torito cat path : /boot.cat
El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
El Torito boot img : 1 BIOS y none 0x0000 0x00 4 8959
El Torito boot img : 2 UEFI y none 0x0000 0x00 832 9205
El Torito img path : 1 /isolinux.bin
El Torito img opts : 1 boot-info-table isohybrid-suitable
El Torito img path : 2 /boot/grub/efi.img
System area options: 0x00000302
System area summary: MBR isohybrid cyl-align-all
ISO image size/512 : 44332
Partition offset : 0
MBR heads per cyl : 64
MBR secs per head : 32
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x80 0x17 0 45056
MBR partition : 2 0x00 0x01 45056 12288
So we have no GPT and no (probably useless) APM here.
There are two EFI partitions exposed.
One is the data file /boot/grub/efi.img of the ISO filesystem
pointed to by the El Torito boot catalog.
The other is an appended blob, marked by MBR partition 2.
(The partition type 0x01 is questionable.)
One can extract them by:
mount -o loop mini.iso /mnt/iso
cp /mnt/iso/boot/grub/efi.img eltorito_efi.img
dd if=mini.iso bs=512 skip=45056 count=12288 of=mbr_efi.img
Both files differ heavily in size
-r--r--r-- 1 thomas thomas 425984 2015-01-27 09:59 eltorito_efi.img
-rw-r--r-- 1 thomas thomas 6291456 2015-01-27 10:00 mbr_efi.img
Inspecting both:
$ mount -o loop eltorito_efi.img /mnt/fat
$ find /mnt/fat
/mnt/fat
/mnt/fat/efi
/mnt/fat/efi/boot
/mnt/fat/efi/boot/bootx64.efi
$ umount /mnt/fat
$ mount -o loop mbr_efi.img /mnt/fat
$ find /mnt/fat
/mnt/fat
$
So the El Torito advertised partition contains files,
whereas the MBR advertised partition has an empty filesystem.
Probably mini.iso will boot via EFI from CD/DVD/BD resp. qemu -cdrom,
but not from USB stick resp. qemu -hdb.
The presence of an isohybrid MBR will allow booting via BIOS emulation.
---------------------------------------------------------------
For comparison the boot anatomy of a amd64 netinst ISO:
$ xorriso-1.3.8 -indev debian-7.7.0-amd64-netinst.iso \
-report_el_torito plain -report_system_area plain
...
Volume id : 'Debian 7.7.0 amd64 1'
El Torito catalog : 847 1
El Torito cat path : /isolinux/boot.cat
El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
El Torito boot img : 1 BIOS y none 0x0000 0x00 4 848
El Torito boot img : 2 UEFI y none 0x0000 0x00 896 860
El Torito img path : 1 /isolinux/isolinux.bin
El Torito img opts : 1 boot-info-table isohybrid-suitable
El Torito img path : 2 /boot/grub/efi.img
System area options: 0x00000102
System area summary: MBR isohybrid cyl-align-on GPT APM
ISO image size/512 : 454656
Partition offset : 0
MBR heads per cyl : 64
MBR secs per head : 32
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x80 0x00 0 454656
MBR partition : 2 0x00 0xef 3440 896
MBR partition path : 2 /boot/grub/efi.img
GPT : N Info
GPT disk GUID : a0093564c5a7464b90530645ae2a43b6
GPT entry array : 12 208 overlapping
GPT lba range : 64 454602 454655
GPT partition name : 1 490053004f00480079006200720069006400
GPT partname local : 1 ISOHybrid
GPT partition GUID : 1 a0093564c5a7464b90510645ae2a43b6
GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 1 0x1000000000000001
GPT start and size : 1 0 454600
GPT partition name : 2 490053004f004800790062007200690064003100
GPT partname local : 2 ISOHybrid1
GPT partition GUID : 2 a0093564c5a7464b90500645ae2a43b6
GPT type GUID : 2 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 2 0x1000000000000001
GPT start and size : 2 3440 896
GPT partition path : 2 /boot/grub/efi.img
APM : N Info
APM block size : 2048
APM gap fillers : 0
APM partition name : 1 EFI
APM partition type : 1 Apple_HFS
APM start and size : 1 860 224
APM partition path : 1 /boot/grub/efi.img
The file /boot/grub/efi.img contains similar stuff as
the El Torito EFI partition of mini.iso :
$ mount -o loop debian-7.7.0-amd64-netinst.iso /mnt/iso
$ mount -o loop /mnt/iso/boot/grub/efi.img /mnt/fat
$ find /mnt/fat
/mnt/fat
/mnt/fat/efi
/mnt/fat/efi/boot
/mnt/fat/efi/boot/bootx64.efi
Note that all boot entrys for EFI point to the same file
/boot/grub/efi.img inside the ISO filesystem.
Partition editors hate this structure, though, because it
represents overlapping (nested) partitions.
An alternative is the product of GRUB2 script grub-mkrescue.
Its partitions do not overlap (at the price that none of
them represents a mountable ISO filesystem).
Have a nice day :)
Thomas
Hi all, The ISO in question was in fact the same one mentioned (http://ftp.fr.debian.org/debian/dists/testing/main/installer-amd64/20150107/images/netboot/mini.iso). I have copied the contents of efi.img to the FAT partition and it does boot on my EFI firmware now. Thank you!
Hi,
Jack Truong wrote:
So your EFI indeed hopped on that partition.
Possibly only because it sees no other indication where to start
the boot process.
I now wonder whether the MBR partition 2 is actually intended
as EFI System Partition.
The partition type 0x01 generally indicates a FAT12 filesystem.
An EFI partition should have type 0xef.
Regrettably there is no file /.disk/mkisofs in the ISO which
tells the used xorriso -as mkisofs options.
I guess that /boot/grub/efi.img got marked by an option like
-e boot/grub/efi.img
To mark this file in MBR and GPT, one may add option
-isohybrid-gpt-basdat
directly after -e and its argument.
I tested by extracting the MBR and partition 2 from mini.iso,
mounting it, and repacking a new mini_with_gpt.iso by:
$ dd if=mini.iso bs=512 count=1 of=mini_template.mbr
$ dd if=mini.iso bs=512 skip=45056 count=12288 of=mini_partition2.img
$ mount -o loop mini.iso /mnt/iso
$ xorriso -as mkisofs -R \
-o mini_with_gpt.iso \
-isohybrid-mbr mini_template.mbr \
-c '/boot.cat' \
-b '/isolinux.bin' \
-no-emul-boot -boot-load-size 4 -boot-info-table \
-eltorito-alt-boot \
-e '/boot/grub/efi.img' \
-isohybrid-gpt-basdat -no-emul-boot \
-append_partition 3 0x01 mini_partition2.img \
/mnt/iso
This yields:
Volume id : 'ISOIMAGE'
El Torito catalog : 45 1
El Torito cat path : /boot.catalog
El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
El Torito boot img : 1 BIOS y none 0x0000 0x00 4 46
El Torito boot img : 2 UEFI y none 0x0000 0x00 832 66
El Torito img path : 1 /isolinux.bin
El Torito img opts : 1 boot-info-table isohybrid-suitable
El Torito img path : 2 /boot/grub/efi.img
System area options: 0x00000102
System area summary: MBR isohybrid cyl-align-on GPT
ISO image size/512 : 45056
Partition offset : 0
MBR heads per cyl : 64
MBR secs per head : 32
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x80 0x00 0 45056
MBR partition : 2 0x00 0xef 264 832
MBR partition : 3 0x00 0x01 45056 12288
MBR partition path : 2 /boot/grub/efi.img
GPT : N Info
GPT disk GUID : 61bf3bd0595c364488002bde6f8fecfb
GPT entry array : 2 248 overlapping
GPT lba range : 64 44992 45055
GPT partition name : 1 490053004f00480079006200720069006400
GPT partname local : 1 ISOHybrid
GPT partition GUID : 1 61bf3bd0595c364488022bde6f8fecfb
GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 1 0x1000000000000001
GPT start and size : 1 0 44992
GPT partition name : 2 490053004f004800790062007200690064003100
GPT partname local : 2 ISOHybrid1
GPT partition GUID : 2 61bf3bd0595c364488032bde6f8fecfb
GPT type GUID : 2 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 2 0x1000000000000001
GPT start and size : 2 264 832
GPT partition path : 2 /boot/grub/efi.img
Now the empty FAT12 filesystem is in MBR partition 3.
It is unclear whether it is usiable on a system that knows about GPT.
(Maybe i should enhance -append_partition so that it publishes
the partition in GPT too.)
Of course, if the empty 6 MB partition is undesired in a 22.5 MB
ISO, then one can omit option
-append_partition 3 0x01 mini_partition2.img \
from above xorriso run.
If one wants to eradicate GPT and only keep MBR:
$ cp mini_with_gpt.iso mini_with_mbr_efi.iso
$ dd if=/dev/zero bs=512 count=1 seek=1 conv=notrunc of=mini_with_mbr_efi.iso
$ dd if=/dev/zero bs=512 count=1 seek=45055 conv=notrunc of=mini_with_mbr_efi.iso
(The backup GPT header is the last 512 block of the ISO
filesystem range. Probably the appended partition data
prevent it from being recognized at all.)
Have a nice day :)
Thomas
On Tue, 2015-01-27 at 15:57 +0100, Thomas Schmitt wrote: [...] Thanks for all the excellent information, I still have fully absorbed it all... http://anonscm.debian.org/cgit/d-i/debian-installer.git/tree/build/config/x86.cfg#n335 and has: xorriso -as mkisofs -r -J -b isolinux.bin -c boot.cat \ -no-emul-boot -boot-load-size 4 -boot-info-table \ -eltorito-alt-boot \ --efi-boot boot/grub/efi.img -no-emul-boot \ -o $(TEMP_MINIISO) $(TEMP_CD_TREE); \ I guess -e is short for --efi-boot? I don't suppose I can tempt you into sending a patch against the above git repo, can I? ;-) Ian.
Hi, But this does not explain how the empty FAT partition got to the end of mini.iso I see in http://anonscm.debian.org/cgit/d-i/debian-installer.git/tree/build/config/x86.cfg#n347 geniso_hybrid_plus_firmware_partition $(TEMP_MINIISO) So obviously the appended partition was not intended for EFI but for additional firmware. Now if mini.iso gets GPT, then http://anonscm.debian.org/cgit/d-i/debian-installer.git/tree/build/util/geniso_hybrid_plus_firmware_partition should add its partition to GPT too. At its end, it uses fdisk to manipulate the MBR partition table. Number 2 would have to become 3 then, because slot 2 is already occupied by /boot/grub/efi.img. I guess that gdisk can do similar for GPT. Problem is that the GPT backup block needs to be moved after the end of the partition data which was appended in line 37: cat "$firmware_volume_file" >> "$iso" I dimly remember that gdisk has a menu item for moving the backup GPT and/or for re-creating the backup GPT. The backup GPT would actually have to be re-located whenever the ISO image gets copied onto a disk device. It might be that gdisk or other partition editors go on strike when they see the nested partitions. In this case, we might need to write MBR and GPT slots on byte level. (A re-write to geniso_hybrid_plus_firmware_partition.c would be a good idea then. Byte manipulation in shell is cumbersome.) It is a bad idea to apply the SYSLINUX program "isohybrid" to an ISO which was equipped with advanced isohybrid by xorriso. So one should delete line 26: isohybrid -h "$heads" -s "$sectors" "$iso" Seems we are not done with development yet. There is the problem that i cannot easily test because i have no EFI and not even a contemporary Debian. If an authorized developer can set up a suitable environment for production of mini.iso, i will help to adapt the scripts. We will have to make decisions. E.g. whether to keep a BIOS-only version of mini.iso and thus cannot hardcode the partition number in geniso_hybrid_plus_firmware_partition. Some experience with gdisk would be helpful. (I guess my Debian 6 test system can give me hints.)--------------------------------------------------------- Alternatively one could really copy the directories and boot program from /boot/grub/efi.img into the appended partition and give it the type 0xef. Having no GPT would avoid problems with the appended partition. But it might be that some EFI implementations will not recognize the MBR partition and insist in GPT. Only tests can tell. Have a nice day :) Thomas
Hi, i just realize that the xorriso run in http://anonscm.debian.org/cgit/d-i/debian-installer.git/tree/build/config/x86.cfg does not produce an isohybrid image. Therefore one currently cannot simply omit the "isohybrid" run in geniso_hybrid_plus_firmware_partition. Option -isohybrid-mbr is a precondition for -isohybrid-gpt-basdat which marks the EFI boot image in MBR and GPT. It needs the disk path to the file isohdpfx.bin in the local SYSLINUX installation (from where file isolinux.bin stems). debian-cd for debian-7*-amd64.iso does: -isohybrid-mbr syslinux/usr/lib/syslinux/isohdpfx.bin (Don't ask me where its pwd is at that moment.)----------------------------------------------------------- So the small improvement without GPT becomes more attractive:-------------- Add an argument to util/geniso_hybrid_plus_firmware_partition + efi_fat="$2" and in the case [ "$(GRUB_EFI)" = y ] hand it down from build/config/x86.cfg : if [ "$(GRUB_EFI)" = y ]; then \ xorriso -as mkisofs ... ; \ + geniso_hybrid_plus_firmware_partition $(TEMP_MINIISO) \ $(TEMP_CD_TREE)/boot/grub/efi.img ; \ else \ xorriso -as mkisofs ... ; \ + geniso_hybrid_plus_firmware_partition $(TEMP_MINIISO) "" ; \ fi - geniso_hybrid_plus_firmware_partition $(TEMP_MINIISO) (Why those backslashes and semicolons in config/x86.cfg, btw ?)-------------- If "$efi_fat" is not empty, (m)copy the directories and files from "$efi_fat" to "$firmware_volume_name" after that one was created by mkfs.msdos. (mtools expert wanted.)-------------- In the fdisk run at the end of util/geniso_hybrid_plus_firmware_partition - # Pedantically, set partition type to 1: FAT 16 + # Set partition type to 0xef: FAT for EFI echo t echo 2 - echo 1 + echo 239----------------------------------------------------------- Now, who wants to set up a debian-installer, make the proposed code changes, and produce a mini.iso which i can inspect ? Have a nice day :) Thomas
Hi, it might be more simple to just append efi.img as additional partition rather than unpacking it into the partition for custom firmware.-------------- The first step with the new argument efi_fat="$1" in util/geniso_hybrid_plus_firmware_partition and the two alternative calls of geniso_hybrid_plus_firmware_partition from build/config/x86.cfg stays as is. The other two steps get replaced by:-------------- If "$efi_fat" is not empty, append it to the image and mark it in MBR. Depending on the fact that the firmware FAT is aligned to full MiB and isohybrid padded up the ISO to full MiB, one could quite duplicate the code of util/geniso_hybrid_plus_firmware_partition from line 37 to end: # Done! echo w ) | fdisk -C "$cylinders" -H "$heads" -S "$sectors" "$iso" + fdisk_ret="$?" + test "$fdisk_ret" = 0 || exit "$fdisk_ret" + + test -z "$efi_fat" && exit 0 + + cat "$efi_fat" >> "$iso" + + # Until the bug is fixed: Need no new exaggerated cylinder count + + ( + + # Make new partition #3 + echo n + echo p + echo 3 + echo # use default start sector + echo # use default end sector + + # Set partition type to 0xef : FAT for EFI + echo t + echo 3 + echo 239 + + # Done! + echo w + ) | fdisk -C "$cylinders" -H "$heads" -S "$sectors" "$iso"---------- The computation of cylinders seems wrong in util/geniso_hybrid_plus_firmware_partition: cylinders="$(($(stat -c %s "$iso") / $heads / $sectors))" "stat -c %s" yields the byte count of the ISO image. Cylinder size is $heads * $sectors * 512 bytes. The 512 is missing in above computation. Just try: $ iso=mini.iso $ echo "$(($(stat -c %s "$iso") / 64 / 32))" 14336 That would be 14+ GiB. More realistic is: $ echo "$(($(stat -c %s "$iso") / 64 / 32 / 512))" 28-------------------------------------------------------- About letting xorriso set up isohybrid and add partitions to MBR and GPT: I am having some conceptional problems in libisofs to get appended partitions marked in GPT. The backup GPT of an EFI-enabled isohybrid is supposed to be inside the block range of the ISO filesystem. The appended partitions are not. So they should come after the backup GPT. But a GPT partition has to be located before the backup GPT. Can you tell me more about the use cases for the firmware partition ? Is it necessary that partition editors can enlarge it ? Have a nice day :) Thomas
On Tue, 2015-01-27 at 17:03 +0100, Thomas Schmitt wrote: Thank you (again?) for all the excellent info you've provided here and in your followups. Ah yes, that would seem to make sense. Given that I don't really understand how this additional firmware partition stuff is supposed to fit together (i.e. what impact changing its partition number or adding another VFAT partition etc would have on the code which finds/loads from it) I'm very reluctant to be the one to suggest that we should change things here for Jessie given we've had the beta and are frozen etc. Perhaps we should park this until Stretch? (Kibi, any thoughts on that?) It's not really clear to me who the intended audience of these mini.iso's are: It seems (IMHO only) that mini.iso is mostly useful for d-i developers to dev test the netboot installer images without needing to rebuild the full netinst images (so both legacy and EFI capabilities are useful there). But I think most end users would be better off being directed to the proper netinst images instead, they are more polished and better tested etc (since they share code with the larger ISO images and what have you). At some point it seems like improving mini.iso more for end users would simply be duplicating effort with the CD team. Ian.
Hi, Ian Campbell: I hope this does not mean that enough is said on that topic. I am currently preparing a conclusion post to this bug, where i emphasize that mini.iso is friendly to partition editors (when on USB stick or alike). The amd64-netinst images are crammed with baits for BIOS and EFI. They violate about any specs and best practice about partitioning. So if the firmware partition on hard-disk-like media has its use cases, then it seems worthwhile to develop a layout for EFI bootability which preserves the feature of mini.iso to look like a sanely partitioned device. I agree as long as only the bootability of the ISO is of interest. An end user trying to edit the partitioning of amd64-netinst will probably get a rough ride. I dimly remember that firmware partitions were used to allow the user to combine a free Debian with proprietary drivers. Something like in https://www.debian.org/releases/stable/i386/ch06s04.html.en the firmware issue is, and how much the booting OS of mini.iso can make use of the FAT filesystem and its content. The partition was not added just for fun, i guess. Are there traces inside the booting Debian that it looks for such a partition and reads files from it ? Above document says: "debian-installer only prompts for firmware needed by kernel modules loaded during the installation." and "Any firmware loaded during the installation will be copied automatically to the installed system." ----- Ok, back to my work in libisofs to enable the production of standard-compliant GPT together with isohybrid and appended partitions. When it is done, i will propose three alternatives: MBR based, GPT based, crammed netinst plus firmware partition. Only the MBR based one can be achieved by released xorriso. Have a nice day :) Thomas
Yup. The *netboot* images are produced directly in the debian-installer build, which doesn't have such things.
It's in a directory just above the root of the temporary CD tree - see http://anonscm.debian.org/cgit/debian-cd/debian-cd.git/tree/tools/boot/jessie/boot-x86 for context. Just before that point, we've extracted a copy of the syslinux package(s) needed to get all the files we need from them. ... It's the syntax for mtools to work on the FAT images, that's all.
Hi, i understand that debian-cd mailing list is in charge for discussion of amd64 mini.iso and its generating software: http://anonscm.debian.org/cgit/d-i/debian-installer.git/tree/build/config/x86.cfg http://anonscm.debian.org/cgit/d-i/debian-installer.git/tree/build/util/geniso_hybrid_plus_firmware_partition For the sake of persistence i add my assessment to bug 776317, although it somewhat exceeds the scope of a bug report.------------------------------------------------------------- The bug reporter's EFI firmware mistook the appended firmware partion (an empty FAT filesystem) for the EFI System Partition. The real EFI partition is located inside the ISO filesystem and not advertised by MBR. So the ISO did not boot from USB stick until the file tree from the EFI partition was copied to the firmware partition. Other EFIs might not be so tolerant and insist in MBR partition type 0xef or in a GPT partition in order to find the first piece of GRUB2 software. The very dense partition layout of debian-7*amd64-netinst.iso obviously serves the needs and quirks of the EFIs which were encountered so far. But it makes partition editors scream. The firmware partition is intended for being used on a hard-disk-like device, where the wish for partition editing is natural, because the device is larger than the ISO image. So how can mini.iso become friendly towards partition editors ? I will discuss three layout alternatives: - UEFI compliant Legacy MBR based - UEFI compliant GPT based - Rogue but well tested amd64-netinst with firmware partition My conclusion is that UEFI compliant Legacy MBR is most promising in respect to work effort and ease of use. The production of this layout is essentially sketched in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=776317#45 and improved in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=776317#50 The following paragraph also shows how to achieve this with xorriso to avoid the need for post-processing the ISO image.----------------------------------------------------------- UEFI 2.4, 5.2.1 "Legacy Master Boot Record (MBR)" and 5.2.2 "OS Types" allow to use a pure MBR partition table and to mark the EFI system partition by partition type 0xef. This proposal lets three MBR partitions emerge which cover one ISO and two FAT filesystems. The EFI partition bears the proper type 0xef. It would ease enlarging of the firmware partition if it is number 3 and the EFI partition becomes number 2. It is desirable to have the ISO filesystem in a mountable partition. Because the El Torito boot image efi.img is inside the ISO and thus inside partition 1, this implies the need for an extra copy of the FAT filesystem with file path /efi/boot/bootx64.efi in it. (I will think about El Torito outside of ISO. It is weird but not impossible.) In this case the appended EFI FAT image should be aligned to full MiB, in order to fit into the cylinder ideology of partition editors. The wasted space is not really usable on an MBR partitioned device, because any further partition would begin at a cylinder boundary anyway. If the SYSLINUX MBR template file isohdpfx.bin is in reach then one can omit the run of isohybrid and fdisk and rather let xorriso do the work of partitioning. This works already with xorriso-1.3.2 as of Debian Jessie resp. Debian testing: xorriso -as mkisofs \ -r -J \ -isohybrid-mbr ...maybe.in.$(TEMP_SYSLINUX).../isohdpfx.bin \ -c boot.cat \ -b isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table \ -eltorito-alt-boot \ -e boot/grub/efi.img -no-emul-boot \ -append_partition 2 0xef $(TEMP_CD_TREE)/boot/grub/efi.img \ -append_partition 3 0x01 $(FIRMWARE_VOLUME_FILE) \ -partition_cyl_align all \ -o $(TEMP_MINIISO) $(TEMP_CD_TREE); \ The variable FIRMWARE_VOLUME_FILE needs to be set to the path of the FAT image, which is currently created in util/geniso_hybrid_plus_firmware_partition by mkfs.msdos . The FAT image would have to be created before the xorriso run and removed aftwerwards. Resulting partition layout: System area summary: MBR isohybrid cyl-align-all ISO image size/512 : 45056 Partition offset : 0 MBR heads per cyl : 64 MBR secs per head : 32 MBR partition table: N Status Type Start Blocks MBR partition : 1 0x80 0x17 0 45056 MBR partition : 2 0x00 0xef 45056 2048 MBR partition : 3 0x00 0x01 47104 12288 With some more waste of space by duplicate ISO and Joliet directory trees, one can achieve an even more conventinal layout where the first partition does not include the MBR: xorriso -as mkisofs \ -partition_offset 16 \ ...all options from previous example... yields: Partition offset : 16 ... MBR partition table: N Status Type Start Blocks MBR partition : 1 0x80 0x17 64 44992 MBR partition : 2 0x00 0xef 45056 2048 MBR partition : 3 0x00 0x01 47104 12288 The ISO filesystem can still be mounted via /dev/sdb and via /dev/sdb1, although both differ in block offset. In this case the price of the duplicate file trees was paid from the waste space of cylinder alignment. With less luck, it could also be one more cylinder (= 1 MiB) of image size. In general it depends on the number of files and the length of their names.----------------------------------------------------------- With GPT it becomes more complicated. UEFI 2.4, 5.2.3 "Protective MBR" and table 16 state that a neat GPT partitioning has an MBR marking a single partition of type 0xee which covers the whole device from LBA 1 to the end. So actually one cannot combine MBR and GPT partitioning but rather must decide for one of both. BIOS needs no MBR partition table for its first boot stage. So if the later boot stages, which might need partitions, understand GPT then there should be no problem with a standards compliant GPT. Question is whether all EFIs understand GPT. {:) If not xorriso shall do the partitioning, then one would have to try whether gdisk or other GPT editors can equip an isohybrid image with GPT. Probably it is necessary to clean the partition table slot of MBR partition 1, which was created by the isohybrid program or by xorriso's option -isohybrid-mbr: dd if=/dev/zero bs=1 count=16 seek=446 conv=notrunc of=mini.iso The GPT editor could then use the first 32 kB of the ISO and the padding at its end (by default 300 kB) in order to establish a GPT and a first partition which covers as much of the ISO as possible.i If xorriso shall do the partitioning: The new option of xorriso -as mkisofs -appended_part_as_gpt marks the ISO and appended partitions by a compliant GPT rather than MBR or a non-compliant MBR+GPT mix. The partition types are used only to distinguish EFI system partitions from others. xorriso-1.3.9 -as mkisofs \ -r -J \ -isohybrid-mbr ...maybe.in.$(TEMP_SYSLINUX).../isohdpfx.bin \ -c boot.cat \ -b isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table \ -eltorito-alt-boot \ -e boot/grub/efi.img -no-emul-boot \ -append_partition 2 0xef $(TEMP_CD_TREE)/boot/grub/efi.img \ -append_partition 3 0x01 $(FIRMWARE_VOLUME_FILE) \ -appended_part_as_gpt \ -o $(TEMP_MINIISO) $(TEMP_CD_TREE); \ This yields: System area summary: MBR isohybrid cyl-align-off GPT ISO image size/512 : 45056 Partition offset : 0 MBR heads per cyl : 64 MBR secs per head : 32 MBR partition table: N Status Type Start Blocks MBR partition : 1 0x00 0xee 1 58239 GPT : N Info GPT disk GUID : 647e4984d224e14597e29c8920e7edc9 GPT entry array : 2 248 separated GPT lba range : 64 58176 58239 GPT partition name : 1 4700610070003000 GPT partname local : 1 Gap0 GPT partition GUID : 1 647e4984d224e14597e69c8920e7edc9 GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7 GPT partition flags: 1 0x1000000000000001 GPT partition name : 2 41007000700065006e006400650064003200 GPT partname local : 2 Appended2 GPT partition GUID : 2 647e4984d224e14597e79c8920e7edc9 GPT type GUID : 2 28732ac11ff8d211ba4b00a0c93ec93b GPT partition flags: 2 0x0000000000000000 GPT start and size : 2 45056 832 GPT partition name : 3 41007000700065006e006400650064003300 GPT partname local : 3 Appended3 GPT partition GUID : 3 647e4984d224e14597e49c8920e7edc9 GPT type GUID : 3 a2a0d0ebe5b9334487c068b6b72699c7 GPT partition flags: 3 0x0000000000000000 GPT start and size : 3 45888 12288 Note that the ISO filesystem cannot be represented by a mountable partition here. It begins at LBA 0, whereas a protectice MBR partition must begin at LBA 1. UEFI 2.4, table 16: "StartingLBA [...] Set to 0x00000001" GPT partitions may not begin before the First Usable LBA which is 64 in xorriso ISOs (and cannot be smaller than 34, according to UEFI 2.4, 5.3.1). With -partition_offset 16 the first GPT partition is mountable and automatically gets a name which advertises its filesystem. (One could put the ISO Volume Id here. But i am unsure whether this is appropriate.) The MBR partition stays unmountable. MBR partition table: N Status Type Start Blocks MBR partition : 1 0x00 0xee 1 58239 ... GPT partition name : 1 490053004f003900360036003000 GPT partname local : 1 ISO9660 ... GPT start and size : 1 64 44992----------------------------------------------------------- What if we don't care about neat partition tables ? libisofs-1.3.9 now puts the backup GPT after eventual appended partitions and registers them as GPT partitions if a GPT emerges by other settings. So one can augment the layout of amd64-netinst by a firmware partition: xorriso-1.3.9 -as mkisofs \ -r -J \ -isohybrid-mbr ...maybe.in.$(TEMP_SYSLINUX).../isohdpfx.bin \ -c boot.cat \ -b isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table \ -eltorito-alt-boot \ -e boot/grub/efi.img -no-emul-boot \ -isohybrid-gpt-basdat \ -append_partition 3 0x01 $(FIRMWARE_VOLUME_FILE) \ -o $(TEMP_MINIISO) $(TEMP_CD_TREE); \ Result is this crammed hybrid: System area summary: MBR isohybrid cyl-align-on GPT ISO image size/512 : 45056 Partition offset : 0 MBR heads per cyl : 64 MBR secs per head : 32 MBR partition table: N Status Type Start Blocks MBR partition : 1 0x80 0x00 0 45056 MBR partition : 2 0x00 0xef 344 832 MBR partition : 3 0x00 0x01 45056 12288 MBR partition path : 2 /boot/grub/efi.img GPT : N Info GPT disk GUID : 9b618f4db2eb824eb10f15cd926e3c72 GPT entry array : 2 248 overlapping GPT lba range : 64 57344 57407 GPT partition name : 1 490053004f00480079006200720069006400 GPT partname local : 1 ISOHybrid GPT partition GUID : 1 9b618f4db2eb824eb10c15cd926e3c72 GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7 GPT partition flags: 1 0x1000000000000001 GPT start and size : 1 0 45056 GPT partition name : 2 490053004f004800790062007200690064003100 GPT partname local : 2 ISOHybrid1 GPT partition GUID : 2 9b618f4db2eb824eb10d15cd926e3c72 GPT type GUID : 2 a2a0d0ebe5b9334487c068b6b72699c7 GPT partition flags: 2 0x1000000000000001 GPT start and size : 2 344 832 GPT partition path : 2 /boot/grub/efi.img GPT partition name : 3 41007000700065006e006400650064003300 GPT partname local : 3 Appended3 GPT partition GUID : 3 9b618f4db2eb824eb10e15cd926e3c72 GPT type GUID : 3 a2a0d0ebe5b9334487c068b6b72699c7 GPT partition flags: 3 0x0000000000000000 GPT start and size : 3 45056 12288 List of sins: This mixes the exclusive alternatives given by UEFI 2.4, 5.2: "LBA 0 (i.e. the first logical block) of the hard disk contains either a legacy Master Boot Record (MBR) (see Section 5.2.1) or a protective MBR (see Section 5.2.2 [actually 5.2.3])." It ignores the demand of UEFI 2.4, 5.2.2 for the legacy MBR: "Each [MBR] partition must not overlap with other partitions." and 5.3.1 for GPT: "Each defined [GPT] partition must not overlap with any other defined partition." Further there is a law against start LBA 0 in GPT partition 1: "All data stored on the volume must be stored between the First Usable LBA through Last Usable LBA, and only the data structures defined by UEFI to manage partitions may reside outside of the usable space." The First Usable LBA is 64, the Last Usable LBA is 57280 here. The (non-)specification of ISOLINUX isohybrid GPT by http://git.kernel.org/cgit/boot/syslinux/syslinux.git/tree/utils/isohybrid.c defines the type GUID efi_system_partition but does not use it. It rather marks the EFI boot image by basic_partition, which is Microsoft Basic Data Partition. Since i learned from isohybrid.c and mjg's Fedora-LiveCD.iso, libisofs does it the same way in the isohybrid case. The code for grub-mkrescue style preparations does it right: GPT type GUID : 2 28732ac11ff8d211ba4b00a0c93ec93b The code for isohybrid does not comply to UEFI 2.4, table 19: GPT type GUID : 2 a2a0d0ebe5b9334487c068b6b72699c7 But is it worth to fix this ? isohybrid GPT is so deep in the mud that only its successful tests can justify it. And the tests succeeded with the bug. After all, the xorrisofs option is -isohybrid-gpt-basdat and not something like "-isohybrid-gpt-efisys". End of sin list. Option -partition_offset 16 can be added for more conventional start addresses of the first partitions: Partition offset : 16 ... MBR partition table: N Status Type Start Blocks MBR partition : 1 0x80 0x00 64 44992 MBR partition : 2 0x00 0xef 568 832 MBR partition : 3 0x00 0x01 45056 12288 ... GPT start and size : 1 64 44992 ... GPT start and size : 2 568 832 ... GPT start and size : 3 45056 12288----------------------------------------------------------- What do the above header-style lines mean ? They stem from xorriso command -report_system_area "plain". Parameter "help" prints the help text from libisofs: $ xorriso-1.3.8 -report_system_area help ... System area summary: word ... word human readable interpretation of system area options and other info The words are from the set: { MBR, CHRP, PReP, GPT, APM, MIPS-Big-Endian, MIPS-Little-Endian, SUN-SPARC-Disk-Label, HP-PA-PALO, protective-msdos-label, isohybrid, grub2-mbr, cyl-align-{auto,on,off,all}, not-recognized, } The acronyms indicate boot data for particular hardware/firmware. protective-msdos-label is an MBR conformant to specs of GPT. isohybrid is an MBR implementing ISOLINUX isohybrid functionality. grub2-mbr is an MBR with GRUB2 64 bit address patching. cyl-align-on indicates that the ISO image MBR partition ends at a cylinder boundary. cyl-align-all means that more MBR partitions exist and all end at a cylinder boundary. not-recognized tells about unrecognized non-zero system area data. ISO image size/512 : decimal size of ISO image in block units of 512 bytes. If an MBR is detected, with at least one partition entry of non-zero size, then there may be: Partition offset : decimal if not 0 then a second ISO 9660 superblock was found to which MBR partition 1 is pointing. MBR heads per cyl : decimal conversion factor between MBR C/H/S address and LBA. 0=inconsistent. MBR secs per head : decimal conversion factor between MBR C/H/S address and LBA. 0=inconsistent. MBR partition table: N Status Type Start Blocks headline for MBR partition table. MBR partition : X hex hex decimal decimal gives partition number, status byte, type byte, start block, and number of blocks. 512 bytes per block. MBR partition path : X path the path of a file in the ISO image which begins at the partition start block of partition X. ... GUID Partition Table can coexist with MBR: GPT : N Info headline for GPT partition table. The fields are too wide for a neat table. So they are listed with a partition number and a text. ... GPT disk GUID : hex_digits 32 hex digits giving the byte string of the disk's GUID GPT entry array : decimal decimal word start block of partition entry array and number of entries. 512 bytes per block. The word may be "separated" if partitions are disjoint, "overlapping" if they are not. In future there may be "nested" as special case where all overlapping partitions are superset and subset, and "covering" as special case of disjoint partitions covering the whole GPT block range for partitions. GPT lba range : decimal decimal decimal addresses of first payload block, last payload block, and of the GPT backup header block. 512 bytes per block. GPT partition name : X hex_digits up to 144 hex digits giving the UTF-16LE name byte string of partition X. Trailing 16 bit 0-characters are omitted. GPT partname local : X text the name of partition X converted to the local character set. This line may be missing if the name cannot be converted, or is empty. GPT partition GUID : X hex_digits 32 hex digits giving the byte string of the GUID of partition X. GPT type GUID : X hex_digits 32 hex digits giving the byte string of the type GUID of partition X. GPT partition flags: X hex 64 flag bits of partition X in hex representation. Known bit meanings are: bit0 = "System Partition" Do not alter. bit2 = Legacy BIOS bootable (MBR partition type 0x80) bit60= read-only GPT start and size : X decimal decimal start block and number of blocks of partition X. 512 bytes per block. GPT partition path : X path the path of a file in the ISO image which begins at the partition start block of partition X.----------------------------------------------------------- I am still testing the new features with GPT. Commit to repositories and upload of a new snapshot xorriso-1.3.9.tar.gz will happen in a few days. Have a nice day :) Thomas
Ian Campbell <ijc@debian.org> (2015-02-01): Revisiting that during the stretch cycle doesn't seem shocking to me. Mraw, KiBi.