Dear Maintainer,

   * What led up to the situation?

On a fresh install of Debian 12, on an OVH baremetal machine with mdadm
raid-0, I upgraded to Debian 13. After upgrade, grub in EFI had been
upgraded in one drive, but not the other.

I can reproduce this, and the drive that does get upgraded appears to be
somewhat random.
If the partition that's mounted to /boot/efi misses the upgrade, the system
will fail to boot and end up in BIOS, from where EFI shell can be accessed.

I have a `script` of the upgrade and will attempt to upload it. I did
not see anything jump out of me there.

To investigate, I used `strings` and mounted the second EFI partition to
`/mnt/second-efi`

Before upgrade, both partitions have grub 2.06, as expected:
```
$ strings /boot/efi/EFI/debian/grubx64.efi | grep -i grub | tail -6
        source $prefix/grub.cfg
        source $cmdpath/grub.cfg
normal (memdisk)/grub.cfg
grub,4,Free Software Foundation,grub,2.06,https://www.gnu.org/software/grub/
grub.debian,4,Debian,grub2,2.06-13+deb12u1,https://tracker.debian.org/pkg/grub2
&Debian Secure Boot Signer 2022 - grub20

$ strings /mnt/second-efi/EFI/debian/grubx64.efi | grep -i grub | tail -6
        source $prefix/grub.cfg
        source $cmdpath/grub.cfg
normal (memdisk)/grub.cfg
grub,4,Free Software Foundation,grub,2.06,https://www.gnu.org/software/grub/
grub.debian,4,Debian,grub2,2.06-13+deb12u1,https://tracker.debian.org/pkg/grub2
&Debian Secure Boot Signer 2022 - grub20
```

Now run an upgrade as per release notes, do not yet reboot, and check
again. Only one partition has the upgraded grub 2.12

```
$ strings /boot/efi/EFI/debian/grubx64.efi | grep -i grub | tail -6
grub_diskfilter_read_node
grub,5,Free Software Foundation,grub,2.12,https://www.gnu.org/software/grub/
grub.debian,5,Debian,grub2,2.12-9+deb13u1,https://tracker.debian.org/pkg/grub2
grub.debian13,1,Debian,grub2,2.12-9+deb13u1,https://tracker.debian.org/pkg/grub2
grub.peimage,2,Canonical,grub2,2.12-9+deb13u1,https://salsa.debian.org/grub-team/grub/-/blob/master/debian/patches/secure-boot/efi-use-peimage-shim.patch
&Debian Secure Boot Signer 2022 - grub20

$ strings /mnt/second-efi/EFI/debian/grubx64.efi | grep -i grub | tail -6
        source $prefix/grub.cfg
        source $cmdpath/grub.cfg
normal (memdisk)/grub.cfg
grub,4,Free Software Foundation,grub,2.06,https://www.gnu.org/software/grub/
grub.debian,4,Debian,grub2,2.06-13+deb12u1,https://tracker.debian.org/pkg/grub2
&Debian Secure Boot Signer 2022 - grub20
```

If `/boot/efi` were old, I'd expect `reboot` to fail here

Manually upgrade the old EFI and verify
```
$ sudo grub-install --target=x86_64-efi --efi-directory=/mnt/second-efi --bootloader-id=debian
$ strings /mnt/second-efi/EFI/debian/grubx64.efi | grep -i grub | tail -6
grub_diskfilter_read_node
grub,5,Free Software Foundation,grub,2.12,https://www.gnu.org/software/grub/
grub.debian,5,Debian,grub2,2.12-9+deb13u1,https://tracker.debian.org/pkg/grub2
grub.debian13,1,Debian,grub2,2.12-9+deb13u1,https://tracker.debian.org/pkg/grub2
grub.peimage,2,Canonical,grub2,2.12-9+deb13u1,https://salsa.debian.org/grub-team/grub/-/blob/master/debian/patches/secure-boot/efi-use-peimage-shim.patch
&Debian Secure Boot Signer 2022 - grub20
```

This resolves the issue.

**Desired behavior**

Upgrade via `sudo apt dist-upgrade` upgrades all EFI partitions in an
mdadm raid setup.

I am unsure which package the issue is in. What determines, during upgrade, where to install Grub EFI? Is it grub? Is it mdadm? Is it the updater itself? Please help steer this bug to the correct place.

Some more context.

I first encountered this issue in June 2025, on a fleet of about 100 servers. After the first few failures, we stopped. I finally got around to tracking the issue down.

I have reproduced the issue on a system with raid0. I do not yet know whether it also occurs with raid1.

The very same, identical, machines were successfully upgraded from Debian 11 to Debian 12. In many cases not just the same hardware class, but the actual machine that went through an 11->12 upgrade.

My reproduction is from a fresh 12 install, to reduce variables.

Kind Regards
Thorsten

While reproducing the bug, I captured upgrade terminal output with script. I forgot to create a timing file, but you can view it with less -R.

Attempting to attach the typescript file.

The issue can be recreated on a raid1 system, as well. Script log attached, this time with timing file

Not the maintainer of any of the relevant packages, but trying to
help this bug forward.

grub's postinst runs grub-install. grub-install determines the
install location. I -think- it just uses /boot/efi, but the grub
maintainers will know better.

mdadm is not involved. There is no "updater" in Debian.

Best,
Chris

Correct.

Correct.

It's a longstanding hole in grub-efi (sorry!), but there is nothing
special here yet to make a RAIDed ESP work. Thorsten: how did you set
things up, exactly?

Apologies, I need to learn to Reply All. 2nd try

Thank you for engaging with this!

This is an OVH baremetal, so on my end it's just "install Debian 12 through OVH interface, they provision efibootmgr, mdadm and raid0/1". Which of course is supremely unhelpful to you.

For starters, here's lsblk

lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,PARTUUID,UUID,MOUNTPOINTS
NAME          SIZE TYPE  FSTYPE            LABEL          PARTUUID                             UUID                                 MOUNTPOINTS
nvme0n1     476.9G disk
├─nvme0n1p1   511M part  vfat              EFI_SYSPART    ded29cd6-9e09-4b32-8826-7f10660f8690 2B1D-A9FA                            /boot/efi
├─nvme0n1p2     1G part  linux_raid_member md2            fea21b2b-2455-4cb2-a9a0-4c338899c517 79fa0b16-1de0-4dc9-6bad-c8c456d3c8c5
│ └─md2      1022M raid1 ext4              boot                                                543ebee6-5c6e-4a5c-86df-0a595ea9829b /boot
├─nvme0n1p3 474.9G part  linux_raid_member md3            f04002cc-5ce7-4cca-a525-6838fd972987 3de91c82-c54a-2edf-bc42-2128fe62bef8
│ └─md3     474.8G raid1 ext4              root                                                100619c3-ea4b-4fee-aaa3-83fb471723ff /
├─nvme0n1p4   512M part  swap              swap-nvme0n1p4 94db751f-7d6d-426d-aae7-55699675d794 77242712-5e31-480d-a57e-3c7c0f1731fa [SWAP]
└─nvme0n1p5     2M part  iso9660           config-2       f514a1f0-532f-498b-a525-7222224beb6f 2026-04-28-08-41-39-00
nvme1n1     476.9G disk
├─nvme1n1p1   511M part  vfat              EFI_SYSPART    06c9b490-e21f-46ae-8bd8-0344a8d34e37 2B2B-4A82
├─nvme1n1p2     1G part  linux_raid_member md2            9b00687f-9d0c-4121-abae-6bf2a454a85b 79fa0b16-1de0-4dc9-6bad-c8c456d3c8c5
│ └─md2      1022M raid1 ext4              boot                                                543ebee6-5c6e-4a5c-86df-0a595ea9829b /boot
├─nvme1n1p3 474.9G part  linux_raid_member md3            bf8223a6-d908-4343-ad03-8cdecb3420fd 3de91c82-c54a-2edf-bc42-2128fe62bef8
│ └─md3     474.8G raid1 ext4              root                                                100619c3-ea4b-4fee-aaa3-83fb471723ff /
└─nvme1n1p4   512M part  swap              swap-nvme1n1p4 c728d6aa-ee6e-4cf0-9e6b-e41d3b4e2934 2f88d2cc-a612-42c8-9df6-a533abdb3226 [SWAP]

`nvme1n1p1` is the second drive's EFI partition

Here's fstab
```
UUID=100619c3-ea4b-4fee-aaa3-83fb471723ff       /       ext4    defaults        0       1
UUID=543ebee6-5c6e-4a5c-86df-0a595ea9829b       /boot   ext4    defaults        0       0
LABEL=EFI_SYSPART       /boot/efi       vfat    defaults        0       1
UUID=77242712-5e31-480d-a57e-3c7c0f1731fa       swap    swap    defaults        0       0
UUID=2f88d2cc-a612-42c8-9df6-a533abdb3226       swap    swap    defaults        0       0
```

Here's efibootmgr -v
```
BootCurrent: 000C
Timeout: 2 seconds
BootOrder: 000C,000D,0007,000B,0001
Boot0001* UEFI: Built-in EFI Shell      VenMedia(5023b95c-db26-429b-a648-bd47664c8012)0000424f
      dp: 04 03 14 00 5c b9 23 50 26 db 9b 42 a6 48 bd 47 66 4c 80 12 / 7f ff 04 00
    data: 00 00 42 4f
Boot0007* UEFI: PXE IPv4 Intel(R) Ethernet Controller X550      PciRoot(0x0)/Pci(0x1b,0x0)/Pci(0x0,0x0)/MAC(d85ed343de10,1)/IPv4(0.0.0.00.0.0.0,0,0)0000424f
      dp: 02 01 0c 00 d0 41 03 0a 00 00 00 00 / 01 01 06 00 00 1b / 01 01 06 00 00 00 / 03 0b 25 00 d8 5e d3 43 de 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 / 03 0c 1b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 / 7f ff 04 00
    data: 00 00 42 4f
Boot0008  UEFI: PXE IPv4 Intel(R) Ethernet Controller X550      PciRoot(0x0)/Pci(0x1b,0x0)/Pci(0x0,0x1)/MAC(d85ed343de11,1)/IPv4(0.0.0.00.0.0.0,0,0)0000424f
      dp: 02 01 0c 00 d0 41 03 0a 00 00 00 00 / 01 01 06 00 00 1b / 01 01 06 00 01 00 / 03 0b 25 00 d8 5e d3 43 de 11 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 / 03 0c 1b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 / 7f ff 04 00
    data: 00 00 42 4f
Boot000B* UEFI: PXE IPv4 Intel(R) Ethernet Controller X550      PciRoot(0x0)/Pci(0x1b,0x0)/Pci(0x0,0x1)/MAC(d85ed343de11,1)/IPv4(0.0.0.00.0.0.0,0,0)0000424f
      dp: 02 01 0c 00 d0 41 03 0a 00 00 00 00 / 01 01 06 00 00 1b / 01 01 06 00 01 00 / 03 0b 25 00 d8 5e d3 43 de 11 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 / 03 0c 1b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 / 7f ff 04 00
    data: 00 00 42 4f
Boot000C* debian        HD(1,GPT,06c9b490-e21f-46ae-8bd8-0344a8d34e37,0x800,0xff800)/File(\EFI\debian\shimx64.efi)
      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 f8 0f 00 00 00 00 00 90 b4 c9 06 1f e2 ae 46 8b d8 03 44 a8 d3 4e 37 02 02 / 04 04 34 00 5c 00 45 00 46 00 49 00 5c 00 64 00 65 00 62 00 69 00 61 00 6e 00 5c 00 73 00 68 00 69 00 6d 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
Boot000D* debian        HD(1,GPT,ded29cd6-9e09-4b32-8826-7f10660f8690,0x800,0xff800)/File(\EFI\debian\shimx64.efi) File(.)
      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 00 00 f8 0f 00 00 00 00 00 d6 9c d2 de 09 9e 32 4b 88 26 7f 10 66 0f 86 90 02 02 / 04 04 34 00 5c 00 45 00 46 00 49 00 5c 00 64 00 65 00 62 00 69 00 61 00 6e 00 5c 00 73 00 68 00 69 00 6d 00 78 00 36 00 34 00 2e 00 65 00 66 00 69 00 00 00 / 7f ff 04 00
    data: 00 00 42 4f
```

You can see that both EFI partitions are configured in efibootmgr

Mdadm for good measure, but that likely doesn't matter here

```
sudo mdadm --detail --scan
ARRAY /dev/md/md2 metadata=1.2 UUID=79fa0b16:1de04dc9:6badc8c4:56d3c8c5
ARRAY /dev/md/md3 metadata=1.2 UUID=3de91c82:c54a2edf:bc422128:fe62bef8
```

Steve and team,

OK I get it now. The plot thickens. That is why it worked 11->12 - the EFI didn't change version. This is the EFI from 11

```
$ strings /mnt/second-efi/EFI/debian/grubx64.efi | grep -i grub | tail -6
        source $prefix/grub.cfg
        source $cmdpath/grub.cfg
normal (memdisk)/grub.cfg
grub,4,Free Software Foundation,grub,2.06,https://www.gnu.org/software/grub/
grub.debian,4,Debian,grub2,2.06-3~deb11u6,https://tracker.debian.org/pkg/grub2
&Debian Secure Boot Signer 2022 - grub20

$ strings /mnt/efi/EFI/debian/grubx64.efi | grep -i grub | tail -6
strings: '/mnt/efi/EFI/debian/grubx64.efi': No such file
debian@ethstaker-mumbai:~$ strings /boot/efi/EFI/debian/grubx64.efi | grep -i grub | tail -6
        source $prefix/grub.cfg
        source $cmdpath/grub.cfg
normal (memdisk)/grub.cfg
grub,4,Free Software Foundation,grub,2.06,https://www.gnu.org/software/grub/
grub.debian,4,Debian,grub2,2.06-3~deb11u6,https://tracker.debian.org/pkg/grub2
&Debian Secure Boot Signer 2022 - grub20
```

Same version as 12. 2.06 in both cases. I'm assuming `2.06-13+deb12u1` and `2.06-3~deb11u6` are binary compatible, that is if the machine boots from EFI `2.06-3~deb11u6` into Debian 12 grub, all is well.

Anecdotally that seems to be true because our 11->12 fleet upgrade went without a hitch in 2023.

With 12->13, the EFI version changes, and just `/boot/efi` gets updated. Then it's up to the vagaries of ... efibootmgr I suppose, whether the updated partition is used or no. In my testing failure is intermittent. The system may or may not reboot on the old EFI, but once it does, the only way back is to go into EFI shell, choose the other fs like say fs1:, and then run shimx64.efi (the upgraded one) from there.

This is not necessarily a bug, it's a (missing) feature. Which bites users going from 12->13 because now EFI version changes.

Can I suggest at a minimum documentation and a warning at https://www.debian.org/releases/trixie/release-notes/upgrading.en.html#possible-issues-during-upgrade and maybe even before then, a warning earlier in the doc to look at the possible issues section and what to do about multi-disk EFI setups.

Completely amazing ofc would be that grub-install or "whatever responsible component" sees all EFI entries in efibootmgr and upgrades / installs into all of them, or has a switch to do so, which gets exercised on grub package update.

Yours
Thorsten

And, because it's still grub 2.06, I can boot from it

Debian 11->12 upgrade. Confirmed /boot/efi only was updated

```
$ strings /mnt/second-efi/EFI/debian/grubx64.efi | grep -i grub | tail -6
        source $prefix/grub.cfg
        source $cmdpath/grub.cfg
normal (memdisk)/grub.cfg
grub,4,Free Software Foundation,grub,2.06,https://www.gnu.org/software/grub/
grub.debian,4,Debian,grub2,2.06-3~deb11u6,https://tracker.debian.org/pkg/grub2
&Debian Secure Boot Signer 2022 - grub20

$ strings /boot/efi/EFI/debian/grubx64.efi | grep -i grub | tail -6
        source $prefix/grub.cfg
        source $cmdpath/grub.cfg
normal (memdisk)/grub.cfg
grub,4,Free Software Foundation,grub,2.06,https://www.gnu.org/software/grub/
grub.debian,4,Debian,grub2,2.06-13+deb12u1,https://tracker.debian.org/pkg/grub2
&Debian Secure Boot Signer 2022 - grub20
```

Old:
```
~$ ls -l /mnt/second-efi/EFI/debian/grubx64.efi
-rwxr-xr-x 1 root root 4150720 Apr 28 12:43 /mnt/second-efi/EFI/debian/grubx64.efi
```

New:
```
~$ ls -l /boot/efi/EFI/debian/grubx64.efi
-rwxr-xr-x 1 root root 4199872 Apr 28 13:27 /boot/efi/EFI/debian/grubx64.efi
```

Reboot into BIOS and choose EFI shell, `sudo systemctl reboot --firmware-setup`

FS0: Old, verified by size of grubx64.efi
FS1: New, verified by size of grubx64.efi

FS0:
cd EFI
cd debian
shimx64.efi

Boots without issue

I know what to do as a workaround. Upgrade the second EFI partition.

What to do on the Debian side, that I have not a lot of input to, other than the previous idea to call this out in Release Notes.

Kind Regards
Thorsten

A thought on how to improve the postinst of grub

Something along these lines - not this exact script, but in spirit - would update grub on all ESPs

```
#!/usr/bin/env bash
set -euo pipefail

ESP_GUID="c12a7328-f81f-11d2-ba4b-00a0c93ec93b"

lsblk -rno NAME,PARTTYPE,FSTYPE | while read -r name parttype fstype; do
  if [[ "$parttype" == "$ESP_GUID" && "$fstype" == "vfat" ]]; then
    dev="/dev/$name"
    mnt="$(mktemp -d)"

    echo "Installing GRUB to $dev"

    mount "$dev" "$mnt"

    grub-install \
      --target=x86_64-efi \
      --efi-directory="$mnt" \
      --bootloader-id=debian \
      --recheck

    umount "$mnt"
    rmdir "$mnt"
  fi
done
```

Thorsten Behrens wrote:

What is the point of having multiple ESPs with RAID 0 ? It does not
provide any redundancy, so if either drive fails the system would fail
to boot anyway.

Both ESPs have the same LABEL (this is wrong, LABELs should be unique)
and fstab mounts the ESP by LABEL. So the mounted ESP is chosen randomly
(or at least not reliably) and only the ESP mounted on /boot/efi is
updated by grub-install.

FWIW this is the contents of the embedded SBAT CSV, not the GRUB
version. The actual GRUB version can be found in another string:

GNU GRUB  version %s
2.06-13+deb12u1

EFI boot entries are not reliable enough. They do not even exist on some
systems.

This is a bad idea. Not all detected ESPs may belong to this system.
A correct solution requires to record the ESPs which are managed by this
system and must be updated.

Fair. The same issue is present with RAID 1, and I am dealing with an installation where multiple ESPs are being created on RAID 0. A solution for RAID 1 would also solve RAID 0 systems.

Thank you for adding color re why my boot failures were random. Not great by OVH; also though with RAID 1, a boot failure would occur on disk failure even on a system with unique labels.

Noted. So, that solution is bad. A solution, of some kind, would be great. As that's clearly not trivial, a note at https://www.debian.org/releases/stable/release-notes/issues.en.html#things-to-be-aware-of-while-upgrading-to-releasename to look out for systems with multiple ESPs such as RAID 1 setups, and manually updating the ones belonging to Debian and not currently mounted to /boot/efi, would be amazing.

Sincerely yours
Thorsten

Thank you for your patience as I learn how Debian hangs together. Is this an equally bad idea?

Get UUID of  currently mounted /boot/grub with CURRENT_UUID=$(lsblk -no UUID $(df /boot/grub | tail -1 | awk '{print $1}'))

Get UUID that `grub.cfg` on the ESP I am looking at chains to with `ESP_UUID=$(grep "search.fs_uuid" "$mnt"/EFI/debian/grub.cfg | awk '{print $2}')`

If the two match, this ESP is managed by this system and should be updated

The main point being whether comparing these two UUIDs is the right way to identify that an ESP is managed by the current system.

My tail and awk and grep stuff may well not be robust to get these two UUIDs in the first place.

Yours
Thorsten

Debian does not support multiple ESPs out of the box, so users who set
this up should already know.

This is not generic and reliable enough. grub.cfg in the ESP is
generated only with signed GRUB images (shim + grub-efi-$arch-signed).
Signed GRUB images are available only for amd64 and arm64.

grub-pc records the devices where GRUB should be installed in debconf,
why wouldn't grub-efi just do the same for EFI partitions ?

That clearly isn't always so in practice, particularly when the Debian setup is done by a provider of rented equipment. I think it'd be friendly and good UX to mention it in release notes; but I can also see a point of "that's too niche, we can't mention everything".

Over to the maintainers of grub-efi. I am not nearly deep enough in Debian to hold an opinion.

Yours sincerely
Thorsten