When I tried extending the /dev/vvm/root partition to include more space from the /dev/vdc1 physical volume, grub fails to load at boot, unable to find the lvm volume. I tried re-installing grub from the debian-installer rescue image, but that did not help. Removing /dev/vdc1 from the volume group fixed the issue, but significantly limits the benefit of LVM... A workaround is, of course, using a separate boot partition, or maybe keeping to only a single physical volume for the logical volume on which /boot resides. live well, vagrant
Hi Vagrant! Can you give us a bit more detail about the system please? With /dev/vdc, this suggests you're in a VM and this is the third drive using virtio? How big is /dev/vdc? What's the partition type? Why am I asking? grub-pc is limited by the platform underneath here when it comes to assembling RAID or LVM volumes. If a complex disk setup depends on a drive that can't be seen/read by grub at boot, it's going to struggle. I'm wondering if this might be the underlying cause of your issue. If possible, on your system, could you also reboot and call up a grub command line (hit "c" from the grub menu)?
... The disk is about 2TB and the partition type is GPT. There is also a 1TB disk as /dev/vdb GPT, unused. The main boot disk is ~500GB /dev/vda MBR. At the moment, this is difficult as I am now doing some long-running builds on it... but will try when I get the chance or maybe reproduce the situation in another VM. Will do! Thanks for the quick response! live well, vagrant
Control: severity -1 minor
Control: tags -1 wontfix
Control: retitle -1 grub-pc: needs reconfiguration with complex storage setup
OK, I can reproduce with something like this:
vda - small-ish dos-partitioned disk, fresh bullseye installation made
using LVM directly for the rootfs, no /boot
(all works)
Then add:
vdb - 8T GPT drive, qcow2, unused
vdc - 8T GPT drive, qcow2, a partition to add to the LVM VG
Rebooting straight away at this point still boots ok.
*However*, using pvmove to move the root LV to vdc fails. I'd expect
similar if you've just extended and grown the rootfs. I get an error:
error: disk `lvmid/<stuff>' not found
(not copying all of the UUID-style path by hand!)
and this is exactly the kind of thing I was looking for - maybe
grub-pc can't access the drive due to BIOS limitations. *However*, I
think I'm wrong and Pascal Hambourg is right here...!
I ran d-i in rescue mode to get into the system, simply ran
dpkg-reconfigure grub-pc (which will run grub-install *and*
update-grub), and the system now boots again. It looks like what we're
seeing might be a limit in what's built in to the core image by
default. grub-pc is deliberately designed to build minimal images
here, to minimise the chance of the image being too large to embed.
Being brutally honest, I think the problem is in your disk
setup. You've found an edge case that *can* be accomodated and
supported, but needs some extra effort to make it work.
I'd *strongly* recommend re-thinking your setup here. If you're going
to set up complicated storage here *for a VM*, then it would be
trivial to set up a /boot partition which grub will never have
problems finding and booting from. Make your life easier and just do
that...
... ... "ls" from the grub prompt did not show the other disk... ...until I made the second disk bootable from libvirt! Then grub now sees both disks, and boots fine! So this is possibly a quirk of the way libvirt exposes boot disks. ... That sounds consistent with the error message I got. I was able to reproduce with a simpler virtual machine (two disks, lvm-backed, 10GB and 20GB)... Re-running grub-install /dev/vda from debian-installer rescue mode did *not* fix the issue for me (though, now I am curious if dpkg-reconfigure grub-pc would do anything more?) I only got into that situation due to a somewhat weird circumstance (e.g. having a user on a system with libvirt access, but not root, and libvirt helpfully re-owns files so that nobody else can mess with them). Needed more space than originally planned, figured I could just add more disks to LVM... and here we are. I had used this sort of setup for at least a release or two of Debian releases on real hardware without much trouble, so was inclined to go all-in on LVM... For the future, I guess I'll just resign myself to a split /boot. Once I saw this bug I figured it was worth reporting, as it appeared different enough from others I coudl see in the BTS. Thanks for poking at this, hopefully someone else who stubs their toe on this in the future will benefit. :) live well, vagrant
On Sun, 30 Apr 2023 17:37:13 -0700 Vagrant Cascadian <vagrant@debian.org> wrote: Apparently. GRUB can only see disks exposed by the BIOS/UEFI/other platform firmware. There are other known situations where a given firmware may not expose some disks, including but not limited to: - disks connected to a SATA controller card without a BIOS expansion ROM - unsupported media types: USB other than the boot disk, NVMe, SD/MMC... The UEFI firmware on an old Intel board only had EFI drivers for the SATA controller in IDE mode and lacked EFI drivers for AHCI and USB. It has been reported that more recent boards had support for NVMe only in EFI mode, not in legacy mode. I wonder how much this policy is still relevant for PC platforms. Originally the core image was designed to fit in the "post-MBR gap" whose typical size was 62 sectors (31 KiB) because the first partition used to start at sector 63. But nowadays a 1-MiB post-MBR gap has been the standard for many years. I do not remember when this was introduced in fdisk and other Linux partition editors, but Windows 7 installer had it. Besides, I observe that the size of the core.img built for LVM+ext4 on my bullseye system is 34 KiB so it would not even fit in a 31-KiB post-MBR gap. The minimal core image policy is even less relevant for EFI images, as the EFI partition size is usually several MB so a few more kB won't hurt. I cannot tell for other platforms. I think that when building the i386-pc core image with support for storage possibly involving multiple disks (LVM, RAID, btrfs), support for at least both MSDOS and GPT partition schemes (other partition schemes are unlikely to be used on PC) could be added unconditionally to prevent such GRUB failure after adding a disk with a different partition scheme to the /boot filesystem. It would add only 2 KB to the core image, and it is likely that the minimal size is already above 31-KiB anyway when the above storage drivers are embedded. Opinions ? - the disk was not exposed by libvirt BIOS - the disk had an unsupported GPT partition scheme grub-install fixed only the second issue. Making the disk "bootable" in libvirt was required to fix the first issue. As Steve wrote, dpkg-reconfigure also runs update-grub to rebuild grub.cfg. AFAIK here the only difference with the old grub.cfg is additional "insmod part_gpt" commands to load GPT support, but the module must already be embedded in the core image so this addition is not required.
... Well, in my original bug report, grub.cfg did contain both: insmod part_gpt insmod part_msdos Although, maybe it is possible that was generated after running update-grub... Even though a perfectly reasonable stance is just to use a split /boot partition, I may keep experimenting with this to get greater confidence we really understand what went wrong. :) live well, vagrant
Even the 1MB post-MBR gap can be a struggle AIUI. BIOS booting is fragile as hell, and there have been lots of discussions over the years on the upstream list about the problems of embedding... Nod, exactly. To be honest, EFI typically makes things *so* much more sane here for exactly these reasons. Then again you get to see more issues with broken firmware. :-/