When using the root=UUID=... syntax (the default with grub2) for a root filesystem in LVM the local-top/lvm2 script does not activate the volume group and the initramfs dumps me into a rescue shell. Running "lvm vgchange -aly" and exiting the shell is sufficent to allow the boot to continue. Using GRUB_DISABLE_LINUX_UUID=true in /etc/default/grub is also sufficient to work around the issue. This is because the script will bail on paths which don't start /dev/mapper. Presumably this can be worked around by using lvm vgchange -aly --ignorelockingfailure when $dev matches UUID=* (and perhaps LABEL=*?) similar to the existing LILO "fe[0-9]*" and /dev/root workarounds. I guess the current scheme would also break if one were to use the alternative /dev/Vol00/root naming rather than /dev/mapper/Vol00-root? Maybe that's a case of "don't do that then". Ian.
* Ian Campbell <ijc@hellion.org.uk>, 2011-02-08, 10:40: Oh yes, it does break in such a case, too. I "tested" it today in practice. :/
--- Please enter the report below this line. --- linux-image-2.6.38-bpo2-amd64 is able to boot from my VG "/dev/Desktop/root" but linux-image-3.0.0-1-amd64 is not able to boot from this VG. I upgraded today my squeeze to wheezy and let the 2.6.38 kernel installed. grub 1.99 Best Regards Georg Gast Debian Release: wheezy/sid 500 testing ftp.de.debian.org --- Package information. --- Depends (Version) | Installed =====================================-+-=============== libc6 (>= 2.3) | 2.13-16 libdevmapper1.02.1 (>= 2:1.02.63) | 2:1.02.63-3.1 libreadline5 (>= 5.2) | 5.2-8 libudev0 (>= 0.140) | 172-1 lsb-base | 3.2-27 dmsetup (>> 2:1.02.47) | 2:1.02.63-3.1 Package's Recommends field is empty. Package's Suggests field is empty.
Hello Georg, 2011/8/23 Georg Gast <schorsch_76@gmx.de>: Could you please provide more information? Error message or some way to reproduce this behaviour. Why do you think is lvm2 related? With best regards,
Hello Georg, Please let's try to keep CC to the bug report, so others can see. 2011/8/24 Georg Gast <schorsch_76@gmx.de>: This line hints your root filesystem was booting into read-only single mode. Likely root filesystem needed fsck. So, it's the problem away? can we close the bug?
Hello Hector, Yes, i think it can be closed :) Bye Bye Georg
* Hector Oron <hector.oron@gmail.com>, 2011-08-24, 19:14: [...] Wait, wait, Georg is not the submitter of this bug. Why do you ask him?
Hello, 2011/8/25 Jakub Wilk <jwilk@debian.org>: Uhm... right, looking to <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612402> shows the full thread. From the email format Georg sent I guessed it was a new bug report. Sorry for that. Cheers,
Hi, I updated the kernel of my squeeze-installation to the new wheezy-kernel and got the same problem (the kernel could not find /dev/mapper/vg-root), with " lvm vgchange -aly" i got it running, but i have to do it on each reboot. i think thats in some way a problem of linux-image-3.0.x, on the same machine the 2.6.39 of backports and 2.6.32 is working fine. regards Nico
Dear maintainers! Here [1] is the link to that image which works. [1] http://cdimage.debian.org/mirror/cdimage/archive/5.0.1/amd64/iso-cd/debian-501-amd64-netinst.iso Best Regards Georg Gast
I had the same problem on a squeeze-wheezy upgrade test today. I have same setup as original poster, using root=UUID=xxxxx I have raised the severity to critical, because the definition of a critical bug includes any bug that causes the whole system to break (even if there is no loss of data). This could impact many users when wheezy is released. At a bare minimum, if UUID is no longer supported, it should be detected and the upgrade should abort with a meaningful error. update-initramfs didn't give any error during upgrade After upgrade, I did a reboot test Boot failed, dropping me in the busybox shell with the "Gave up waiting for root device" message It was not necessary to use any recovery CD, in busybox, I just did: lvm vgchange -ay exit exit and it continued booting from there After booting, it is necessary to take these steps to fix (workaround) permanently: Edit /etc/default/grub, make sure this setting is present: GRUB_DISABLE_LINUX_UUID=true After changing the setting, run update-grub update-initramfs -u and then reboot and it comes up normally. Here are the versions of related packages: # dpkg --list | egrep 'lvm|initramfs|grub' | tr -s ' ' | cut -f2,3 -d' ' grub 0.97-66 grub-common 1.99-22.1 grub-doc 0.97-66 grub-legacy-doc 0.97-66 grub-pc 1.99-22.1 grub-pc-bin 1.99-22.1 grub2-common 1.99-22.1 initramfs-tools 0.107 libklibc 2.0.1-1 liblvm2app2.2:amd64 2.02.95-4 lvm2 2.02.95-4
I discovered another issue related to this
Basically, for btrfs RAID, where the RAID is formed from logical
volumes, scripts/local-top/lvm2 only calls
lvchange -ay something
for the first LV in the RAID.
E.g. if I have root=/dev/mapper/vg00-lv0 then my other LV,
/dev/mapper/vg00-lv1 is never activated.
This means that `btrfs dev scan' fails to find the array.
One way to solve this issue would be listing the btrfs filesystem's UUID as
root=UUID=xxxxxx
Here is a patch that solves the original issue and provides a way to
boot btrfs RAID1, I tested this on a fresh installed of wheezy from
beta1 DVD:
--- scripts/local-top/lvm2.orig 2012-08-29 14:37:04.672826476 +0200
+++ scripts/local-top/lvm2 2012-08-29 14:38:04.118536514 +0200
@@ -35,6 +35,10 @@
lvm vgchange -aly --ignorelockingfailure
exit 0
;;
+ /dev/disk/by-uuid/*)
+ lvm vgchange -aly --ignorelockingfailure
+ exit 0
+ ;;
esac
# Make sure that we have a d-m path
This is also needed to correctly generate /boot/grub/grug.cfg with UUID syntax for btrfs RAID on top of lvm:--- /etc/grub.d/10_linux.orig 2012-08-29 15:01:29.808976702 +0200 +++ /etc/grub.d/10_linux 2012-08-29 15:06:36.574297444 +0200 @@ -50,7 +50,7 @@ if [ "x${GRUB_DEVICE_UUID}" = "x" ] || [ "x${GRUB_DISABLE_LINUX_UUID}" = "xtrue" ] \ || ! test -e "/dev/disk/by-uuid/${GRUB_DEVICE_UUID}" \ - || uses_abstraction "${GRUB_DEVICE}" lvm; then + || (uses_abstraction "${GRUB_DEVICE}" lvm && [ ${GRUB_FS} != btrfs ] ); then LINUX_ROOT_DEVICE=${GRUB_DEVICE} else LINUX_ROOT_DEVICE=UUID=${GRUB_DEVICE_UUID}
If you *don't* use LVM this is normal, as device names are not stable. I just did some install and upgrade tests: 1. I installed squeeze and selected guided partitioning using LVM. The installer put /dev/mapper/* names in /etc/fstab (and also created a non-LVM /boot formatted as ext2!). Upgrading to wheezy worked fine. 2. I installed squeeze and selected 'manual partitioning' and created a pure LVM layout with root and swap LVs. This also resulted in /dev/mapper/* names in /etc/fstab. 3. Running 'dpkg-reconfigure linux-base' did not change these device names, as expected (it should only touch IDE and SCSI device names). So it seems that this is only going to be an issue if users take the unusual step of changing /etc/fstab to refer to LVs by UUID. But maybe there are management tools that do that as a matter of course? Ben.
They are not used by default for filesystems on LVM, because LVM volume names are already unique identifiers and fs UUIDs are *not* due to snapshotting. So the patch here may or may not be appropriate to apply, but in no case should you be referencing LVM volumes by UUID on /etc/fstab.
One should never use UUIDs in fstab to refer to non-unique filesystems (which is pretty much a normal thing with LVM, as the use of snapshotting is common). Any management tools that screw this up need a reality check (and post-haste bug fix). Doing otherwise is courting a data loss scenario. We've seen it happen a large number of times over the years because of the md 0.90 layout, until userspace tools started to add defensive layers to make it less likely to happen.
I'm not suggesting that squeeze systems were installed that way by default, although people who have migrated an FS from a raw partition to an LV may have this in fstab. As noted in the bug, somebody using a btrfs root FS (re-assembly of multiple volumes) may also have a problem if all those LVs are not active, and UUID may solve that for them. Once again, that is not an FS layout that any previous installer would have created automatically, but it is a valid way to mount a root filesystem. Maybe this is just something to be noted in the release notes, or if there are other issues like this that people have in mind, maybe it would be possible to write a pre-upgrade check script that users can download and run to find out about things like this before they upgrade. I don't mind helping out with such a script if nobody else has started on one already.
I vaguely recall the occasion which caused me to file this bug report but I can't recall any of the specifics about how I got into this state. (Which points to something of a shortcoming in my bug report, sorry). I do notice that I sent the report from a work machine so it might be something to do with Xen, but it is equally likely to be something on one of our server machines or I just happened to send it from work in my lunch hour. I can't think of a reason why I would have deliberately switched from the LV name to a UUID in fstab, but that's not to say I didn't. It's also possible that this was just a transient issue in the installer or grub or some other component which is gone now. Sorry for not being much help here. Ian.
Hi folks, is there a known grep (or similar) command to identify
affected /etc/fstab files before the wheezy upgrade?
If an affected file is found, what is an appropriate clean up?
For example, I found the following in my own /etc/fstab and, having read
this bug from top to bottom, am still unclear whether or not I am
affected. Are only "^root=UUID=" lines affected, or are "UUID=" lines
also affected?
/etc/fstab:
# /boot was on /dev/sda1 during installation
UUID=130c3014-afa2-4c10-9ab0-bdbfccb61aaf /boot ext2 defaults 0 2
Thanks for your time and help,
Nick
I just got a similar problem, where my testing amd64 machine stopped booting after I had to shut it down. Previous boot had occured on May 22th, and quite a lot of packages were updated since that time, including linux-image-3.2.0-4-amd64, which was the running kernel, from 3.2.41-2 to 3.2.46-1. As others described, I get a complain about a partition UUID not being found, get dropped to an initramfs shell, and after manually running "vgchange -ay" the boot does proceed fine. Any hint about what to look for ?
can verify that ROOT=/dev/disk/by-uuid/xxxx, and furthermore, I also
have resume=UUID=xxxx trying to reference my swap partition.
If I just set them to the /dev/mapper/vg-lv paths expected by
local-top/lvm2 and rerun the script, the LVs are properly activated.
Now:
* I did not recall changing anything in that field, so whatever change
in whatever package, that is responsible for this seems to warrant
the "critical" severity this bug used to have. It may be more
tricky than just "grub uses UUID by default and LVM does not want
that", but we have to track this problem.
* both ROOT and resume seem to be set by
/usr/share/initramfs-tools/init which belongs to package initramfs.
Although it was updated from 0.112 to 0.113 since the last succesful
boot, there is change in that area, and what it does to set ROOT
from the kernel commandline seems legit. I can't say the same for
"resume", which apparently should go through the same processing but
fails to.
* that switches the focus to whoever sets root=UUID= on the
commandline, and /etc/grub.d/10_linux seems to be where the thing
originates. There we have a test looking at « uses_abstraction
"${GRUB_DEVICE}" lvm » that should just avoid the issue completely,
and obviously fails. Grub had been updated one month before from
1.99-27 to 1.99-27+deb7u1 - this change includes something labelled
"TPU upload to lose dependency blockage on LVM." and in fact Daniel
Pocock already reported Bug#707831 with a short and clear statement,
just a pity this was not mentionned here...
I'll raise the severity of Bug#707831. You may want to keep this one
open so users are aware of the problem, or just reasign/merge with
that other bug...
As a datapoint, I got a VM from a low-end VPS provider, and it came configured exactly this way; lvm with UUID in fstab. I can reproduce the bug by reimaging the VM and upgrading it from stable to unstable. This configuration doesn't make much sense. However, there are many possible configurations that don't make much sense, and leaving the user with an unbootable system on upgrade is not the experience Debian strives for.
I've found another bug in grub's lvm probing that caused this same problem with the UUID to have surfaced in my case. See #743126 I hesitate to say that fixing that bug closes this one. There seem to be multiple failure modes that get a UUID into the kernel command line for LVM root filesystems, and so it would probably be good to defend against that in depth by noticing when this has happened and either a) fix it or b) make the initramfs robust enough to manage to boot successfully.
Hi, since Jessie will attempt to mount /usr in initramfs when using Systemd, this issue has somewhat broadened in scope. See the merged bug 781661. While I agree with the reasoning that UUID and LABEL are not correct for LVM volumes, and had in fact all our systems changed, this issue will still cause a system to be unable to boot after upgrading to Jessie. Not sure if it should be fixed or if it is enough to give a loud warning on upgrade. Regards, Marc