Dear Maintainer,
this is a corner-case situation but one which may indeed happen, especially
when upgrading the distro e.g. from squeeze to wheezy, ending in an
unbootable system without apparent motivation.
I have a system with two twin disks, partitioned identically with partitions
then combined in pairs to have md raid 1 devices. Recently, one of the disks
failed, and I had to replace it with what I had at hand. The new disk was a
"green" series disk, with aggressive built-in energy saving that does not
play very well with raid. Indeed it will be replaced as soon as possible,
but this was a "better than nothing" temporary arrangement.
This happened just when wheezy was released, so I set up for a distro
upgrade which, after the usual amount of relatively minor problems (there is
a huge amount of sw on that machine, so there were some circular
dependencies I had to break etc.) I completed the upgrade (which included
upgrading grub) and rebooted. Or rather tried to, since reboot stopped at
the grub rescue prompt, after a cryptic message of "incompatible license"
problems.
After much cursing, and failing to find problems, I nailed the thing down,
this is the reconstruction:
1) during the distro upgrade, the new "green" disk went in standby once too
often, and was dropped from the disk array, which kept working flawlessly on
the good disk
2) grub was upgraded (in the overall upgrade) when the raid was out of sync,
so it was upgraded only on one disk and not on the other
3) at the next reboot, grub still recognised the "failed" disk as a member
of the raid 1 array containing /boot, and accidentally chose to read its
modules off it
4) the modules on the "failed" disk were of the older grub version from
squeeze, hence grub stopped cold with the cryptic (for me) message about
incompatible license
Quick&dirty solution: either unplug the "green" disk altogether, or re-add
it to the raid 1 array to have it resynced. Both worked.
Grub problems: It would be really much nicer if the mdraid1x.mod (or
whatever other part of grub is involved) could detect that a partition is
an out of sync member of a raid array and tried to use good ones first.
I see that grub.cfg uses the UUIDs of the member partitions to identify
where to find boot and root partitions, and these are identical for all
members in case of a raid 1 partition. This is a good thing, as it means one
can boot the system if any of them is present. However, if a partition drops
out of sync but is not physically unplugged, it will still show the same
UUID and grub may attempt to use it (with potentially very bad
consequences). Is there any way that grub can use to distinguish that a
member of a raid array is out of sync versus the "good" ones? The kernel and
mdadm can do it, so it must be possible, but maybe it's too tricky to
include in grub, I am not such an expert in the internal workings of linux
software raid so I cannot tell.
In case this is too complicated, this should be documented clearly.
I could not (maybe it was my fault) find this possible issue
documented anywhere.
This issue is very easy to reproduce for testing:
1) setup a raid 1 array and install a minimal system on it, make it bootable
with grub
2) note the names of the participating devices, e.g. sda1 and sdb1
3) mark the first one as failed with mdadm, and remove it from the array
(e.g. mdadm /dev/md0 -f /dev/sda1 ; mdadm /dev/md0 -r /dev/sda1). But leave
the "failed" disk connected to the system
4) change the version of grub on md0, which now is running only on sdb1, so
that its modules are incompatible with the previously installed ones (e.g.
switch between the squeeze and wheezy versions of grub), making sure to run
grub-install and update-grub. Everything will run without errors
5) try to reboot. If grub tries to load modules from the "failed" /dev/sda1
it will stop with an "incompatible license" message.
I hope this is not a waste of time for grub maintainers. I already solved my
specific problem, and my brutal solution (unplug the "failed" disk) will
work for other people running into the same problem. However, it may take
quite a bit of time to figure out what is happening, since everything
appears to be perfect if one inspects the raid device, the grub boot sector
on the MBR, etc., everything appears to be ok. The problem is hidden in the
_contents_ of the "failed" (but still physically connected) raid member
which fell out of sync.
So I am filing this bug report in the hope that it may save a couple of days
(a couple of hours, if they are much smarter than me) of time to other
sysadmins.