#983107 os-prober: generic subvolume support for btrfs

Package:
os-prober
Source:
os-prober
Description:
utility to detect other OSes on a set of drives
Submitter:
Osamu Aoki
Date:
2021-02-21 13:21:07 UTC
Severity:
normal
#983107#5
Date:
2021-02-19 14:08:31 UTC
From:
To:
Issue:
Currently Debian os-prober support only btrfs root-filesystem on the root of
the btrfs, i.e., ID 5 (FS_TREE).  This makes auto generated grub.cfg to miss
Linux install to btrfs for some Ubuntu and Suse since they put root-system
under @ subvolume.

Existing patch in other distro:
Ubuntu ships patched os-prober 1.77 to address its subvolume use (@ as root-
filesystem) with hardcoded path and very rudamental check for /lib directory.
--------
diff -pruN 1.77/linux-boot-probes/common/50mounted-tests 1.77ubuntu1/linux-
boot-probes/common/50mounted-tests
--- 1.77/linux-boot-probes/common/50mounted-tests       2018-08-10
19:23:18.000000000 +0000
+++ 1.77ubuntu1/linux-boot-probes/common/50mounted-tests        2020-11-02
11:12:51.000000000 +0000
@@ -54,6 +54,19 @@ if type grub-mount >/dev/null 2>&1 && \
        mounted=1
        type="$(grub-probe -d "$partition" -t fs)"
        [ "$type" ] || type=fuseblk
+
+       case "$type" in
+           btrfs)
+                       if [ -x "$tmpmnt/@/lib" ] && \
+                          ! mount --bind "$tmpmnt/@" "$tmpmnt"; then
+                               warn "failed to mount btrfs subvolume @ on
$partition"
+                               if ! umount $tmpmnt; then
+                                       warn "failed to umount $tmpmnt"
+                               fi
+                               mounted=
+                       fi
+                       ;;
+       esac
 fi

 if [ "$mounted" ]; then
------

Discussion:
Since we should offer the choice for the subbvolume name, this hardcoding "@"
here is not elegant.  We should get it as:
-----
RSUBVOL=$(btrfs subvolume get-default $tmpmnt |cut -d' ' -f 9)
-----

Proposed fix:
So updated patch should be more like
---
+       case "$type" in
+           btrfs)
+                       RSUBVOL=$(btrfs subvolume get-default $tmpmnt |cut -d'
' -f 9)
+                       if [ -n "$RSUBVOL" ] && [ -x "$tmpmnt/$RSUBVOL/lib" ]
&& \
+                          ! mount --bind "$tmpmnt/$RSUBVOL" "$tmpmnt"; then
+                               warn "failed to mount btrfs subvolume $RSUBVOL
on $partition"
+                               if ! umount $tmpmnt; then
+                                       warn "failed to umount $tmpmnt"
+                               fi
+                               mounted=
+                       fi
+                       ;;
+       esac
---

This is much simpler patch than ones discussed in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=688336
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=699189

Since we expect any sane person set-default to the root-filesystem.

You could add fall-back with "elif" to use Ubuntu specific hardcoded subvolume
name.

This should make Debian friendly to older Ubuntu with btrfs.

What do you think?

Osamu

#983107#10
Date:
2021-02-20 06:12:47 UTC
From:
To:
Hi,

Knowing bulleseye 11 release is near and this issue may stay, people
may need to work around this issue of btrfs subvolume as root
partition.  Here is my WORKAROUND suggestion.

(Please note that my patch proposal in my original bug report is not-
yet tested well.)

The current Debian Grub2 code at /etc/grub.d/30_os-prober requires to
access some root partion contents from the btrfs ID 5 (FS_TREE) to
function well.  (subvol=/)

If you want to use timeshift (Ubuntu origin?) or snapper (Suse origin
?), they seem to use non-ID-5 subvol for the system install, i.e.,
root-FS.  People use these tools on Debian too.

https://qa.debian.org/popcon-graph.php?packages=timeshift+snapper+snapper-gui+btrbk+btrfsmaintenance+duperemove&show_installed=on&want_legend=on&want_ticks=on&from_date=&to_date=&hlght_date=&date_fmt=%25Y-%25m&beenhere=1


=== WORKAROUND ===

Let's suppose your another linux installation on another formatted in
btrfs at partion "/dev/nvme1n1p1" has its root-FS content in
"subvol=@", please perform followings:
----- $ sudo mount -t btrfs -o subvol=/,defaults /dev/nvme1n1p1 /mnt $ cd /mnt $ sudo ln -sf @/boot boot $ sudo ln -sf @/etc etc $ sudo ln -sf @/usr usr $ sudo ln -sf @/lib lib $ sudo ln -sf @/initrd.img initrd.img $ sudo ln -sf @/vmlinuz vmlinuz $ sudo ln -sf @/initrd.img.old initrd.img.old $ sudo ln -sf @/vmlinuz.old vmlinuz.old $ sudo btrfs subvol set-default /mnt/@ $ cd / $ sudo umount /mnt ----- This assumes you do not have conflicting contents in subvol=/. If you happen to use other subvol for the root-FS content, replace its subvol name in place of "@". === Additional Tips === You can convert a system from EXT2/3/4 to btrfs as follows by booting system from another system on the multiboot set-up. File system conversion is trivial as described in https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3
----- $ sudo fsck.ext3 -f /dev/xxx $ sudo btrfs-convert /dev/xxx $ sudo mount -t btrfs /dev/xxx /mnt ----- You need to make some adjustment to cope with new filesystem ant its new blkid by adjustng at least /etc/fstab and /boot/grub/grub.cfg . Please note UUID of partition changes after btrfs-convert. Please check
----- --- fstab.orig 2021-02-19 14:49:32.768895933 +0900 +++ fstab 2021-02-19 14:49:19.860921576 +0900 @@ -6,6 +6,6 @@ # # <file system> <mount point> <type> <options> <dump> <pass> # / was on /dev/nvme1n1p1 during installation -UUID=491a357c-822f-4f38-b56e-b998baea81a5 / ext4 errors=remount-ro 0 1 +UUID=9be0d928-5892-4cdd-a647-1e8cce937b2e / btrfs defaults 0 1 # /boot/efi was on /dev/nvme0n1p1 during installation ----- Also, you need to update many relevant parts of grub.cfg, too. Roughly as ... ----- --- grub.cfg-orig 2021-02-17 09:32:35.855910912 +0900 +++ grub.cfg 2021-02-19 14:26:12.728005239 +0900 ... -insmod ext2 +insmod btrfs ... - search --no-floppy --fs-uuid --set=root 491a357c-822f-4f38-b56e- b998baea81a5 + search --no-floppy --fs-uuid --set=root 9be0d928-5892-4cdd-a647- 1e8cce937b2e ... - linux /boot/vmlinuz-5.10.0-3-amd64 root=UUID=491a357c-822f- 4f38-b56e-b998baea81a5 ro quiet + linux /boot/vmlinuz-5.10.0-3-amd64 root=UUID=9be0d928-5892- 4cdd-a647-1e8cce937b2e ro quiet ... ----- The UUID change to the "linux ..." line is important since it will affect how this alternative partion update is grub configuration. If you wish to move all this nely migrated system to a subvolume, just snapshot it to "@" or anything you like using "btrfs subvolume snapshot /mnt /@" command. Of course, if your new system is at subvol=@, you need to adjust /etc/fstab as;
----- --- fstab.orig 2021-02-19 14:49:32.768895933 +0900 +++ fstab 2021-02-19 14:49:19.860921576 +0900 @@ -6,6 +6,6 @@ # # <file system> <mount point> <type> <options> <dump> <pass> # / was on /dev/nvme1n1p1 during installation -UUID=491a357c-822f-4f38-b56e-b998baea81a5 / ext4 errors=remount-ro 0 1 +UUID=9be0d928-5892-4cdd-a647-1e8cce937b2e / btrfs subvol=@,defaults 0 1 # /boot/efi was on /dev/nvme0n1p1 during installation ----- And then move on to use WORKAROUND as described in the above.
#983107#15
Date:
2021-02-20 06:19:16 UTC
From:
To:
loop += Nicholas, who has been working on btrfs integration.

Cheers,

#983107#20
Date:
2021-02-20 09:06:54 UTC
From:
To:
Hi,

Speaking of btrfs integration to grub, I noticed some strange cruft
code in /etc/grub.d/30_os-prober.  (grub-common package)  Maybe this is
some backward compatibility feature code to address older os-prober.
We may need to be careful around here or this may be an non-issue.

In /etc/grub.d/30_os-prober
---
BTRFS="`echo ${OS} | cut -d ':' -f 5`"
if [ "x$BTRFS" = "xbtrfs" ]; then
      BTRFSuuid="`echo ${OS} | cut -d ':' -f 6`"
      BTRFSsubvol="`echo ${OS} | cut -d ':' -f 7`"
fi
---

${OS} is basically 's/\s/^/g' on os-prober output.  But there should be
only 4 fields.  So this yield BTRFS="" etc.

So the following code
---
    linux)
      if [ "x$BTRFS" = "xbtrfs" ]; then
         LINUXPROBED="`linux-boot-prober btrfs ${BTRFSuuid}
${BTRFSsubvol}  2> /dev/null | tr ' ' '^' | paste -s -d ' '`"
      else
         LINUXPROBED="`linux-boot-prober ${DEVICE} 2> /dev/null | tr '
' '^' | paste -s -d ' '`"
      fi
---

So always *else* side is executed with the current os-prober.
Current linux-boot-prober in os-prober has its internal FS detection
for btrfs where Ubuntu's patch (and my improvement suggestion patch)
uses and make bind mount subvolume.

#983107#25
Date:
2021-02-20 19:07:37 UTC
From:
To:
Hi Osamu!

§1
Would you like to join/co-found the nascent "Debian btrfs enablement"
team?

In the coming years there will be an increasing number of software that
will need a "get all valid bootable rootfs candidates for a btrfs
volume".  Right now we have GRUB, Debian Rescue (TODO), bootloaders for
ARM (TODO) and in the future systemd-boot (aspirationally TODO
bookworm), in addition to whatever software will be used for "boot
environments".

I wonder if os-prober should be the site for this "return list of
bootable subvolumes" functionality, rather than duplicating the logic in
multiple places?  Cyril?  I'm not sure because os-prober depends on
grub-common, which AFAIK isn't used on ARM.  Do we need a NEW package
for btrfs-related support functions?  Alternatively, btrfs-progs seems
like a logical place for functions like "get a list of all bootable
subvolumes".  Adam, what do you think, is btrfs-progs the right place
for helper functions that return volume layout, noting when a subvolume
is bootable, perhaps as bundled shell scripts?  Could "btrfs subvolume
list" be the right place for this?  And yes, I think this tooling should
ideally exist upstream and not be Debian-specific.

I suspect this functionality may be duplicated in Snapper.  P.S. Hideki
Yamane, who maintains Debian's Snapper package, ACKed installing Debian
to a subvolume (without the use of set-default) a long time ago, but
it's probably time to check in with him again (CCing him).

With truly generic subvolume support for grub that checks for valid
bootable rootfs candidates and creates a menu entry for each candidate
we will have a working prototype of "boot environments" today.  As far
as I know, only SUSE has this (IIRC via Snapper, probably with extra
grub hooks), plus Arch--on an opt-in basis--with grub-btrfs.

Osamu Aoki <osamu@debian.org> writes:

Thank you for bringing this issue to my attention!  Yes, I agree, we
should fix this.
Agreed, that hardcoding is not elegant; although, it does mirror the
hardcoding in their installer, so it's somewhat sensible for the
Ubuntu-specific case.  I imagine their plan was to prototype with
hardcoding, and later add user-defined config to their installer...but
then configurable subvol layout was never added.  My plan is to help
solve the "install to a named subvolume" case for bullseye, and to adapt
partman-LVM to btrfs semantics for bookworm's partman-btrfs, thus
enabling user-configurable subvolume layout in the installer.  IIRC
users installing Debian to btrfs using Calamares already get the
Ubuntu-desktop-style subvolume layout.

To solve the Ubuntu-specific case, I must ask: Do you know if Ubuntu's
upcoming new installer allows configurable subvolume layout?
https://www.omgubuntu.co.uk/2021/02/ubuntu-is-working-on-a-brand-new-installer

If not, I'm not sure there's any urgency or benefit to accommodate what
they would call an officially unsupported custom config (to see why
set-default=@ is unsupported see §3).  I just tested the new "curtin"
Ubuntu server 20.10 installer, which installed directly to subvolid=5,
without creating any subvolumes.

I then tried installing Ubuntu Desktop 20.10, where the btrfs-flavour
partitioning recipe appears to have been replaced with ZFS, but where
manual partitioning still works.  Here Ubuntu still uses subvol=@,
without set-default=@.

So the Ubuntu-specific cases to support are:
1. Hardcoded subvol=@
   * used by desktop installer
2. The new direct installation to subvolid=5
   * used by new "curtin" server installer on 20.10
3. [Optional] Custom user-configured subvolume layouts
4. Tentative NACK to supporting set-default on Ubuntu
   * See §3 for rationale

Ubuntu installations with set-default=@ are nowhere to be found.  Could
this be a Timeshift or Snapper-specific behaviour, or perhaps caused by
some other tool?

This bug might need a severity level bump if Calamares installs rootfs
to a subvolume.  Fedora is likewise affected.  Definitely not RC though.
How does this address the Ubuntu case? ie: "The btrfs-tools command
'set-default' will break Ubuntu's layout"
https://help.ubuntu.com/community/btrfs#The_btrfs-tools_command_.27.27set-default.27.27_will_break_Ubuntu.27s_layout

Given that it's titled "btrfs-tools", this is from an old version, so I
wonder if it's still officially broken?  Even if that's the case, should
we really implement a solution that depends on the expectation that
Ubuntu users ignore their distribution's own documentation?

§4 however, this seems to be a noop for SUSE (set-default=@), and
wherever a sysadmin has used set-default.  Without this call, @ will be
the root of /some/mount/point/suse, which is detected by os-prober,
which will then generate a boot entry without subvol=@, which is fully
functional because set-default=@ makes subvol=@ unnecessary on the
kernel command line.  Or does GRUB still not support btrfs' set-default
subvol?  ie: will GRUB fail to load SUSE's kernel from @/boot/vmlinuz,
which links to @/boot/vmlinuz-$version?

Also, for a given bootable subvolume, isn't it more accurate to parse
@foo/etc/fstab, or @foo/.../snapper-config, when it generates flags into
grub for boot environment support, or /e/default/grub (or equivalent),
and then generate entries for all bootable subvolumes, nested under a
top-level $alternative-distribution GRUB menu entry, (for each partition
containing an $alternative-distribution) rather than using "get-default"
as a indicator for the blessed subvolume?  Unfortunately neither
approach guards against the case where Debian is being used to rescue
another distribution that crashed during an upgrade, before its fstab
could be updated (see §5).

ISTM that without something like an LFS 4.0 that specifies a
standardised layout for btrfs boot environments, supporting
installations on btrfs means that os-probe will eventually need to
support all kinds of distro-specific layouts (or layouts created by a
snapshotting tool :-o)...not to mention the variety of user-configured
layouts btrfs enables.

Our Rescue CD also needs bootable subvolume detection logic, and it
seems like os-prober should use the same logic.  That's why I find this
an interesting bug and topic!  Apologies that this reply is a bit long,
by the way; because I'm invested in solving the Rescue CD case, I'm using
this bug for perspective on how that solution ought to look.

§5
I don't think this check is the right approach for the general case.
For example, a subvolume named "@var" (or @/var on SUSE) will usually
contain "lib", yet will be unbootable.  Likewise, ro subvolumes should
be excluded.  ISTM that the criteria should be 1) rw subvolume, 2)
contains "/sbin/init" rather than contains "/lib".  Set-default=@
appears to only be relevant to SUSE installations.  I also tested a
Fedora 33 installation (defaults to btrfs, with no set-default=rootfs,
and which uses -o subvol=rootfs).

I think it might be a good idea to additionally parse
@subvolume/etc/issue, for each bootable subvolume, so that the grub menu
entry accurately describes the OS version on a subvolume for the pre and
post upgrade case (ie: @20.04 and @, where @ now contains 20.10), but
that's arguably wishlist/nice-to-have if all generated entries can be
safely booted (ie: never booting a @new_rootfs with @old_var nor
@old_rootfs with @new_var.  See §4).

Citation please.  P.S. There's no need to imply that everyone who
doesn't hold your view is a person who is mentally ill with a broken
ability to process logical problems...  Please consider that your claim
is also a logical fallacy; it's an appeal to an imaginary authority
mixed with a personal attack that fails to rationally address the topic,
stated as a conclusion that contains an invalid implied premise.  Also,
no, this isn't an accurate use of "we".

My view is that set-default=@subvolume is an unneeded feature when 1)
Grub already supports a default entry for OS, kernel, and subvolume, 2)
Grub already supports alternative boot entries for OS and kernel version
3) Grub already supports alternative boot entries where rw snapshots can
be booted using subvol=@alt_rootfs (nascent boot environments, or in
SUSE's case, actual boot environments).  4) systemd-boot also supports
all of this, and is just missing auto-generation of loader entries.

Fedora 33 creates "root" (for rootfs) and "home" subvolumes, and also
doesn't use set-default=root...so of the major distributions, it would
seem only SUSE meets your criteria for "sane".

I think we can, and should, do better! :-)  The universal operating
system should avoid special casing, whenever possible.

Supporting set-default=@user_defined for a second Debian installation
(on another partition) seems reasonable, but it isn't something I'm
interested in supporting, which is why I've invited you to join/co-found
the nascent "Debian btrfs enablement" team :-)  Also, we've met, cross
signed GPG keys, and I respect you, so I'm sure collaborating would be a pleasure!

Thank you,
Nicholas

#983107#30
Date:
2021-02-20 19:14:35 UTC
From:
To:
Hi Osamu,

Correction for previous email: Fedora 33 does not use "subvol=rootfs",
it uses "subvol=root".  I'm not sure if they changed this sometime in
the last few years, or if I misremembered and typed "rootfs" by habit.

Reply follows inline:

Osamu Aoki <osamu@debian.org> writes:

<snip>

Doesn't this mean the new subvol=@rootfs (without set-default=@rootfs)
actually fulfills the assumptions required by Timeshift (Ubuntu
subvol=@, no set-default=@) and Snapper?

Hideki, would you please confirm that the changes introduced in
partman-btrfs 53 do not break Snapper?  If so, is it difficult to fix
this in Snapper, and if necessary, do you have time to do so before
bullseye's release or would you like help?  I still worry that Snapper
might have SUSE-specific assumptions in its design (see previous message
at bug #983107 for more context), and if the "subvolid=5" aka "subvol=/"
(Debian pre-bullseye) has hidden a "set-default=@" (SUSE) assumption
that might now manifest as broken functionality in Snapper.
Alternatively, maybe snapper installs a grub config hook?

<snip>

Ahhh.  Is the ext4-to-btrfs via btrfs-convert case the basis for your
preference for the use of subvol=/ or set-default=@foo?

<snip>

Given "You can convert a system from EXT2/3/4 to btrfs as follows by
booting system from another system on the multiboot set-up":

From a live boot with live/rescue media, update-grub already does the
right thing for subvolume renames, or moving rootfs from subvolid=5 to a
named subvolume mounted with "-o subvol=@foo", and I don't know why the
btrfs-convert case is special.  Why not, post-btrfs-convert:

1. umount the just-converted partition
   * suppose it's /dev/sda2
2. mount /dev/sda2 -o subvol=/ /target
3. btrfs sub snap /target /target/@rootfs
4. btrfs sub sync /target && btrfs fi sync /target
   * hasn't been necessary since linux-4.4 IIRC
5. umount /target
6. mount /dev/sda2 -o subvol=@rootfs /target
7. (make bind mounts)
8. chroot to /target as usual
9. update-grub as usual

ISTM that update-grub will do the right thing ("-insmod ext2; +insmod
btrfs") if the partition is unmounted, then remounted before running
update-grub.

Cheers,
Nicholas

#983107#35
Date:
2021-02-21 13:16:27 UTC
From:
To:
Hi Osamu,

Sorry, I think I misunderstood what you meant by "Since we expect any
sane person set-default to the root-filesystem".  I thought you meant
this: since we expect any sane person will run set-default [@] to /

Which really surprised me, because I didn't think anyone had this
position, and I struggled to find reasons to value that perspective
(hence the research!).  Thank you for your patience with that long
reply; it would have been *much* shorter had I not misunderstood.  Now I
think you meant this: since we expect any sane person will set-default
to subvol=/ One good thing that came out of this investigation is
knowing that SUSE uses set-default=@.  Obviously their grub supports
this, because their /boot is the OS partition, and this is btrfs, and it
successfully boots...but I'm not sure if ours does, and if os-prober
will need extra work to support this, or if our grub would benefit from
the hypothetical SUSE patch.

Just now, to be thorough, I checked to see if anything in Debian might
be recommending set-default, or actually executing it, so I did a
codesearch:

https://sources.debian.org/src/systemd/247.3-1/docs/DISCOVERABLE_PARTITIONS.md/?hl=176#L176
 * Recommends using it

https://sources.debian.org/src/btrbk/0.27.1-1.1/doc/FAQ.md/?hl=144#L144
 * Lists it as an equal option to the subvol=@foo method

Luckily we don't have anything unexpected in codesearch that executes
set-default.

Cheers,
Nicholas