Issue:
Currently Debian os-prober support only btrfs root-filesystem on the root of
the btrfs, i.e., ID 5 (FS_TREE). This makes auto generated grub.cfg to miss
Linux install to btrfs for some Ubuntu and Suse since they put root-system
under @ subvolume.
Existing patch in other distro:
Ubuntu ships patched os-prober 1.77 to address its subvolume use (@ as root-
filesystem) with hardcoded path and very rudamental check for /lib directory.
--------
diff -pruN 1.77/linux-boot-probes/common/50mounted-tests 1.77ubuntu1/linux-
boot-probes/common/50mounted-tests
--- 1.77/linux-boot-probes/common/50mounted-tests 2018-08-10
19:23:18.000000000 +0000
+++ 1.77ubuntu1/linux-boot-probes/common/50mounted-tests 2020-11-02
11:12:51.000000000 +0000
@@ -54,6 +54,19 @@ if type grub-mount >/dev/null 2>&1 && \
mounted=1
type="$(grub-probe -d "$partition" -t fs)"
[ "$type" ] || type=fuseblk
+
+ case "$type" in
+ btrfs)
+ if [ -x "$tmpmnt/@/lib" ] && \
+ ! mount --bind "$tmpmnt/@" "$tmpmnt"; then
+ warn "failed to mount btrfs subvolume @ on
$partition"
+ if ! umount $tmpmnt; then
+ warn "failed to umount $tmpmnt"
+ fi
+ mounted=
+ fi
+ ;;
+ esac
fi
if [ "$mounted" ]; then
------
Discussion:
Since we should offer the choice for the subbvolume name, this hardcoding "@"
here is not elegant. We should get it as:
-----
RSUBVOL=$(btrfs subvolume get-default $tmpmnt |cut -d' ' -f 9)
-----
Proposed fix:
So updated patch should be more like
---
+ case "$type" in
+ btrfs)
+ RSUBVOL=$(btrfs subvolume get-default $tmpmnt |cut -d'
' -f 9)
+ if [ -n "$RSUBVOL" ] && [ -x "$tmpmnt/$RSUBVOL/lib" ]
&& \
+ ! mount --bind "$tmpmnt/$RSUBVOL" "$tmpmnt"; then
+ warn "failed to mount btrfs subvolume $RSUBVOL
on $partition"
+ if ! umount $tmpmnt; then
+ warn "failed to umount $tmpmnt"
+ fi
+ mounted=
+ fi
+ ;;
+ esac
---
This is much simpler patch than ones discussed in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=688336
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=699189
Since we expect any sane person set-default to the root-filesystem.
You could add fall-back with "elif" to use Ubuntu specific hardcoded subvolume
name.
This should make Debian friendly to older Ubuntu with btrfs.
What do you think?
Osamu
Hi, Knowing bulleseye 11 release is near and this issue may stay, people may need to work around this issue of btrfs subvolume as root partition. Here is my WORKAROUND suggestion. (Please note that my patch proposal in my original bug report is not- yet tested well.) The current Debian Grub2 code at /etc/grub.d/30_os-prober requires to access some root partion contents from the btrfs ID 5 (FS_TREE) to function well. (subvol=/) If you want to use timeshift (Ubuntu origin?) or snapper (Suse origin ?), they seem to use non-ID-5 subvol for the system install, i.e., root-FS. People use these tools on Debian too. https://qa.debian.org/popcon-graph.php?packages=timeshift+snapper+snapper-gui+btrbk+btrfsmaintenance+duperemove&show_installed=on&want_legend=on&want_ticks=on&from_date=&to_date=&hlght_date=&date_fmt=%25Y-%25m&beenhere=1 === WORKAROUND === Let's suppose your another linux installation on another formatted in btrfs at partion "/dev/nvme1n1p1" has its root-FS content in "subvol=@", please perform followings:----- $ sudo mount -t btrfs -o subvol=/,defaults /dev/nvme1n1p1 /mnt $ cd /mnt $ sudo ln -sf @/boot boot $ sudo ln -sf @/etc etc $ sudo ln -sf @/usr usr $ sudo ln -sf @/lib lib $ sudo ln -sf @/initrd.img initrd.img $ sudo ln -sf @/vmlinuz vmlinuz $ sudo ln -sf @/initrd.img.old initrd.img.old $ sudo ln -sf @/vmlinuz.old vmlinuz.old $ sudo btrfs subvol set-default /mnt/@ $ cd / $ sudo umount /mnt ----- This assumes you do not have conflicting contents in subvol=/. If you happen to use other subvol for the root-FS content, replace its subvol name in place of "@". === Additional Tips === You can convert a system from EXT2/3/4 to btrfs as follows by booting system from another system on the multiboot set-up. File system conversion is trivial as described in https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3----- $ sudo fsck.ext3 -f /dev/xxx $ sudo btrfs-convert /dev/xxx $ sudo mount -t btrfs /dev/xxx /mnt ----- You need to make some adjustment to cope with new filesystem ant its new blkid by adjustng at least /etc/fstab and /boot/grub/grub.cfg . Please note UUID of partition changes after btrfs-convert. Please check----- --- fstab.orig 2021-02-19 14:49:32.768895933 +0900 +++ fstab 2021-02-19 14:49:19.860921576 +0900 @@ -6,6 +6,6 @@ # # <file system> <mount point> <type> <options> <dump> <pass> # / was on /dev/nvme1n1p1 during installation -UUID=491a357c-822f-4f38-b56e-b998baea81a5 / ext4 errors=remount-ro 0 1 +UUID=9be0d928-5892-4cdd-a647-1e8cce937b2e / btrfs defaults 0 1 # /boot/efi was on /dev/nvme0n1p1 during installation ----- Also, you need to update many relevant parts of grub.cfg, too. Roughly as ... ----- --- grub.cfg-orig 2021-02-17 09:32:35.855910912 +0900 +++ grub.cfg 2021-02-19 14:26:12.728005239 +0900 ... -insmod ext2 +insmod btrfs ... - search --no-floppy --fs-uuid --set=root 491a357c-822f-4f38-b56e- b998baea81a5 + search --no-floppy --fs-uuid --set=root 9be0d928-5892-4cdd-a647- 1e8cce937b2e ... - linux /boot/vmlinuz-5.10.0-3-amd64 root=UUID=491a357c-822f- 4f38-b56e-b998baea81a5 ro quiet + linux /boot/vmlinuz-5.10.0-3-amd64 root=UUID=9be0d928-5892- 4cdd-a647-1e8cce937b2e ro quiet ... ----- The UUID change to the "linux ..." line is important since it will affect how this alternative partion update is grub configuration. If you wish to move all this nely migrated system to a subvolume, just snapshot it to "@" or anything you like using "btrfs subvolume snapshot /mnt /@" command. Of course, if your new system is at subvol=@, you need to adjust /etc/fstab as;----- --- fstab.orig 2021-02-19 14:49:32.768895933 +0900 +++ fstab 2021-02-19 14:49:19.860921576 +0900 @@ -6,6 +6,6 @@ # # <file system> <mount point> <type> <options> <dump> <pass> # / was on /dev/nvme1n1p1 during installation -UUID=491a357c-822f-4f38-b56e-b998baea81a5 / ext4 errors=remount-ro 0 1 +UUID=9be0d928-5892-4cdd-a647-1e8cce937b2e / btrfs subvol=@,defaults 0 1 # /boot/efi was on /dev/nvme0n1p1 during installation ----- And then move on to use WORKAROUND as described in the above.
loop += Nicholas, who has been working on btrfs integration. Cheers,
Hi,
Speaking of btrfs integration to grub, I noticed some strange cruft
code in /etc/grub.d/30_os-prober. (grub-common package) Maybe this is
some backward compatibility feature code to address older os-prober.
We may need to be careful around here or this may be an non-issue.
In /etc/grub.d/30_os-prober
---
BTRFS="`echo ${OS} | cut -d ':' -f 5`"
if [ "x$BTRFS" = "xbtrfs" ]; then
BTRFSuuid="`echo ${OS} | cut -d ':' -f 6`"
BTRFSsubvol="`echo ${OS} | cut -d ':' -f 7`"
fi
---
${OS} is basically 's/\s/^/g' on os-prober output. But there should be
only 4 fields. So this yield BTRFS="" etc.
So the following code
---
linux)
if [ "x$BTRFS" = "xbtrfs" ]; then
LINUXPROBED="`linux-boot-prober btrfs ${BTRFSuuid}
${BTRFSsubvol} 2> /dev/null | tr ' ' '^' | paste -s -d ' '`"
else
LINUXPROBED="`linux-boot-prober ${DEVICE} 2> /dev/null | tr '
' '^' | paste -s -d ' '`"
fi
---
So always *else* side is executed with the current os-prober.
Current linux-boot-prober in os-prober has its internal FS detection
for btrfs where Ubuntu's patch (and my improvement suggestion patch)
uses and make bind mount subvolume.
Hi Osamu! §1 Would you like to join/co-found the nascent "Debian btrfs enablement" team? In the coming years there will be an increasing number of software that will need a "get all valid bootable rootfs candidates for a btrfs volume". Right now we have GRUB, Debian Rescue (TODO), bootloaders for ARM (TODO) and in the future systemd-boot (aspirationally TODO bookworm), in addition to whatever software will be used for "boot environments". I wonder if os-prober should be the site for this "return list of bootable subvolumes" functionality, rather than duplicating the logic in multiple places? Cyril? I'm not sure because os-prober depends on grub-common, which AFAIK isn't used on ARM. Do we need a NEW package for btrfs-related support functions? Alternatively, btrfs-progs seems like a logical place for functions like "get a list of all bootable subvolumes". Adam, what do you think, is btrfs-progs the right place for helper functions that return volume layout, noting when a subvolume is bootable, perhaps as bundled shell scripts? Could "btrfs subvolume list" be the right place for this? And yes, I think this tooling should ideally exist upstream and not be Debian-specific. I suspect this functionality may be duplicated in Snapper. P.S. Hideki Yamane, who maintains Debian's Snapper package, ACKed installing Debian to a subvolume (without the use of set-default) a long time ago, but it's probably time to check in with him again (CCing him). With truly generic subvolume support for grub that checks for valid bootable rootfs candidates and creates a menu entry for each candidate we will have a working prototype of "boot environments" today. As far as I know, only SUSE has this (IIRC via Snapper, probably with extra grub hooks), plus Arch--on an opt-in basis--with grub-btrfs. Osamu Aoki <osamu@debian.org> writes: Thank you for bringing this issue to my attention! Yes, I agree, we should fix this. Agreed, that hardcoding is not elegant; although, it does mirror the hardcoding in their installer, so it's somewhat sensible for the Ubuntu-specific case. I imagine their plan was to prototype with hardcoding, and later add user-defined config to their installer...but then configurable subvol layout was never added. My plan is to help solve the "install to a named subvolume" case for bullseye, and to adapt partman-LVM to btrfs semantics for bookworm's partman-btrfs, thus enabling user-configurable subvolume layout in the installer. IIRC users installing Debian to btrfs using Calamares already get the Ubuntu-desktop-style subvolume layout. To solve the Ubuntu-specific case, I must ask: Do you know if Ubuntu's upcoming new installer allows configurable subvolume layout? https://www.omgubuntu.co.uk/2021/02/ubuntu-is-working-on-a-brand-new-installer If not, I'm not sure there's any urgency or benefit to accommodate what they would call an officially unsupported custom config (to see why set-default=@ is unsupported see §3). I just tested the new "curtin" Ubuntu server 20.10 installer, which installed directly to subvolid=5, without creating any subvolumes. I then tried installing Ubuntu Desktop 20.10, where the btrfs-flavour partitioning recipe appears to have been replaced with ZFS, but where manual partitioning still works. Here Ubuntu still uses subvol=@, without set-default=@. So the Ubuntu-specific cases to support are: 1. Hardcoded subvol=@ * used by desktop installer 2. The new direct installation to subvolid=5 * used by new "curtin" server installer on 20.10 3. [Optional] Custom user-configured subvolume layouts 4. Tentative NACK to supporting set-default on Ubuntu * See §3 for rationale Ubuntu installations with set-default=@ are nowhere to be found. Could this be a Timeshift or Snapper-specific behaviour, or perhaps caused by some other tool? This bug might need a severity level bump if Calamares installs rootfs to a subvolume. Fedora is likewise affected. Definitely not RC though. How does this address the Ubuntu case? ie: "The btrfs-tools command 'set-default' will break Ubuntu's layout" https://help.ubuntu.com/community/btrfs#The_btrfs-tools_command_.27.27set-default.27.27_will_break_Ubuntu.27s_layout Given that it's titled "btrfs-tools", this is from an old version, so I wonder if it's still officially broken? Even if that's the case, should we really implement a solution that depends on the expectation that Ubuntu users ignore their distribution's own documentation? §4 however, this seems to be a noop for SUSE (set-default=@), and wherever a sysadmin has used set-default. Without this call, @ will be the root of /some/mount/point/suse, which is detected by os-prober, which will then generate a boot entry without subvol=@, which is fully functional because set-default=@ makes subvol=@ unnecessary on the kernel command line. Or does GRUB still not support btrfs' set-default subvol? ie: will GRUB fail to load SUSE's kernel from @/boot/vmlinuz, which links to @/boot/vmlinuz-$version? Also, for a given bootable subvolume, isn't it more accurate to parse @foo/etc/fstab, or @foo/.../snapper-config, when it generates flags into grub for boot environment support, or /e/default/grub (or equivalent), and then generate entries for all bootable subvolumes, nested under a top-level $alternative-distribution GRUB menu entry, (for each partition containing an $alternative-distribution) rather than using "get-default" as a indicator for the blessed subvolume? Unfortunately neither approach guards against the case where Debian is being used to rescue another distribution that crashed during an upgrade, before its fstab could be updated (see §5). ISTM that without something like an LFS 4.0 that specifies a standardised layout for btrfs boot environments, supporting installations on btrfs means that os-probe will eventually need to support all kinds of distro-specific layouts (or layouts created by a snapshotting tool :-o)...not to mention the variety of user-configured layouts btrfs enables. Our Rescue CD also needs bootable subvolume detection logic, and it seems like os-prober should use the same logic. That's why I find this an interesting bug and topic! Apologies that this reply is a bit long, by the way; because I'm invested in solving the Rescue CD case, I'm using this bug for perspective on how that solution ought to look. §5 I don't think this check is the right approach for the general case. For example, a subvolume named "@var" (or @/var on SUSE) will usually contain "lib", yet will be unbootable. Likewise, ro subvolumes should be excluded. ISTM that the criteria should be 1) rw subvolume, 2) contains "/sbin/init" rather than contains "/lib". Set-default=@ appears to only be relevant to SUSE installations. I also tested a Fedora 33 installation (defaults to btrfs, with no set-default=rootfs, and which uses -o subvol=rootfs). I think it might be a good idea to additionally parse @subvolume/etc/issue, for each bootable subvolume, so that the grub menu entry accurately describes the OS version on a subvolume for the pre and post upgrade case (ie: @20.04 and @, where @ now contains 20.10), but that's arguably wishlist/nice-to-have if all generated entries can be safely booted (ie: never booting a @new_rootfs with @old_var nor @old_rootfs with @new_var. See §4). Citation please. P.S. There's no need to imply that everyone who doesn't hold your view is a person who is mentally ill with a broken ability to process logical problems... Please consider that your claim is also a logical fallacy; it's an appeal to an imaginary authority mixed with a personal attack that fails to rationally address the topic, stated as a conclusion that contains an invalid implied premise. Also, no, this isn't an accurate use of "we". My view is that set-default=@subvolume is an unneeded feature when 1) Grub already supports a default entry for OS, kernel, and subvolume, 2) Grub already supports alternative boot entries for OS and kernel version 3) Grub already supports alternative boot entries where rw snapshots can be booted using subvol=@alt_rootfs (nascent boot environments, or in SUSE's case, actual boot environments). 4) systemd-boot also supports all of this, and is just missing auto-generation of loader entries. Fedora 33 creates "root" (for rootfs) and "home" subvolumes, and also doesn't use set-default=root...so of the major distributions, it would seem only SUSE meets your criteria for "sane". I think we can, and should, do better! :-) The universal operating system should avoid special casing, whenever possible. Supporting set-default=@user_defined for a second Debian installation (on another partition) seems reasonable, but it isn't something I'm interested in supporting, which is why I've invited you to join/co-found the nascent "Debian btrfs enablement" team :-) Also, we've met, cross signed GPG keys, and I respect you, so I'm sure collaborating would be a pleasure! Thank you, Nicholas
Hi Osamu,
Correction for previous email: Fedora 33 does not use "subvol=rootfs",
it uses "subvol=root". I'm not sure if they changed this sometime in
the last few years, or if I misremembered and typed "rootfs" by habit.
Reply follows inline:
Osamu Aoki <osamu@debian.org> writes:
<snip>
Doesn't this mean the new subvol=@rootfs (without set-default=@rootfs)
actually fulfills the assumptions required by Timeshift (Ubuntu
subvol=@, no set-default=@) and Snapper?
Hideki, would you please confirm that the changes introduced in
partman-btrfs 53 do not break Snapper? If so, is it difficult to fix
this in Snapper, and if necessary, do you have time to do so before
bullseye's release or would you like help? I still worry that Snapper
might have SUSE-specific assumptions in its design (see previous message
at bug #983107 for more context), and if the "subvolid=5" aka "subvol=/"
(Debian pre-bullseye) has hidden a "set-default=@" (SUSE) assumption
that might now manifest as broken functionality in Snapper.
Alternatively, maybe snapper installs a grub config hook?
<snip>
Ahhh. Is the ext4-to-btrfs via btrfs-convert case the basis for your
preference for the use of subvol=/ or set-default=@foo?
<snip>
Given "You can convert a system from EXT2/3/4 to btrfs as follows by
booting system from another system on the multiboot set-up":
From a live boot with live/rescue media, update-grub already does the
right thing for subvolume renames, or moving rootfs from subvolid=5 to a
named subvolume mounted with "-o subvol=@foo", and I don't know why the
btrfs-convert case is special. Why not, post-btrfs-convert:
1. umount the just-converted partition
* suppose it's /dev/sda2
2. mount /dev/sda2 -o subvol=/ /target
3. btrfs sub snap /target /target/@rootfs
4. btrfs sub sync /target && btrfs fi sync /target
* hasn't been necessary since linux-4.4 IIRC
5. umount /target
6. mount /dev/sda2 -o subvol=@rootfs /target
7. (make bind mounts)
8. chroot to /target as usual
9. update-grub as usual
ISTM that update-grub will do the right thing ("-insmod ext2; +insmod
btrfs") if the partition is unmounted, then remounted before running
update-grub.
Cheers,
Nicholas
Hi Osamu, Sorry, I think I misunderstood what you meant by "Since we expect any sane person set-default to the root-filesystem". I thought you meant this: since we expect any sane person will run set-default [@] to / Which really surprised me, because I didn't think anyone had this position, and I struggled to find reasons to value that perspective (hence the research!). Thank you for your patience with that long reply; it would have been *much* shorter had I not misunderstood. Now I think you meant this: since we expect any sane person will set-default to subvol=/ One good thing that came out of this investigation is knowing that SUSE uses set-default=@. Obviously their grub supports this, because their /boot is the OS partition, and this is btrfs, and it successfully boots...but I'm not sure if ours does, and if os-prober will need extra work to support this, or if our grub would benefit from the hypothetical SUSE patch. Just now, to be thorough, I checked to see if anything in Debian might be recommending set-default, or actually executing it, so I did a codesearch: https://sources.debian.org/src/systemd/247.3-1/docs/DISCOVERABLE_PARTITIONS.md/?hl=176#L176 * Recommends using it https://sources.debian.org/src/btrbk/0.27.1-1.1/doc/FAQ.md/?hl=144#L144 * Lists it as an equal option to the subvol=@foo method Luckily we don't have anything unexpected in codesearch that executes set-default. Cheers, Nicholas