Hi, when creating a bookworm schroot e.g. with sbuild-createchroot --merged-usr bookworm /var/lib/schroot/schroots/bookworm-amd64-sbuild http://localhost:3142/deb.debian.org/debian/ the resulting schroot has an unmerged /usr. This creates some interesting problems, like upgrade tests failing. Or in theory it's possible to successfully build and test packages that would be broken when installed on a "real" bookworm. Is there a reason for this? At least for me this break user expectation, as a merged /usr is required per release notes [0]. I've tentatively set this to severity serious, as I believe this could have been an oversight. If this is by design, I think it's best if sbuild-createchroot would very prominently say so on the end of the build run. Thanks in advance! Regards, Lee [0] https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.html#a-merged-usr-is-now-required
Control: severity -1 normal For bookworm, the expectation was (and is!) that builds are performed on unmerged chroots. This was the only "supported" usecase for having an unmerged install on bookworm. And yes, upgrading these is not supported. Chris
Hi Lee, Quoting Chris Hofstaedtler (2024-08-29 02:33:26) we talked in IRC #debian-devel and decided that this bug is actually a documentation bug. I filed this MR against debootstrap: https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/125 Do you think that this change addresses your concerns? Thanks! cheers, josch
Lee Garret wrote: I'm doing this: sbuild-createchroot --merged-usr bookworm /tmp/mychroot http://deb.debian.org/debian and it works for me. I get a usrmerged system that way. I'm using sbuild and debootstrap in bookworm (i.e. 0.85.0 and 1.0.128+nmu2+deb12u1). Johannes Schauer Marin Rodrigues wrote: I don't. If it works in bookworm, I don't see a good reason why it should suddenly stop working. In fact, without --merged-usr, a non usrmerged is created but having the usr-is-merged package installed. That's very bad and I hope somebody can address it. Thanks.
Do you also get an unmerged system having the usr-is-merged package installed? That would explain that upgrades do not work. Chris Hofstaedtler wrote: I don't see any good reason why a chroot created by a (non-buggy version of) sbuild or debootstrap could not be upgraded. We need that to work to be able to test things. Thanks.
Hi Santiago, Quoting Santiago Vila (2024-08-29 14:02:43) I can confirm this on bookworm. So now there are multiple bugs. One is, that on bookworm, --merged-usr --variant=buildd is creating a merged-/usr chroot. The other bug is that in unstable, there is nothing in the debootstrap output which tells the user that their bookworm buildd chroot will not be merged despite that flag. We have to talk to more people to figure out how this is supposed to work... Thanks! cheers, josch
Hi Lee, there seems to be some confusion (also from my end) -- read further below. Quoting Lee Garrett (2024-08-28 20:35:06) Your package selection suggests that you are on bookworm. I am as well. As far as I can see, I have the same package versions installed for sbuild and debootstrap as you do. $ apt-cache policy sbuild debootstrap sbuild: Installed: 0.85.0 [...] debootstrap: Installed: 1.0.128+nmu2+deb12u1 But I am unable to reproduce this. It seems that debootstrap should indeed respect the --merged-usr flag even when building bookworm buildd chroots. I'm running this and I fail to reproduce your issue: $ sudo sbuild-createchroot --merged-usr bookworm tmp/debian-bookworm [...] $ ls -lha ~/tmp/debian-bookworm total 144K drwxr-xr-x 18 root root 4.0K Aug 29 14:40 . drwxrwxrwt 156 josch josch 72K Aug 29 14:37 .. lrwxrwxrwx 1 root root 7 Aug 29 14:39 bin -> usr/bin drwxr-xr-x 2 root root 4.0K Mar 29 18:20 boot drwxrws--- 2 sbuild sbuild 4.0K Aug 29 14:40 build drwxr-xr-x 4 root root 4.0K Aug 29 14:39 dev drwxr-xr-x 31 root root 4.0K Aug 29 14:40 etc drwxr-xr-x 2 root root 4.0K Mar 29 18:20 home lrwxrwxrwx 1 root root 7 Aug 29 14:39 lib -> usr/lib drwxr-xr-x 2 root root 4.0K Aug 29 14:39 media drwxr-xr-x 2 root root 4.0K Aug 29 14:39 mnt drwxr-xr-x 2 root root 4.0K Aug 29 14:39 opt drwxr-xr-x 2 root root 4.0K Mar 29 18:20 proc drwx------ 2 root root 4.0K Aug 29 14:39 root drwxr-xr-x 4 root root 4.0K Aug 29 14:39 run lrwxrwxrwx 1 root root 8 Aug 29 14:39 sbin -> usr/sbin drwxr-xr-x 2 root root 4.0K Aug 29 14:39 srv drwxr-xr-x 2 root root 4.0K Mar 29 18:20 sys drwxrwxrwt 2 root root 4.0K Aug 29 14:40 tmp drwxr-xr-x 11 root root 4.0K Aug 29 14:39 usr drwxr-xr-x 11 root root 4.0K Aug 29 14:39 var This seems to be the intended behaviour. Can you try again? Thanks! cheers, josch
Upgrading a non-merged-/usr bookworm chroot with bookworm security/stable
updates is OK, you can do that.
Upgrading a non-merged-/usr bookworm chroot to trixie/sid is explicitly
not supported. This transition had 4 stages:
1. Only non-merged-/usr is supported (approximately Debian <= 9)
2. Both merged-/usr and non-merged-/usr are supported (Debian 10 and 11)
3. In general all installations should be merged-/usr, but disposable
chroots/containers used for buildds and QA can still be forced to be
non-merged-/usr, and upgrades from Debian 11 also need to work, so
every package must still be installable either way (Debian 12)
4. Everything must be merged-/usr and packages are allowed to assume it
(Debian 13)
We have to draw a line somewhere for the break between (3.) and (4.), and
as per TC resolution #994388 in 2021 [1], the place where that line was
drawn is, in theory, trixie/sid immediately after the bookworm release.
(In practice, it was unsupported but sort-of-worked until
usrmerge (>= 38) and then base-files (>= 13.3) made merged-/usr genuinely
unavoidable.)
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#110
If you are doing upgrade tests from bookworm to trixie/sid (piuparts
or similar), TC resolution #994388 says you can't validly use a
non-merged-/usr chroot for that. Please use a separate merged-/usr
base chroot or tarball for that. It's probably also better to use
--variant=minbase rather than --variant=buildd for this sort of tests,
which is another reason why sbuild-createchroot is not the most appropriate
tool for purposes other than sbuild.
smcv
have a higher precedence than debootstrap's default behaviour, which is dependent on the suite and variant. For those who are reading the debootstrap source, it's a tristate: the default is MERGED_USR="" which allows automatic selection based on the suite and variant; --merged-usr sets MERGED_USR=yes; and --no-merged-usr sets MERGED_USR=no. The intended truth table *without* --[no-]merged-usr is: https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/93#note_410656 and this should happen in a fully-updated installation of Debian 11 or later as the host system (Debian 11.0 and 12.0 didn't do this, debootstrap's behaviour was changed in a subsequent point release). When --[no-]merged-usr explicitly selects a layout that is allowed for a particular suite, the expected result is that layout. In particular, for Debian 10, 11 and 12, --no-merged-usr should produce a non-merged-/usr chroot and --merged-usr should produce a merged-/usr chroot, regardless of variant. If --[no-]merged-usr is used to select a layout that is *not* allowed for a particular suite (for example trixie with --no-merged-usr), I would say that the expected result is undefined: it might just fail, or it might produce a non-functional chroot, or it might produce a working chroot that does not have the requested layout. Which of those three possibilities actually happens, and whether there is a warning, is a quality-of-implementation issue. smcv
working as designed. As part of the transition from "merged-/usr is
opt-in" to "merged-/usr is the default" in Debian 12, the usr-is-merged
package was repurposed from meaning "/usr is merged" (as it did in
Debian 11) to meaning: either /usr is merged, or the operator of the
chroot/container has explicitly opted-out of merged-/usr (which implies
also opting out from the ability to upgrade to trixie) by creating the
flag file /etc/unsupported-skip-usrmerge-conversion.
This opt-out was technically possible from usrmerge 27 to 37 inclusive. It
is no longer possible in trixie/sid, in which usr-is-merged (>= 38) and
base-files (>= 13.3) enforce a merged-/usr layout.
For Debian 12, the TC advised in resolution #994388 [1] that bugs of the
same category as #993675 [2] should be treated as RC, consistent with the
earlier resolution #914897 [3] (see §(Building packages) in #994388).
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#110
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993675
[3] https://lists.debian.org/debian-devel-announce/2019/03/msg00001.html
However, as a courtesy to users upgrading packages with that category of
bug from Debian 11, official buildds for Debian 12 and older use a
non-merged-/usr build chroot, because that is the layout that is most
likely to result in a .deb that works equally well on either merged-/usr
or non-merged-/usr systems during the upgrade from Debian 11 to 12.
smcv
My use case is that sometimes I need to know in which date some package stopped failing to build in testing because an unknown build-dependency finally propagated to testing. For that to happen I need either: (A) Create a usr-merged buildd chroot of bookworm, then upgrade to the desired trixie at some unspecified date using snapshot.debian.org. (B) Create a non-usr-merged buildd chroot of bookworm, usr-merge it, then upgrade to snapshot.debian.org. What is the route that I'm supposed to follow? Do you disagree that at least one of those two options should be fully supported? Thanks.
Hi Santiago, Quoting Santiago Vila (2024-08-30 14:25:39) Why the upgrade step? Why do you not create a trixie buildd chroot from a given snapshot.d.o timestamp directly? Do you do that because you also want to test the upgrade scenario? If you do not need the upgrade part, there is the tool debootsnap which creates a chroot with precisely the package versions you require without upgrading, for example from a buildinfo file. There is also the tool "debbisect" which I usually use to track down when a package started to FTBFS in unstable (but you can use it for testing all the same). Just recently I ran this to track down #1077452: $ DEBIAN_BISECT_SRCPKG=picolibc debbisect --cache=./cache 2024-02-26 2024-08-27 /usr/share/doc/devscripts/examples/debbisect_buildsrc.sh That this is possible does of course not invalidate your question whether either of your options above should be fully supported. I just wanted to let you know that maybe there is a practical solution for your use-case that works better than what you are currently doing. Thanks! cheers, josch
Hey josch, Oof, indeed! What a hell of a mistake of mine. It seems that I did a copy/paste mistake in the original bug report. To clarify: I had an issue with sbuild-createchroot bookworm /var/lib/schroot/schroots/bookworm-amd64-sbuild http://localhost:3142/deb.debian.org/debian/ creating an unmerged /usr by default. After some discussion here and on #debian-devel I understand that it is intended behaviour. It is indeed documented in the man page of sbuild-createchroot that it is the default. However, the tripfall is that my expectation (and likely of other's not aware of the background) is bookworm being merged /usr, as that's enforced on "normal" installations, so I didn't check this specific flag. There is also no command output hinting at it. As such I'll downgrade the bug, and send a PR for sbuild-createchroot to output prominently that it builds a non-default unmerged /usr. Thanks! Lee
Sure, that makes sense.
You are meant to be able to do this with
debootstrap --merged-usr --variant=buildd bookworm TARGET
possibly wrapped as `sbuild-createchroot --merged-usr ...`. That's meant
to take precedence over the suite- and variant-dependent default behaviour.
Lee reported that this actually didn't always create a merged-/usr chroot,
but josch was unable to reproduce that problem. If you can find a
reproducer for `debootstrap --merged-usr ... bookworm TARGET` creating a
non-merged-/usr chroot at TARGET, that would be a valid bug, in which case
please describe the steps-to-reproduce in more detail (ideally starting
from a throwaway VM of a specified suite).
I don't think you can do this in any supported way, because a
non-usr-merged bookworm chroot has already opted-out from being validly
upgradable by creating the /etc/unsupported-skip-usrmerge-conversion
flag file. It might be possible to force it by deleting the flag file
and then explicitly installing usrmerge.
Either your (A) above, or:
(C) as josch suggested, bootstrap a trixie chroot directly from a former
state of trixie, as available from snapshot.d.o.
(D) start from a bookworm minbase instead of a bookworm buildd chroot
(minbase is merged-/usr by default anyway), and then add build-essential
to it after you upgrade (or even let sbuild install it on-demand).
I would personally go for what josch said, (C) above, because creating
a new chroot/container will be less susceptible to configuration drift
(more reproducible) than upgrading an old chroot/container.
smcv
I'm avoiding full-quoting smcv's previous mail, because (with my usrmerge-not-really-hat on) I fully agree with it. I don't think there is anything in sbuild to fix. About the general points that apply to usrmerge in Debian, ISTM the train has left the station before bookworm was released. Chris
I think it's just a minor documentation bug. IMHO when running `sbuild-createchroot bookworm /var/lib/schroot/schroots/bookworm-amd64-sbuild http://localhost:3142/deb.debian.org/debian/` it should output a warning/info that it created an unmerged-/usr. This is a tripfall because setting "$run_piuparts = 1;" in .sbuildrc will make sbuild use that schroot to do upgrade tests with piuparts, which will fail (a separate bug though). I spent a few hours trying to figure out what I did wrong creating my build environment and would like to avoid any newcomers running into the same issue. I looked at the code at https://salsa.debian.org/installer-team/debootstrap/-/blob/master/functions?ref_type=heads#L1513, and IMHO that line should be removed, so it will unconditionally emit this warning.