Dear maintainer,

It is regularly claimed that mmdebstrap should be used instead of
debootstrap. However contrary to debootstrap [1], it does not install
ca-certificates when a https mirror is used. This makes the generated
chroot unusable.

Regards
Aurelien

[1] https://sources.debian.org/src/debootstrap/1.0.139/scripts/debian-common/?hl=55#L55

Hi,

Quoting Aurelien Jarno (2025-01-01 19:44:16)

how are the regular claims of some relevant to this report?

How would your preferred solution look like?

A proper solution would need to call "apt-get indextargets" on all invocations
to figure out whether a https mirror was used, slowing down even non-https
runs.

Are you using mmdebstrap from the terminal or from a script? In what use-case
do you come across this issue?

Thanks!

cheers, josch

Hi,

This is to give some context about why I explored replacing debootstrap
by mmdebstrap on the build daemons.

It started with the removal of usrmerge from the archive (see #1088212),
which broke the generation of the chroots on the build daemons using
debootstrap. People on IRC just said that mmdebstrap should be used
instead debootstrap. In the meantime an empty package has been
reuploaded until debootstrap is fixed in stable.

From what I understand the only protocol supported by apt that need
ca-certificates is the https case, so it should be enough to just check
the URL like debootstrap is doing.

I used it from a script, as said above to replace the one used by the
buildds. The script can be changed easily to pass
--include=ca-certificates, but I believe that users trying to reproduce
the build daemon setup might encounter the same issue.

This can be reproduced that way:

| # mmdebstrap --variant=buildd sid sid.dir https://deb.debian.org/debian
| I: automatically chosen mode: root
| I: chroot architecture amd64 is equal to the host's architecture
| I: finding correct signed-by value...
| done
| I: automatically chosen format: directory
| I: running apt-get update...
| done
| I: downloading packages with apt...
| done
| I: extracting archives...
| done
| I: installing essential packages...
| done
| I: installing remaining packages inside the chroot...
| done
| done
| I: cleaning package lists and apt cache...
| done
| done
| I: success in 11.0252 seconds
| # chroot sid.dir
| # apt-get update
| Ign:1 https://deb.debian.org/debian sid InRelease
| Ign:1 https://deb.debian.org/debian sid InRelease
| Ign:1 https://deb.debian.org/debian sid InRelease
| Err:1 https://deb.debian.org/debian sid InRelease
|   SSL connection failed: error:80000002:system library::No such file or directory / Success [IP: 2a04:4e42:6a::644 443]
| Reading package lists... Done
| W: Failed to fetch https://deb.debian.org/debian/dists/sid/InRelease  SSL connection failed: error:80000002:system library::No such file or directory / Success [IP: 2a04:4e42:6a::644 443]
| W: Some index files failed to download. They have been ignored, or old ones used instead.
| #

Regards
Aurelien

Hi,

Quoting Aurelien Jarno (2025-01-02 12:41:24)

hmmm... why would you do that? Is debootstrap doing something that you don't
like? Is mmdebstrap adding anything to the table that makes it superior to
debootstrap for that use-case?

Okay, I see. I think some people are a bit too excited about mmdebstrap some
times. It has its place and so has debootstrap. I'd advocate for replacing
debootstrap by mmdebstrap everywhere. They are similar but still different. For
example, the goal of mmdebstrap is to go away. In an ideal world (and Julian is
experimenting with this right now), there would be an "apt bootstrap" command
which creates an initial chroot for you. In mmdebstrap I explore ways to do
that as mmdebstrap itself is just a wrapper around apt, passing it the right
options. In essence, you can replace mmdebstrap by doing this:

mkdir -p "$2/etc/apt" "$2/var/cache" "$2/var/lib"
cat << END > "$2/apt.conf"
Apt::Architecture "$(dpkg --print-architecture)";
Apt::Architectures "$(dpkg --print-architecture)";
Dir "$(cd "$2" && pwd)";
Dir::Etc::Trusted "$(eval "$(apt-config shell v Dir::Etc::Trusted/f)"; printf "$v")";
Dir::Etc::TrustedParts "$(eval "$(apt-config shell v Dir::Etc::TrustedParts/d)"; printf "$v")";
END
echo "deb http://deb.debian.org/debian/ $1 main" > "$2/etc/apt/sources.list"
APT_CONFIG="$2/apt.conf" apt-get update
APT_CONFIG="$2/apt.conf" apt-get --yes --download-only install '?essential'
for f in "$2"/var/cache/apt/archives/*.deb; do dpkg-deb --extract "$f" "$2"; done
chroot "$2" sh -c "dpkg --install --force-depends /var/cache/apt/archives/*.deb"

This is not much. You could even replace debootstrap on the build daemons with something like the above, just add one additional step installing build-essential.

Unfortunately, in contrast to debootstrap, there is not "the URL". One of the
features of mmdebstrap over debootstrap is, that it supports multiple mirrors.
With debootstrap, you get only a single mirror url, so detecting https is
trivial. With mmdebstrap, not only do you have apt deb822 sources.list as
input, you also have hooks which can do arbitrary things. To cover these cases,
you'd need to run:

    apt-get indextargets --format '$(URI)'

This would probably happen at the very end, just before the apt cache is
cleaned up.

From the feedback I get from other users, mmdebstrap is usually used from a
script. This is also how I mostly am using it. This feature is not for users of
mmdebstrap from a script because those can easily add --include=ca-certificates
as you said. This is for users who manually run mmdebstrap from the command
line and who want to automatically create a chroot with the right set of
packages without having to type too much.

My problem with this is indeed that implementing this properly means that
apt-get indextargets will be called by default. One of the main reasons I found
that people recommend mmdebstrap is, that it is really fast. I also like this
feature and would like to keep it that way. Unfortunately, running apt-get
indextargets on my machine takes half a second which is considerable if
creating the whole chroot takes 9 seconds. That's more than 5% runtime
increase.

Also, if this is added by default, there needs to be a way to disable this
feature. That would probably be a new skip option like
--skip=customize/auto-mirror.

Another possibility would be to create a hook that does the right thing and
then the user would have to add:

Hi Josch,

In the long term, once we can get rid of bullseye and bookworm, or
alternatively fix their package so that they all build in unshare mode,
we will like to also generate the chroot in unshare mode.

mmdebstrap is not the only solution, sbuild-createchroot can also just
do that, and is closer than the current solution. OTOH mmdebstrap is
closer to what is recommended for reproducing the build daemon setup.

In any case testing it the build daemon context is always interesting to
be able to answer people asking why we don't use it on the build
daemons.

Thanks for the detailed explanation, I never look at mmdebstrap
internals, it's actually quite interesting to see how existing code,
apt, is reused there.

Ok, the key point here is the hooks, I understand better now. That said,
from what I understand the hooks are executed outside of the
chroot/unshare, with the host apt, which has access to the CA
certificates, so it should be enough to do determine that as the last
step once all the hooks have been executed. But without knowing the
mmdebstrap internals, I probably miss a point.

Tests here shows that's more around 0.6% (0.08 sec over 13 sec), but i
guess it depends on cpu/network/storage speed/ratio and many other
factors. And at then end 1% plus 1% plus 1%... can end-up with something
slow at at the end.

Yes, I fully agree that this is not really an issue for scripts, they
can be adjusted, but it is more problematic for interactive users,
either directly or probably through sbuild when adding extra chroots.

Unfortunately at this stage I also don't have any better idea to
provide.

Cheers
Aurelien

Hi everyone,

In light of todays discussion about build tarballs, a simpler
solution for ca-certificates struck me: why not raise the Priority:
of ca-certificates to important or higher?

I see two good reasons to do this:

1) the (non-)presence of ca-certificates in a build environment
should not depend on the used mirror to create the chroot. It's
(non-)presence can cause other packages to FTBFS or otherwise differ
in their builds. Having it always be present would give us a more
consistent build environment.

2) I'd argue an OS install that cannot practically speak TLS to
anything is not very useful.

What do people think?

Should we approach the ca-certificates maintainers and ftpmasters?

Chris

Hi,

Quoting Chris Hofstaedtler (2025-02-22 00:52:59)

you can do that but it will not fix this problem. Debian Policy says:

Our buildd chroots used to install much more than that until recently when
this changed in debootstrap:

https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/106

The majority of the work was done by Santiago Vila who has been diligent with
finding and filing bugs for packages which FTBFS with only Essential and
build-essential installed.

Packages with Priority:required are no longer installed for Trixie
and later. There is a bug for mmdebstrap to do the same: #1091866

Note though that even after this change, 'apt' still gets installed into buildd
chroots even though it is neither Essential nor build-essential. This is
because sbuild cannot (yet) work with chroots that do not have apt installed in
them.

Not having it present also makes the build environment consistent. So
consistency is not an argument *for* installing it. In the long run, I propose
to go the other way: lets not have apt inside the chroot. Without apt inside
the chroot, there is also no ca-certificates required inside the chroot and the
problem is solved as well in a consistent way, no?

Thanks to reproduce.d.n we also have continuous CI testing package builds in
chroots without apt today. This highlights another problem of installing more
than build-essential inside buildd chroots: the .buildinfo files will not store
information about packages like apt, fakeroot or ca-certificates and thus,
rebuilds on reproduce.d.n will not contain them.

I agree. But this is not an OS install. I'd also argue that an OS install
should have an init but we do not install and run systemd inside buildd chroots
either. I think in contrast to an OS install, a buildd chroot should be as
minimal as possible so that we can find problems involving undeclared build
dependencies early.

I'd rather work towards a future that makes buildd chroots smaller, not large.
The current soft-blocker in sbuild for apt-less buildd chroots to happen is
schroot. In contrast to the unshare backend, the schroot backend is unable to
support running apt from outside the chroot. It's not a hard-blocker because
using a lot of special-casing, we can make the unshare backend special and let
it do different things. But that's not work I have done yet. Thanks to Jochen
the first step in this direction is done though and buildds run the unshare
backend. I expect that during Trixie, sbuild will also support apt-less
chroots.

Thanks!

cheers, josch