Hi,

Paul asked me to take this to email and he's probably right. Johannes
Schauer repeatedly tried to test mmdebstrap --mode=unshare in
autopkgtests and reportedly this fails. The most recent failure
reportedly happened three days ago in experimental, but retrieving
experimental logs turns out to be difficult.

What is needed to make this work? mmdebstrap --mode=unshare requires the
following features:
 * unprivileged unsharing of user namespaces
   - This is prohibited on DSA machines via a sysctl
   - It works on most other systems
   - Test case: unshare -U true
 * A subuid allocation in /etc/subuid
   - Allocated by default during user creation
   - Test case: grep -q ^$(id -un): /etc/subuid
 * subuid allocation must be mapped by container technology (if any)
   - I suppose the unshare backend fails this. Likely also unprivileged
     podmand.
 * It must be possible to mount proc in the unshared user+mount+pid
   namespace.
   - This should always work but may be restricted by the container
     technology for some reason.
   - Test case: unshare -U -m -p -f -r --mount-proc true
   - Paul tried this in the operational lxc containers. Successfully.
   - I tried this in a local autopkgtest-unstable lxc container.
     Successfully (unprivileged).
   - Johannes reported that this would be the step that fails.
 * Maybe more, but I don't know what would be missing. Maybe Johannes
   knows.

In this bug, I ask for a new restriction and propose calling it
"needs-userns". It should cover the aspects mentioned above (probably in
more precision).

How does this apply to backends?

 * chroot/schroot: Unsure. Maybe it works, maybe not.
 * docker/podman: If unprivileged, it very likely does not work as the
   user as the user running it would need at least two subuid ranges,
   which is not the default and the container technology would have to
   map the second subuid range to the subuid range of the test user. If
   privileged, it might work (like lxc).
 * lxc/lxd: Our testing indicates that this works.
 * qemu: Any kind of VM-based backend probably just works.
 * unshare: This very much is like unprivileged docker/podman and very
   likely does not work.

What packages would use this beyond mmdebstrap? I see at least
chromium's sandbox using user namespaces for isolation (though with less
required features). Adoption of user namespaces will probably grow.

So initially, it could be as simple as adding this restriction and
statically saying that it is supported by lxc/lxd/qemu and nothing else.

The other part probably is figuring out how to make it work, which may
be either mmdebstrap bugs or missing features on debci, but diagnosing
this is evidently hard.

A number of tests that would benefit from this capability can be run
with needs-root or fakechroot (when it works) instead.

Helmut

Hi,

Quoting Helmut Grohne (2023-02-05 21:23:18)

there is also the autopkgtest of sbuild which I currently run inside qemu to
have unshare support during autopkgtest:

https://salsa.debian.org/debian/sbuild/-/blob/main/debian/tests/unshare-qemuwrapper

The complexity of this script can be much reduced by using debvm.

Apart from that, the autopkgtest unshare backend could also be tested in an
autopkgtest that supports it.

I'm unsure whether a needs-userns restriction would give me much. Even if that
restriction existed, usually I don't want to say "run this test if unshare
works" but I want to say "run this if unshare works and this if it doesn't".
For example in case of mmdebstrap I'd like to say:

 - use unshare if it works
 - use fakechroot otherwise

In case of sbuild I'd like to say:

 - use unshare if it works
 - run the test inside qemu if it doesn't

Since currently I know of no good heuristic to check whether unshare works, I'm
running the sbuild autopkgtest always inside qemu and I'm running the
mmdebstrap autopkgtest either as root or explicitly using fakechroot.

Benjamin Drung attempted to work around this problem in sbuild by checking
whether unshare works or not using

    syscall(&SYS_unshare, $CLONE_NEWNS | $CLONE_NEWUSER);

But that is not enough. On salsaci, even if this succeeds, mounting /proc later
will fail. The mounting failure can maybe be worked around by using different
mount options or bind-mounting if mounting a real proc fails but it showcases
that "unshare is working" is not easy to check without actually performing the
actions one is interested in. The second thing that Benjamin's merge request
shows is, that there is a lot of overhead in determining what test to run:

https://salsa.debian.org/debian/sbuild/-/merge_requests/27

Thanks!

cheers, josch

Quoting Helmut Grohne (2023-02-05 21:23:18)

as a datapoint, on salsaci this happens:

+ runuser -u debci -- unshare -U -m -p -f -r --mount-proc true
unshare: mount /proc failed: Operation not permitted

Next I'm trying just to bind-mount /proc... lets see...

Quoting Johannes Schauer Marin Rodrigues (2023-02-09 11:03:05)

I now followed a more structured approach and tried the following six variants
under `runuser -u user -- unshare -U -m -p -f -r` in QEMU, salsaci as well as
debci:

 1) `mount -t proc proc $tmp` -- the correct way to mount /proc, aka the way
    that debootstrap mounts /proc

 2) --mount-proc as an argument to the unshare command

 3) `mount -t proc -o nosuid,nodev,noexec proc $tmp` which results in the same
    mount syscall as the --mount-proc argument to the unshare command

 4) `mount -o bind /proc $tmp` just bind mount /proc into the chroot

 5) `mount -o bind,ro /proc $tmp` maybe bind-mounting read-only is doing
    something different?

 6) `mount -o rbind /proc $tmp` lets just throw this in for good measure and
    see what happens. According to the mount man page, there is no read-only
    variant for recursive bind mounting.

The following table displays whether the command worked or failed with outputs
like:

    wrong fs type, bad option, bad superblock on /proc, missing codepage or helper program, or other error

or:

    mount point not mounted or bad option

Even when performing the mount syscall manually I found no way to get better
error output about what is actually not working...

     QEMU    salsaci debci
 1   yes     no      yes
 2   yes     no      yes
 3   yes     no      yes
 4   no      no      no
 5   no      no      no
 6   yes     yes     yes

Do you see what I see? Apparently a recursive bind-mount of /proc works on
salsaci! I already tried this out in mmdebstrap and by using rbind I can
successfully run mmdebstrap in unshare mode on salsaci. So the secret of
bind-mounting proc in a privileged docker container is to use --rbind.

Thanks!

cheers, josch

I assume this is because if you have "covered up" a sensitive or dangerous
part of /proc to stop processes inside the container from poking it
(for example mounting an empty file or inaccessible device node over
/proc/sysrq-trigger), doing a non-recursive mount would "uncover" it,
which is undesirable if you want your container to be anything vaguely
resembling a security boundary.

    smcv

To make this genuinely useful for typical container use-cases, you
usually want a block of 65536 uids; otherwise you'll tend to get weird
failures. User creation allocates a suitable block by default.

Container technologies that don't unshare the user namespace (uid 0 on
the host = uid 0 in the container), notably Docker and older privileged
setups for lxc, have no userns-based protection from misuse of things
like /proc/sysrq-trigger; so they have to either prevent those attacks
some other way, which tends to involve disallowing mounting /proc, or
accept that root in the container gives you root in real life (leading
to the mantra "containers don't contain").

A somewhat weaker form of user namespace support could be summarized
as "enough to run bubblewrap", which requires unprivileged unsharing
of user namespaces and the ability to mount a new instance of /proc,
but does not require a subuid allocation or any setuid helpers like
newuidmap/newgidmap.

This would be enough for single-user single-app sandboxes like bubblewrap,
Flatpak, epiphany-browser (GNOME Web), and probably Chromium's sandboxing
(which doesn't use bubblewrap, but makes similar syscalls itself),
but not enough for containers that want multiple uids, such as podman
and mmdebstrap/unshare.

    smcv