The idea of dropping the schroot virt server has been discussed
informally; this bug is to make the discussion more public and to serve
as future reference. Somehow related is [1].

The schroot virt server has a number of issues:

* There is no tools/autopkgtest-build-schroot script that prepares
  autopkgtest schroots by also running setup-testbed, which is the
  script that customizes and in part "normalizes" images for execution
  of autopkgtests. This means that schroot runs may differ from
  autopkgtst runs done using other virt server.

* It has no isolation capabilities, which are offered by the container
  based virt servers, which are equally fast.

* I am not aware of Debian (or Ubuntu) developers actively using it.
  This includes autopkgtest developers, and makes the virt server not
  well tested.

* There are better options which AFAIK have no significant drawbacks.
  Dropping the schroot virt server would lower the maintenance burden.

* Using a schroot does not play well the idea of standardizing the
  communication channel to the testbed to always be SSH.

If we agree we want this, I can do the code removal and update the
documentation.

Cheers,

Hello,

Bug #1074556 in autopkgtest reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:

https://salsa.debian.org/ci-team/autopkgtest/-/commit/8d48686f437ef4cde13f78274dbc712814516f2f
------------------------------------------------------------------------
Drop the schroot virt server (autopkgtest-virt-schroot)

Other virt servers offer the same functionality but rely on more widely
used, actively developed and better supported tools.

Closes: #986665, #1015145, #1074556
Gbp-Dch: full
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/1074556

Hi,

Thanks a lot for all autopkgtest people for the work!

Just as a data-point there are actually users, as for the autopkgtests
run within the pkg-perl group this was enough, so there is
https://salsa.debian.org/perl-team/modules/packages/pkg-perl-tools/-/blob/master/examples/check-build?ref_type=heads#L162
used within the group.

https://perl-team.pages.debian.net/autopkgtest.html as well relies on
it yet (but needs then to be adapted for the team).

But that said, I guess the people currently using it can switch to one
of the other virt server backends for autopkgtests as well. I'm
explicitly CC'ing Gregor as well here.

So this by no means does invalidate the other points, in particular
lowering the maintenance burden for your team.

Regards,
Salvatore

Hi Salvatore, Gregor,

Debian buildds are migrating away from schroot (AFAIK only
stable/security still run with schroot) and schroot is considered
deprecated in Debian. In my opinion the perl team should move on too.
I'll also note that that last link (autopkgtest.html) might need an
overhaul as there's quite some information outdated it seems.

Paul

It looks as though that script can be configured to use any autopkgtest
virtualization backend by setting AUTOPKGTEST_VIRT_SERVER and
AUTOPKGTEST_VIRT_SERVER_ARGS.

I would personally suggest podman, which doesn't require root (only a
record in /etc/subuid and /etc/subgid) and has the advantage of being a
tool that is widely used outside the Debian/Ubuntu bubble, using the same
OCI image format as Docker for its container images. There is an EXAMPLES
section in autopkgtest-build-podman(1) and autopkgtest-virt-podman(1)
illustrating how to use it.

Container images for autopkgtest are also suitable for interactive
debugging using podman-run(1). If the contents of the container are
trusted (no security boundary intended between your uid in the container
and your uid on the host), then it can be convenient to use toolbox(1)
from the podman-toolbox package as a replacement for interactive schroot,
with the same convenient sharing of the home directory as schroot.

Or, for sbuild users, the unshare backend autopkgtest-virt-unshare(1)
is a relatively natural choice, since it uses the same mechanisms as
the sbuild unshare backend that is now used on our official buildds, and
also doesn't require root (only a record in /etc/subuid and /etc/subgid,
the same as podman and mmdebstrap). It's Debian-specific, though.

The unshare backend automatically reuses the same tarballs that are used
by sbuild in unshare mode, which is very convenient (although ideally
tests would be run in a separate tarball based on "minbase", to allow
detection of missing dependencies on build-essential packages, which
cannot be detected if the base tarball has those packages preinstalled).

    smcv

Hi Paul, hi Simon,

Just to be clear, the message from me was by *no* means meant to raise
a flag to ask to please reconsider. Just saying there were actual
still used setups, which will migrate to one of the suggested better
backends (thanks for the summary).

Again, thanks for all the work you put in to make Debian better :)

Regards,
Salvatore

I'm afraid I don't agree with this change.  I realise I'm late to
this party.

The root of my position is this:

The purpose of the autopkgtest-virt-* protocol is to allow
applications (not just autopkgtest) to use the widest possible set of
virtualisation or containment approaches, with the minimul amount of
special-purpose code.

To serve this purpose, an autopkgtest virt server should be provided
for every virtualisation method that exists.

It follows that the autopkgtest virt server for schroot should not be
removed unless and until schroot itself is removed.

I should deal with the points raised by Paride:

I don't think these are reasons for removal.

I use schroot very regularly, with autopkgtest-virt-schroot.  It's my
usual use of autopkgtest.  It works well for me.

This idea is not going to fly anyway.  -null and -unshare don't work
that way, and future virt approaches also might not be able to do
this.

Paul Gevers writes:

Can you provide a reference to that?

I find schroot extremely useful in my local development.  I do not
intend to stop using it any time soon.  If there were suggestions that
it might be removed, I would probably adopt it.

Ian.

Simon McVittie proposes some alternatives:

podman is perhaps often a good choice.  However, sometimes it can be
useful or even essential to use a setup with *less* isolation.
That is often the case for me.  I have multiple situations where I
want to use software in a chroot and *allow* it access to my normal
home directory, full process tree, and so on.

The autopkgtest-virt-unshare backend seems quite primitive compared to
schroot.  schroot offers user-switching and an intermediate level of
privsep.  This is a feature that can be useful.  It is also highly
configurable.

Also, it hasn't had the level of testing and maturity that the schroot
backend has.  (When I first tried it out it I discovered a howler
bug.)

I don't think these are realistic replacements for schroot.

Ian.

We had a discussion on irc.  The upshot is that we are going to move
the autopkgtest-virt-schroot backend into a new source package,
provisionally called autopkgtest-virt-extra.  I will be preparing that
package.

The principal motivation is social: Simon McVittie, one of the
principal current contributors to src:autopkgtest maintainers, feels
it is a maintenance burden.

At a techical level, there is a difficulty with the VirtSubproc.py
module.  The -virt-* implementations all use this module, which has
important shared functionality.  It does not have an advertised stable
API and is shippwed under /usr/share/autopkgtest/.

Simon has asked me to *copy* this file rather than referencing it.

I will think about ways to automatically detect or handle it getting
out of date.

I hope to prepare the new package in the next week or two.  We will
need to ask for expedited NEW review, because (at least IMO) we don't
want the new autopkgtest to migrate to testing until the removed parts
are available in the new package.

Ian.

Hi Ian,

I went through the whole IRC conversation. I am happy that a good
compromise was eventually found.

What we want as autopkgtest maintainers is to be free to modify
VirtSubproc.py without coordination.

I'm quite curious to see how to fix for #1052119 works in practice, but
I don't think we are especially in a hurry for an upload right now.

Cheers,

Paride

Paride Legovini writes ("Re: Bug#1074556: autopkgtest: Drop the schroot virt server"):

FTR, I do not consider this compromise "good".

Ian.