systemd-nspawn supports --template and/or --ephemeral (provided that you use btrfs)
so that you can have throw-away chroots. It also supports various
networking related options (--private-network notably) so that you can get
proper network isolation.

It would thus be nice to be able to use it with autopkgtest.

I'd like to report that another reason to support systemd-nspawn, which
really is an upgraded version of chroot, is that it Just Does The Right
Thing. For example, using autopkgtest-virt-chroot without further
preparation besides calling debootstrap causes my test to fail due to
not having /dev/ set up.

But systemd-nspawn takes care to mount the /dev filesystem.

This seems like a good idea to me, as well.

I haven't done much hacking on autopkgtest, but if you have pointers for
how to add a new isolation-container backend, i'd be happy to try to
work on it.

What's the best approach?  Should someone who wanted to add an nspawn
isolation container try to copy ./virt/autopkgtest-virt-lxc and
./tools/autopkgtest-build-lxc (and their associated manpages) and tweak
them to use nspawn instead of lxc?

Or is there a better way forward?

I couldn't think of anything better, so attached to this mail is a
start.

This is based on https://github.com/Truelite/nspawn-runner/blob/main/nspawn-runner
and on https://github.com/ARPA-SIMC/moncic-ci/blob/main/moncic/container.py#L463

This is what currently happens with it:

  autopkgtest-virt-nspawn: DBG: executing open
  autopkgtest-virt-nspawn: DBG: will start a container (with isolation-container capability)
  autopkgtest-virt-nspawn: DBG: using container name 8a087ee0-a8c1-4c87-a2f1-e43709e11184
  autopkgtest-virt-nspawn: DBG: execute-timeout: sudo systemd-run --quiet --property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Slice=machine.slice --property=Delegate=yes --property=TasksMax=16384 --property=WatchdogSec=3min systemd-nspawn --quiet --directory=/var/lib/machines/sid-build --machine=8a087ee0-a8c1-4c87-a2f1-e43709e11184 --boot --notify-ready=yes --resolv-conf=replace-host --volatile=overlay --read-only --suppress-sync=yes systemd.hostname=8a087ee0-a8c1-4c87-a2f1-e43709e11184
  autopkgtest-virt-nspawn: DBG: container started
  autopkgtest-virt-nspawn: DBG: execute-timeout: sudo systemd-run --quiet --machine=8a087ee0-a8c1-4c87-a2f1-e43709e11184 -- sh -c getent passwd | sort -t: -nk3 | awk -F: '{if ($3 >= 1000 && $3 <= 59999) { print $1; exit } }'
  autopkgtest-virt-nspawn: DBG: determine_normal_user: no uid in [1000,59999] available
  autopkgtest-virt-nspawn: DBG: auxverb = ['sudo', 'systemd-run', '--quiet', '--machine=8a087ee0-a8c1-4c87-a2f1-e43709e11184', '--', 'env', '-i', 'bash', '-c', 'set -a; [ -r /etc/environment ] && . /etc/environment 2>/dev/null || true; [ -r /etc/default/locale ] && . /etc/default/locale 2>/dev/null || true; [ -r /etc/profile ] && . /etc/profile 2>/dev/null || true; set +a;"$@"; RC=$?; [ $RC != 255 ] || RC=253; set -e;myout=$(readlink /proc/$$/fd/1);myerr=$(readlink /proc/$$/fd/2);myout="${myout/[/\\\\[}"; myout="${myout/]/\\\\]}";myerr="${myerr/[/\\\\[}"; myerr="${myerr/]/\\\\]}";PS=$(ls -l /proc/[0-9]*/fd/* 2>/dev/null | sed -nr \'\\#(\'"$myout"\'|\'"$myerr"\')# { s#^.*/proc/([0-9]+)/.*$#\\1#; p}\'|sort -u);KILL="";for pid in $PS; do    [ $pid -ne $$ ] && [ $pid -ne $PPID ] || continue;    KILL="$KILL $pid";done;[ -z "$KILL" ] || kill -9 $KILL >/dev/null 2>&1 || true;exit $RC', '--'], downtmp = None
  autopkgtest-virt-nspawn: DBG: execute-timeout: sudo systemd-run --quiet --machine=8a087ee0-a8c1-4c87-a2f1-e43709e11184 -- env -i bash -c set -a; [ -r /etc/environment ] && . /etc/environment 2>/dev/null || true; [ -r /etc/default/locale ] && . /etc/default/locale 2>/dev/null || true; [ -r /etc/profile ] && . /etc/profile 2>/dev/null || true; set +a;"$@"; RC=$?; [ $RC != 255 ] || RC=253; set -e;myout=$(readlink /proc/$$/fd/1);myerr=$(readlink /proc/$$/fd/2);myout="${myout/[/\\[}"; myout="${myout/]/\\]}";myerr="${myerr/[/\\[}"; myerr="${myerr/]/\\]}";PS=$(ls -l /proc/[0-9]*/fd/* 2>/dev/null | sed -nr '\#('"$myout"'|'"$myerr"')# { s#^.*/proc/([0-9]+)/.*$#\1#; p}'|sort -u);KILL="";for pid in $PS; do    [ $pid -ne $$ ] && [ $pid -ne $PPID ] || continue;    KILL="$KILL $pid";done;[ -z "$KILL" ] || kill -9 $KILL >/dev/null 2>&1 || true;exit $RC -- mktemp --directory --tmpdir autopkgtest.XXXXXX
  autopkgtest-virt-nspawn: DBG: execute-timeout: sudo systemd-run --quiet --machine=8a087ee0-a8c1-4c87-a2f1-e43709e11184 -- env -i bash -c set -a; [ -r /etc/environment ] && . /etc/environment 2>/dev/null || true; [ -r /etc/default/locale ] && . /etc/default/locale 2>/dev/null || true; [ -r /etc/profile ] && . /etc/profile 2>/dev/null || true; set +a;"$@"; RC=$?; [ $RC != 255 ] || RC=253; set -e;myout=$(readlink /proc/$$/fd/1);myerr=$(readlink /proc/$$/fd/2);myout="${myout/[/\\[}"; myout="${myout/]/\\]}";myerr="${myerr/[/\\[}"; myerr="${myerr/]/\\]}";PS=$(ls -l /proc/[0-9]*/fd/* 2>/dev/null | sed -nr '\#('"$myout"'|'"$myerr"')# { s#^.*/proc/([0-9]+)/.*$#\1#; p}'|sort -u);KILL="";for pid in $PS; do    [ $pid -ne $$ ] && [ $pid -ne $PPID ] || continue;    KILL="$KILL $pid";done;[ -z "$KILL" ] || kill -9 $KILL >/dev/null 2>&1 || true;exit $RC -- chmod 1777
  autopkgtest: DBG: got reply from testbed: ok
  autopkgtest: DBG: TestbedFailure sent `open', got `ok ' (0 result parameters), expected 1 result parameters
  autopkgtest: DBG: testbed stop
  autopkgtest: DBG: testbed close, scratch=None
  autopkgtest: DBG: sending command to testbed: quit
  autopkgtest-virt-nspawn: DBG: executing quit
  autopkgtest-virt-nspawn: DBG: cleanup...
  autopkgtest [13:56:31]: ERROR: testbed failure: sent `open', got `ok ' (0 result parameters), expected 1 result parameters
  autopkgtest: DBG: testbed stop

I know a thing or three about using nspawn containers, but I don't know
much about autopkgtest, and I'd love to team up with someone with the
opposite kind of knowledge to make this work.


Enrico