Hi,

I'm thinking of adding autopkgtests to src:nbd.

One thing which the nbd-client package adds is initramfs hooks for
supporting root-on-NBD. To do that, you need:

- A machine on the network running nbd-server
- A machine running the initramfs hooks, with kernel command-line
  parameters like "nbdroot=1.2.3.4:nbexport root=/dev/nbd0"

The second machine can then run diskless.

This is not possible with the current architecture of autopkgtest. I've
given this some thought, and came up with the following, which I think
strikes a balance between "little work needed to make basic usage of
this possible", and "flexible enough to allow for future extensibility".
The following demonstrates the idea I have in mind:
---8<---
Test-Skeleton:
  create testbed1
  create testbed2
  start-test testbed1 test1
  start-test testbed2 test2
  end-test testbed2
  set testbed1 VAR=$(ip ad sh | ...)
  start-test testbed2 test3 $VAR
  end-test testbed2
  end-test testbed1
  kill testbed1
  kill testbed2
Restrictions: multihost

Tests: test1
Restrictions: multihost-guest, breaks-testbed

Tests: test2

Tests: test3
Features: takes-variable
--->8---

This adds two restrictions and a variable, and creates a DSL that is
meant to be run by way of the "Test-Skeleton" header:

- The "multihost" restriction means to say that it isn't actually a test
  which is run directly, but that it's a "meta" test in the test suite.
  It requires the "Test-Skeleton" thing to mark that we're using the DSL
  here (but I guess we can do without that if that would make sense).
  The DSL would know the following things:
  - "create": creates a testbed. If multiple testbeds are created in
    this manner, they must be connected to eachother on a (virtual or
    physical) network. Probably we might want to extend this command
    later on to specify the layout of the network etc, but for now that
    probably shouldn't be required.
  - "start-test": starts a "guest" test in a given testbed, but does not
    wait for it to end (probably shouldn't be allowed to recurse).
  - "end-test": waits for the test that was started earlier in a given
    testbed to end. If that test returned failure, fail the whole test
    (and abort, probably). If it succeeded, we move on. If no test was
    running there, should probably error?
  - "set": runs an arbitrary command in a given testbed, captures the
    output (probably stdout only), and assigns it to a variable.
  - "kill": destroys a testbed. If a test is still running, the result
    of that test is ignored.
- The "multihost-guest" restriction means to say that this particular
  test makes no sense at all unless it's run as part of the multihost
  test defined elsewhere.
- The "takes-variable" feature, to say that it optionally takes a
  parameter.

There are no loops or conditionals here; that is on purpose. I don't
think this needs to be a Turing-complete language; if a test needs to
loop or test a condition, then that should be done in one of the tests
that is actually being run on one of the testbeds.

With that, in the above example, the "test1" test could configure the
system so that a server is running there, which can then be used by the
other tests that run in the other testbed. To discover on which address
the other testbed is running, the "host" test can run commands there
which assign variables, and then "guest" tests can use that output to do
"something useful" with them. This can be used for passing on network
configuration things, but it could also be used for other things; e.g.,
randomly-generated credentials could be passed from one testbed to the
other.

Thoughts?

Hi Wouter,

Sorry it took so long for you to get a reply on this. I didn't feel
competent enough to respond, but apparently nobody else felt like
replying either.

Great.

<Great explanation of what you think is your problem and how it could be
solved>

To be honest, this feels like lots of work for a tiny corner case.
Therefor, I don't expect anybody to take this on soon, except if you do
that yourself. And still, the code must be supported, autopkgtest is
already quite big.

One thing I note on your proposal, autopkgtest takes care by itself if
testbeds can be reused or not. I am not sure if you want testbed2 in
your example to be really the same testbed, or if testbed2 is just
semantics for "another testbed than testbed1".

Paul

No worries.

I'm not sure it's a tiny corner case.

The original thing that I thought of (which is not what I proposed in
the bug report) would have been, but I simplified it a lot. This
proposal would allow testing of any network-based system; e.g., it would
allow test suites to test that auto-detecting things on the network will
work (cups, routing software, avahi stuff, etc), or that a piece of
client/server software that has issues with the client running on the
same host as the server works correctly.

I can do that.

Sure.

I'm honestly not sure anymore, it's been a while since I wrote that
up...

Hi Ian

I guess this is best discussed in https://bugs.debian.org/908274 (added
in the To)? Maybe with Wouter and other interested parties?

Paul

Paul Gevers writes ("Re: Multi-host networking software, autopkgtests"):

Hmmm.  Well, a way of doing this "officially" in autopkgtest would be
nice.  But:

Think such an "official" thing ought to allow the specification of
different dependencies for the different hosts in the test.  And I
don't much like the mini-DSL suggestion.  Maybe it would be better to
offer the test script an adt virt server interface to control the
"other" hosts.

This all seems very complex.  I definitely want to have something
working before something like that could exist.  Also, I think it
would be a good idea to do something ad-hoc, ideally in a number of
packages, to gain experience so we know what shape the "official"
thing ought to be.

I think therefore that I need to pursue some kind of within-testbed
nesting, as an interim approach at the very least.  I was hoping that
someone else had solved (part of) this problem already...

Ian.

Hi,

Quoting Ian Jackson (2023-01-06 17:59:58)

there already exist a number of autopkgtest scripts that start a qemu virtual
machine and then run some script inside of it by connecting to it via ssh.
Maybe this can be helpful to you for now:

https://sources.debian.org/src/dropbear/2022.83-1/debian/tests/remote-unlocking/?hl=112#L137
https://sources.debian.org/src/sbuild/0.85.0/debian/tests/unshare-qemuwrapper/?hl=206#L206
https://sources.debian.org/src/cryptsetup/2%3A2.6.0-2/debian/tests/utils/cryptroot-common/#L505

Thanks!

cheers, josch

Hi Ian,

Unfortunately not.

We also had a discussion during the "autopkgtest office hours" session
during debconf21[1], where an alternate method (that would be slightly
easier to implement) was proposed: to have an autopkgtest helper command
that allows you to easily create and run a Debian VM for the host
architecture, and run stuff on it. This might be a bit easier to
implement than the dsl in the proposal.

[1] https://debconf-video-team.pages.debian.net/videoplayer/#/event/2021/debconf21?video=debconf21-82-autopkgtest-office-hours.webm&start=2028

Johannes Schauer Marin Rodrigues writes ("Re: Multi-host networking software, autopkgtests"):

Thanks.  I considered this but it seemed overkill (and it won't
inherit the dependency versions selected by autopkgtest).

For now I'm working on an adhoc thing that uses
  - Restrictions: needs-root
  - overlayfs to make an editable clone of the fs provided by autopkgtest
  - chroot
  - ip netns
  - apt-mark and apt-autoremove to get rid of unwanted deps

It's not particularly pretty but it's not much code and I feel I'm
making progress.  I'll report back here when I've succeeded - or
failed :-).

Thanks,
Ian.

Quoting Ian Jackson (2023-01-07 16:35:17)

maybe

Yes it will. When creating the chroot for the virtual machine, you use the apt
sources configured by the autopkgtest runner:

https://sources.debian.org/src/sbuild/0.85.0/debian/tests/unshare-qemuwrapper/#L62

or here:

https://sources.debian.org/src/dropbear/2022.83-1/debian/tests/remote-unlocking/?hl=80#L80

Ian Jackson writes ("Re: Multi-host networking software, autopkgtests"):
in the CI.  The CI environment is really quite different from the
real one (sometimes in what seem like strange ways).

Right now I am having the following impossible thing occurring. [1]
This code:

    tunfd= open("/dev/net/tun", O_RDWR);
    if (tunfd < 0) sysfatal("open /dev/net/tun");

    r= fcntl(tunfd, F_GETFD);
    if (r==-1) sysfatal("fcntl(tunfd,F_GETFD)");

is resulting in this output:

    2023-01-10T22:58:46.142075+00:00 ci-010-c4ebefae hippotatd[153]:
    error: ipif stderr: userv-ipif service: fatal system error:
    fcntl(tunfd,F_GETFD): Bad file descriptor

The executable which prints this message is not multithreaded and has
no signal handlers.  I suspect ? ? CI ? container ? hates tun ? ?,
or ? ? overlay fs on /dev ? strange effect on /dev/net/tun ? ?.

(Also in the test I have that *doesn't* do the pid namespace and
chroot thing, service(8) seems to have been disabled.  Maybe I need to
add a Restriction for that?)


I looked again at the links you helpfully provided.  That does look
like it would be doable, but it's decidedly nontrivial.

Each of those scripts has its own ad-hoc answer (or not) to questions
like "cope with the file:// apt urls not working" "map the Debian
architecture to a qemu binary" "configure the within-qemu environment
so we can do anything to it" "manage the lifetime of the qemu" etc.


I'm considering a third option: "disable the tests in debci" :-/.

Ian.

[1]
 Test script source code
https://browse.dgit.debian.org/hippotat.git/tree/adt/acommon?h=archive/debian/1.1.5%2bexp7
 Recent test log (full of distracting crap):
https://ci.debian.net/data/autopkgtest/unstable/amd64/h/hippotat/30238962/log.gz
 Source code for "ipif" program experiencing the EBADF error
https://browse.dgit.debian.org/userv-utils.git/tree/ipif/service.c?h=archive/debian/0.6.1-2#n711

Hi Helmut and others.

Some time ago we had a conversation on debian-devel about how to make
autopkgtests that spawn multiple nodes and communicate between them,
eg for testing network protocols[1].

To summarise that discussion: at that time the best available solution
that worked in ci.d.n seemed to be to write an ad-hoc script to run
the tests in qemu; three packes had done that, each separately, with
complex scripts with many moving parts.

I saw debvm, and wondered if it was suitable for this purpose.
But, then I looked at its debian/test/control and I see that the tests
are marked as flaky.[2]  So maybe it isn't reliable enough :-/.
I have other questions too, particularly to do with the way I would
need autopkgtest to be able to influence package selection in the
nested testbeds.

Everyone else: has there been any other progress on the multi-node
autopkgtest problem ?

Thanks,
Ian.

[1]
https://lists.debian.org/debian-devel/2023/01/msg00059.html
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=908274

[2]
https://salsa.debian.org/helmutg/debvm/-/blob/main/debian/tests/control

Hi Ian,

In principle, debvm is supposed to target that particular use case.
There are two limitations that currently make this infeasible.

The reliability of tests is ok. The reason for marking them flaky is
that they currently test the "wrong" packages. ci.d.n sets up chroots in
a delicate way to combine particular packages to see which combinations
cause breakage. Then debvm just creates an unstable system and tests
that. In effect, it currently tests unstable (inside those virtual
machines) rather than what it is supposed to be testing.

Johannes solved this problem on the mmdebstrap side and mmdebstrap's
tests no longer are flaky in this way. Therefore this should be solvable
on the debvm side. I just haven't gotten do figuring out the right runes
thus far. Roughly speaking, the hosts' apt configuration, pinning and
sources.lists should be used inside the created virtual machine.

Exactly. That's currently the missing piece to remove the flakiness
annotation.

There is another practical problem. None of the autopkgtest nodes
support kvm. Emulation will always use tcg. For one thing, tcg is slow.
It can be so slow on some architectures that RCU becomes unhappy as its
grace periods become too long. For another, tcg is buggy. It has
emulation bugs even on release architectures that make some expected
functionality fail. For instance, gdb reliably segfaults when run in
s390x tcg emulation.

That kvm aspect kinda seems like an unresolvable blocker. While most
autopkgtest machines are physical machines, they use kvm for running the
actual autopkgtest nodes and then lxc for individual test isolation.
We'd have to use nested kvm here and somehow get it through lxc.

Disregarding these two aspects, debvm should get you quite far. You
probably need to take network management into your own hands. I expect
that vde2 would be a good way to implement this in an unprivileged way.
Your debvm invocations may become a little longer that way, but that
should be fine given that you store all of that in scripts.

Helmut

Hi,

Quoting Helmut Grohne (2023-09-23 20:18:21)

the three packages are probably sbuild, dropbear and cryptsetup?

right now the mmdebstrap autopkgtest only mimics sources and pinning of the
outside system. I had not considered that apt configuration on salsaci or debci
was set to something that influenced the tests. Is the apt configuration on
those systems set to something that is not the default and should be considered
as well?

There is really not much magic. The core of it is to pass this to your
mmdebstrap or debvm-create invocation:


Architectures that are not amd64 or arm64 are a common source of problems. The
current sbuild autopkgtest hits another issue when running qemu on s390x:

514s ^M[   28.399829] illegal operation: 0001 ilc:1 [#1] SMP
514s ^M[   28.400490] Modules linked in: chacha_s390(+) libchacha virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio_blk
514s ^M[   28.402977] CPU: 4 PID: 163 Comm: cryptomgr_test Not tainted 6.5.0-1-s390x #1  Debian 6.5.3-1
514s ^M[   28.403150] Hardware name: QEMU 8561 QEMU (KVM/Linux)
514s ^M[   28.403285] Krnl PSW : 0704c00180000000 0000000000000042 (0x42)
514s ^M[   28.403964]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
[...]
514s ^M[   28.408739] Last Breaking-Event-Address:
514s ^M[   28.408758]  [<000003ff800fb084>] chacha_crypt_generic+0x54/0xfd0 [libchacha]
514s ^M[   28.409610] ---[ end trace 0000000000000000 ]---

https://ci.debian.net/data/autopkgtest/testing/s390x/s/sbuild/38123402/log.gz

In addition to debvm, these tests should maybe directly depend on qemu so that
they get run on new qemu uploads that trigger these kinds of regressions.
Assuming this is really a qemu problem and not something a new kernel version
introduced...

Thanks!

cheers, josch

Hi,

How the unstable to testing migration runs work is that they have a
testing testbed (with apt pinning making testing the
APT::Default-Release without using that configuration option for
$REASONS) with unstable added as an available suite. autopkgtest will
create apt pinning for only those packages that were requested on the
interface (by britney2, our migration software) to be added to testing.
That way, we try to see what happens in testing if we would migrate the
candidate source package to testing without all the rest of unstable.

Be aware, there's also an ugly fall back in autopkgtest where it will
remove all the pinning if it can't install all test dependencies with
the pinning set-upped as described above, effectively allowing all
packages from unstable to be installed. However, for the current
use-case that probably happens *before* debvm/mmdebstrap runs, so that
detail should not matter.

Paul

Hi Johannes,

This sounds simple, but reality is a little more elaborate.

For one thing, there also is
/usr/share/mmdebstrap/hooks/copy-host-apt-sources-and-preferences. This
hook directory is similar but subtly different from the above setup
hook:
 * It does not perform the translation of file:// uris into copy://uris.
 * It is more accurate in terms of following non-standard locations for
   the various configuration items.
 * Neither of these clear the 0000sources.list created by mmdebstrap by
   default.
 * The latter one verifies that you have the same package versions
   inside and outside.

Did I accurately represent the differences? Which one would you prefer
in which situation?

Adding any of this to debvm-create will not just work, because
debvm-create also adds the maybe-merged-usr hook and any pass-through
arguments you add come later. Therefore the maybe-merged-usr hook would
come before this hook and it fails if you pass an empty sources.list,
which would be most useful. As a workaround, you may add

Hi,

Quick followup given new insights.

The gist is that accessing file:// URIs from within the mmdebstrap
chroot won't work out of the box. One can either turn the into copy://
URIs or use the file-mirror-automount hook to issue bind mounts for
them. According to Johannes, the latter is to be considered more
reliable.

This is subtly wrong. The file-mirror-automount hook must come after
copy-host-apt-sources-and-preferences or it may miss URIs to mount and
it must come before maybe-merged-usr or it won't have done its job in
time. So rather use this pattern:

    debvm-create
        --skip=usrmerge
	...
	--
	--hook-dir=/usr/share/mmdebstrap/hooks/copy-host-apt-sources-and-preferences
	--hook-dir=/usr/share/mmdebstrap/hooks/file-mirror-automount
	--hook-dir=/usr/share/mmdebstrap/hooks/maybe-merged-usr
	""

The sbuild autopkgtest uses something roughly like this and debvm's
autopkgtest also now use this (and actually pass that way).

Beware of one horny detail. When mmdebstrap fails resolving dependencies
(and that can happen during debci), it kills its process group as a
mechanism to get rid of its children. This works fine if your
autopkgtest does not have needs-root. If it does, this failure mode can
currently damage debci infrastructure (yes, really). So if you use this
together with needs-root, please also wrap it in "setsid -w" to keep
debci in a healthy state.

And with these instructions, I think I've also resolved #1036919!

Helmut