Hi all,

The following issues have come up several times over the years. I
propose to discuss them in one place (this bug report) to define the
solution strategy. I haven't gone through all the details myself, so I
might be thinking in the wrong direction, please correct me if you think
so. Please also voice agreement, if not on the details, then on the
general concept.

Problem statements
==================

* runner/autopkgtest talks to the backend with a simple text protocol.
While this enables users to add another backend without changes to the
src:autopkgtest code trivially, the drawback of that is loosing all
nuance of what really is going on on both sides of the communication.
That is particularly bad when unexpected events happen. All events need
handling on both sides, including unexpected events.

* every backend has its own virt server that does the real communication
with the testbed. A result of that is subtle differences in test results
between different backends when they don't do exactly the same (code
easily goes out of sync).

* most backends don't automatically provide a testbed as a user would
see when working on a system. I recall smcv saying words like "user
session", "dbus something-something" and the like.

* [mostly orthogonal] currently the autopkgtest code has a lot of state
in a non-Pythonic way. Reasoning about what goes on and debugging
autopkgtest code flow is non-trivial.

Solution direction
==================

* unify the communication with test beds via ssh. This ensures that the
environment is much more likely to be the same across the different
backends and also ensures the right session.

* each virt server would only need to ensure an ssh server is setup and
running in the testbed and leaving the rest of the communication to a
common driver. (Maybe with the exception of the null, chroot and schroot
virt servers, to be investigated.) Obviously it's still responsible for
the tear down of the testbed.

* handle communication between runner/autopkgtest and the virt servers
and the ssh driver via Python classes instead of the text based
protocol. Do this in a "plugin" friendly way such that backends can
still easily be used without changes to src:autopkgtest.

Alternatives
============

* make the change to use ssh for communication, without a change of the
virt server protocol.

Open Questions
==============

* we could consider supporting the current protocol in parallel, which
would enable us to migrate one backend at a time and enable our users to
migrate their own backends at their own pace. However, it means we'd
need to support two code paths. So the open question is: (how long) do
we want to maintain the current protocol. I wonder how many other
backends are out there.

* although I don't know where it hooks in, but sbuild is using
autopkgtest's backends for some of its functionality. We don't want to
break sbuild, so the question is how the connection works.

* we already have an ssh virtual server, is that good enough to be the
ssh driver, or is it missing functionality and/or deserves a rewrite by
itself? To answer the last question, probably yes if we want to move
away from the current protocol.

Tasks
=====

[ ] discuss this idea and get consensus on the way forward
[ ] create working branch and generate a PoC with one of the backends
[ ] figure out how sbuild hooks into our backends
[ ] while changing code, add Python typing where applicable
[ ] ...

Paul
PS: would it be worth it to enable dashboards for autopkgtest on salsa
to manage this project? I assume issues on salsa are disabled on purpose
to avoid bug reports in multiple places. Could we make adding issues
project members only?

Hi Paul,

I'm not a maintainer but I use autopkgtest a lot. I hope it's OK if I
contribute input.

I generally agree with all of what you said, and would add the following:

It is indeed very difficult to keep track of what's going on. A lot of
state is kept in/communicated through globals, and it can be challenging
to remember which role the running threads are playing, and in which
relationship.

(smcv put this into historical context once.)
separate communications channels.

Example: the Debian ROCm Team requested the --timeout-poweroff option
for the QEMU backend because the hardware we pass in needs a clean
shutdown procedure. But it is not possible to trigger a shutdown when a
test is running, because on the I/O channel is being waited on for
output. So a timeout still ends with a SIGTERM of the testbed.

Best,
Christian

Hi,

Yes, absolutely. Comments from all people that want to contribute
constructively are welcome. Even more so from users and contributors.

Paul

Hi Paul,
easy to repurpose the virt servers for other uses, like what sbuild does.
These other projects do not need to be written in Python, or we could in
principle have a virt-server not written in Python.

+1 to these.

I agree nowadays ssh is a reasonable common denominator. As you noted
below, there are some virt servers where it doesn't apply well, but
they are probably special enough to be treated differently.
And there is also autopkgtest-virt-unshare (probably falling under the
chroot category).

I think standardizing on ssh is desirable, but this implicitly means
that we'll have some more specific requirements for the testbed images
(in random order: sshd, some sort pubkey authentication, a "normal"
(non-root) user, cloud-init to initialize all these things?, ...).
We are currently shipping tools to prepare test images
(tools/autopkgtest-build*), but at the same time we are very flexible
on the image requirements. Should we accept being more strict on this,
and state that the virt server are meant to be used to purpose built
images? Or should we have a better spec on what the virt servers
expect from the image?

Currently autopkgtest-virt-ssh works around this by using the
"ssh setup" script, but my impression is that we want to move
away from that kind of approach.

Do you think this can be done incrementally, that is:

1. modify the virt-servers we have to use ssh for communication
with the testbed systems, keeping it in a common library.

2. keeping the implementation above, or most of it, also
reimplement the autopkgtest<->backend communication protocol.

The two problems should be quite decoupled after all, and while
I'm convinced that point 2 is good, I am less sure about point 1,
for the reasons I stated initially.

Are we aware of any other consumer of the virt servers apart from
autopkgtest itself and sbuild?

I think we'll probably want a pure Python implementation of that,
written using paramiko.

I'm in favor of experimenting with that, of course keeping actual
bugs in the bts.

Thanks for bringing this up!

Hi,

I have enabled the issue tracker on salsa for members only (to avoid
getting bug reports there that should go to the BTS). I started a plan
board [1]. I'm not used to use interfaces like this, so please help me
get it into an useful shape if you have ideas on how to improve it and
even speak up if you think I'm doing it "wrong". Feel totally free to
add more issues if you spot something's missing. Ideally I very much
don't do this alone.

Paul

[1] https://salsa.debian.org/ci-team/autopkgtest/-/boards

Hi,

That's exactly what I meant.

Currently all our backends are written in Python, so the requirement to
have it already exists. I could imaging (just an idea) that we have a
(hopefully small) glue script that connects the new or rewritten
backends to the old protocol for users like that.

Obviously if there are too many of those, the value of the redesign drops.

What I was thinking (but absolutely open for debate) is that the new
virt-servers would be responsible for setting up the sshd-server in
their supported testbed and handling the key generation and passing on
of the information. But, as you suggested, we could also leave that to
the image generators (being them our "own" or otherwise created) and
document what we need in the testbed. E.g. we could spec that the ssh
key lives somewhere in the testbed, to be picked up by the virt-server
which transfers the info to the ssh driver. If we go that route, we
would remove quite some logic from the `autopkgtest` code path, into the
preparation step, which I think would be good. However, a large price
(is this worrying? at first sight it doesn't look like anything bigger
than already is exposed) would be that the key is probably going to be
shared between all autopkgtest runs that are started from the same
template testbed.

So that was my initial idea, but indeed, might be better to transfer as
much of that as possible to to the testbed creation phase. I'm wondering
how much is possible with those maas and nova setups? I would assume
they already do only what's needed.

That's what I was thinking indeed. But it's hard to judge where it's
better to try and keep steps small and manageable to get results versus
the addition work required if we finally reach the end goal.

I think I miss the point you are trying to make here.

I was not, but last days I learned that reprotest copied our virt
servers:
https://salsa.debian.org/reproducible-builds/reprotest/-/tree/master/reprotest/virt
(documentation ongoing in issue #3)

I have never heard of paramiko.

Paul