tl;dr:

  autopkgtest is slower than it needs to be for packages with many
  tests.  I can see at least one easy opportunity for a potentially
  substantial improvement, and have another more doubtful idea.

Background and factual information:

src:dgit has over a hundred autopkgtests (115 in all right now, but
some don't run in CI due to Restrictions; only 108 run in CI).  This
is convenient for isolating failures.  The tests are grouped into
(currently) 26 stanzas in debian/tests/control.

These tests usually take around 40 minutes to run in a formal run
under autopkgtest.  Sometimes the time taken exceeds the default
1-hour timeout in salsa CI.

Some of the tests are very fast - one or two seconds.  Some take much
longer.

(During a manual run outside autopkgtest, a complete run of the tests
takes about 8 minutes, but that's with CPU parallelism so isn't
comparable.  I haven't done a non-parallel benchmark run.)

Proposal 1:

Observing the logs, I see that even for tests in the same stanza,
autopkgtest prpeares and installs an autopkgtest-satdep package,
between every test.

For short tests, this dominates the execution time.  In the logs eg
https://salsa.debian.org/dgit-team/dgit/-/jobs/5270905
it seems to take about 6 seconds.

So I think skipping the testbed re-preparation for tests within the
same stanza would save about 6 seconds per such test, or 530-ish
seconds for each run.  Ie, it might cut 20% off the total runtime for
this package.

I'm hoping that this would be relatively straightforward to implement.

Proposal 2:

*Re*installing packages involved with many of the tests seems quite
expensive.  For src:dgit, most of the tests involve installing
dgit.deb.  dgit and its dependencies seem to take about 20s to
install.

If the test stanzas could be sequenced into "chains" where each test
is has a monotonically longer dependency list, the intervening satdep
installation, to get from one test stanza in the chain, to the nest
stanza, wouldn't need to involve *reinstalling* anything.

I'm not sure how much time this would save.  Also it's not clear to me
that this would always be 100% faithful.  After dependency resolution
is not monotonic: an extra dependency might result in *deinstallation*
of packages.

So this proposal is more fanciful but I thought I would share it
anyway.

Regards,c
Ian.

Hi,

[Mostly documenting a conversation we had in the team.]

While performance is obviously interesting, there's also the concern of
test isolation, see e.g.
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2058040. We
discussed this and agreed for the moment that we are more in favor of
ensuring testbeds are actually isolated from one-another, because it has
the property of least surprise (like the launchpad bug is requesting)
than for more optimization of sharing testbeds between different tests.
Now obviously we can work hard to have both, but reverting a backend
that supports that is much easier than figuring out all corner-cases
where tests can influence each other if we need to do it ourselves
between tests. So when moving forward we might go that route to keep
src:autopkgtest maintainable.

Paul

[...]

This is a good idea, but at the same time autopkgtest development is
going towards the idea of providing a pristine testbed to every test.
This means we don't want more testbed reuse (we want less of that), but
I believe there is a way.

Among autopkgtest and debci maintainers, there is an increased interest
in the -virt-incus and -virt-lxd virt servers. In particular, incus will
soon become the default virt server used by ci.debian.net, and because
of that I expect -virt-incus will become the go-to virt-server for most
local autopkgtest runs.

Incus and LXD support taking snapshots of instances. We could take a
snapshot of an instance after test-deps installation, and then re-use it
for tests in same stanza. I already have a branch that implements some
snapshot+revert logic:

https://salsa.debian.org/paride/autopkgtest/-/tree/snapshot-revert

but in that implementation the snapshot only happens once, before the
first test, after the testbed is updated/upgraded (so skipping the
initial update+upgrade is the optimization that branch provides). But it
shouldn't be to difficult to make it take a snapshot at the beginning of
a multi-test stanza, saving us from multiple test-deps installs.
As mentioned above, autopkgtest development is moving in the opposite
direction: each test should get a pristine testbed, so that tests can't
interact or interfere with each other, and test order does not matter.
This is already a thing in master:

https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/624

But I believe snapshots will give us most of the benefits of testbed
reuse without the drawbacks.

Cheers,

Paride