Package: autopkgtest
Version: 5.22
Severity: important

Dear Maintainer,

While running two tests with autopkgtest-build-qemu I was surprised to
see the database created from the first one not being wiped before
running the other test.

    $ cat debian/tests/control
    Test-Command: date -R | tee /dev/shm/stamp
    Depends: coreutils

    Test-Command: date -R && cat /dev/shm/stamp
    Depends: coreutils

    $ autopkgtest /tmp/coreutils_8.32-4.1_amd64.deb ./ -- schroot sid-amd64-sbuild
    […]
    autopkgtest [16:47:12]: test command1: date -R | tee /dev/shm/stamp
    autopkgtest [16:47:12]: test command1: [-----------------------
    Fri, 01 Jul 2022 14:47:12 +0000
    autopkgtest [16:47:12]: test command1: -----------------------]
    autopkgtest [16:47:12]: test command1:  - - - - - - - - - - results - - - - - - - - - -
    command1             PASS
    autopkgtest [16:47:12]: test command2: preparing testbed
    […]
    autopkgtest [16:47:16]: test command2: date -R && cat /dev/shm/stamp
    autopkgtest [16:47:16]: test command2: [-----------------------
    Fri, 01 Jul 2022 14:47:16 +0000
    Fri, 01 Jul 2022 14:47:12 +0000
    autopkgtest [16:47:16]: test command2: -----------------------]
    autopkgtest [16:47:16]: test command2:  - - - - - - - - - - results - - - - - - - - - -
    command2             PASS
    autopkgtest [16:47:16]: @@@@@@@@@@@@@@@@@@@@ summary
    command1             PASS
    command2             PASS

With autopkgtest-build-qemu, the VM isn't rebooted and files created by
some autopkgtest are also accessible by subsequent tests in the same
autopkgtest(1) invocation.

    $ cat debian/tests/control
    Test-Command: uptime -s && date -R | tee /dev/shm/stamp /stamp
    Depends: coreutils, procps
    Restrictions: needs-root

    Test-Command: uptime -s && date -R && cat /dev/shm/stamp /stamp
    Depends: coreutils, procps
    Restrictions: needs-root

    $ autopkgtest /tmp/coreutils_8.32-4.1_amd64.deb ./ -- qemu --overlay-dir=/dev/shm /tmp/debian-unstable.img
    […]
    autopkgtest [16:49:47]: test command1: uptime -s && date -R | tee /dev/shm/stamp /stamp
    autopkgtest [16:49:47]: test command1: [-----------------------
    2022-07-01 14:49:18
    Fri, 01 Jul 2022 14:49:47 +0000
    autopkgtest [16:49:48]: test command1: -----------------------]
    autopkgtest [16:49:48]: test command1:  - - - - - - - - - - results - - - - - - - - - -
    command1             PASS
    autopkgtest [16:49:49]: test command2: preparing testbed
    […]
    autopkgtest [16:49:59]: test command2: uptime -s && date -R && cat /dev/shm/stamp /stamp
    autopkgtest [16:49:59]: test command2: [-----------------------
    2022-07-01 14:49:18
    Fri, 01 Jul 2022 14:49:58 +0000
    Fri, 01 Jul 2022 14:49:47 +0000
    Fri, 01 Jul 2022 14:49:47 +0000
    autopkgtest [16:49:59]: test command2: -----------------------]
    autopkgtest [16:50:00]: test command2:  - - - - - - - - - - results - - - - - - - - - -
    command2             PASS
    autopkgtest [16:50:00]: @@@@@@@@@@@@@@@@@@@@ summary
    command1             PASS
    command2             PASS

Granted the documentation says nothing about state reset/preservation,
but shouldn't each test run a clean environment?  Otherwise the tests
might succeed when run separately but fail in a single autopkgtest
invocation.  (This was indeed the case for me: a test broke due to a
leftover database created during an earlier test, but each test passed
just fine in isolation.)

Thanks for maintaining autopkgtest!

Oh sorry for the noise, it appears tests under ‘Restrictions:
breaks-testbed’ don't preserve the state.

While in retrospect the documentation is pretty clear about it, I
somehow thought it was an alias for a suitable ‘Restrictions:
isolation-*’ hence was confused why the state wasn't reset in deeper
isolation levels.  Should have tried it out before filing #-1 :-)