Dear maintainer(s),

I looked at the results of the autopkgtest of your package. I noticed
that it recently started to fail a lot on ppc64el. The failures seem
related to the host that runs the test. Several weeks ago, we
commissioned a new host (ci-worker-ppc64el-05) with more cores, more RAM
and faster disk and most recent tests ran on that host. Apart from the
hardware, we also use /tmp on tmpfs (with lots of swap). Overall, I
expect the host to be *faster* than the old hosts, but ironically the
tests that seems to fail is: __main__.SchrootRunner.test_copy_timeout.
As also amd64 has /tmp on tmpfs, I don't immediately suspect that to be
the problem; also at least one run on ci-worker-ppc64el-05 passed
(44524634).

Because the unstable-to-testing migration software now blocks on
regressions in testing, flaky tests, i.e. tests that flip between
passing and failing without changes to the list of installed packages,
are causing people unrelated to your package to spend time on these
tests.

Don't hesitate to reach out if you need help and some more information
from our infrastructure.

Paul

Hi,

Yes, it's too fast. The test is testing a huge file and expects the copy
to fail because it should take longer. Well, with tmpfs the autopktest
inside the test passes and hence the outer test fails.

https://ci.debian.net/packages/a/autopkgtest/testing/ppc64el/44781763/
640s 16:22:15 O: handling copying timeout ... FAIL
640s 16:22:15 E: # command stdout:
640s 16:22:15 E: #     to                   PASS
...
640s 16:22:15 E: # exit status: 0

I forgot that on amd64, autopkgtest's autopkgtest now runs in qemu which
doesn't benefit yet as much from tmpfs as lxc does, so it's not a good
comparison.

Is there a way to detect if we're on tmpfs? We should skip this test then.

Paul

The mystery continues.
"""
root@elbrus:/tmp/autopkgtest-lxc.ww6f2cls/downtmp/build.VWV/src# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           250G   44G  207G  18% /
none            492K  4.0K  488K   1% /dev
tmpfs           126G     0  126G   0% /dev/shm
tmpfs            51G   52K   51G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
root@elbrus:/tmp/autopkgtest-lxc.ww6f2cls/downtmp/build.VWV/src#
AUTOPKGTEST_TEST_SCHROOT=testing-amd64-sbuild tests/autopkgtest
SchrootRunner.test_copy_timeout
test_copy_timeout (__main__.SchrootRunner.test_copy_timeout)
handling copying timeout ... ok
----------------------------------------------------------------------
Ran 1 test in 5.768s

OK
"""

So it's not only tmpfs. I ran the test on ci-worker-ppc64el-05 (no other
tests were running at the time) with adapted timeout. If I reduce the
current 3 sec copy-timeout to 1 sec the test passes (2 seconds is too
long, only int allowed). If I try the same thing in the opposite
direction on ci-worker13, raising to 4 seconds causes failure in the
"""self.assertLess(huge_size, 10000000000)""", with bumping to 5 seconds
fails in the same way. ci-worker13 was running quite some tests at the
same time, not sure if that's related.

Anyways, it looks like we could just lower the timeout to 1 second and
hope were fine for some time to come.

Paul

Hi,

No. 1 second is *too short* for the PodmanRunner.test_copy_timeout test
on salsa. So I'll just disable the test for now.

Paul

Control: severity -1 normal
Control: retitle -1 src:autopkgtest: test_copy_timeout skipped because it's dependent on speed of testbed
Control: tags -1 - pending

This was done in 5.35, making this bug non-RC. I'm retitling it and
leaving it open to represent the technical debt of missing test-suite
coverage, but I'm not really seeing any good way to test this particular
corner case: it's difficult to trigger a timeout during copying on-demand
without making the test vulnerable to timing out for an unrelated reason.

If other maintainers don't think this is worth having a bug open for,
please close it.

    smcv