Dear maintainer(s),

I looked at the results of the autopkgtest of you package on s390x
because it was showing up on our alert page [1]. I noticed that there
were several runs that took 27h before timing out, while successful runs
only take about 15 to 20 minutes. This is not limited to s390x, also
other architectures fail regularly. In stable, the test also fails once
in a while, but I didn't spot the timeouts there.

Because the unstable-to-testing migration software now blocks on
regressions in testing, flaky tests, i.e. tests that flip between
passing and failing without changes to the list of installed packages,
are causing people unrelated to your package to spend time on these
tests.

On top of that, when a test just hangs that's not good for our
infrastructure. I'll put gscan2pdf on our reject_list for amd64, armhf,
i386, and s390x.

Don't hesitate to reach out if you need help and some more information
from our infrastructure.

Paul

[1] https://ci.debian.net/status/alerts/

Examples of 27h time out
https://ci.debian.net/data/autopkgtest/unstable/amd64/g/gscan2pdf/21000327/log.gz
https://ci.debian.net/data/autopkgtest/testing/armhf/g/gscan2pdf/22210007/log.gz
https://ci.debian.net/data/autopkgtest/unstable/i386/g/gscan2pdf/21000345/log.gz
https://ci.debian.net/data/autopkgtest/testing/s390x/g/gscan2pdf/22222205/log.gz

This is proving hard to nail down.

The examples you give all time-out on different tests.

I've never had a timeout problem whilst testing interactively or in a
local schroot.

I see the status on the runs that time out is always "tmpfail". What
does tmpfail mean? How is the status determined?

The timeouts seem rare and even then something that has only recently
started. Changes to gscan2pdf in the last year have only been cosmetic,
and I wonder therefore if a dependency is causing this.

I see error messages such as:

AT-SPI: Error retrieving accessibility bus address:
org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not
provided by any .service files

Which according to

https://gist.github.com/jeffcogswell/62395900725acef1c0a5a608f7eb7a05

Might be fixed with

   sudo apt install at-spi2-core

On the other hand, this seems very hacky, as these really shouldn't have
anything to do with gscan2pdf - maybe a dependency, e.g. gtk+.

But these also occur in runs which pass, such as:

https://ci.debian.net/data/autopkgtest/stable/amd64/g/gscan2pdf/23364484/log.gz

I'd be grateful for any suggestions

Regards

Jeff

Hi Jeff,
"""
Don't hesitate to reach out if you need help [...]
"""

So, either using debian-ci@lists.debian.org or the submitter's address
(mine) seems appropriate.

In this case:
we can trigger the test from the backside, such that you can get a fresh
log, but I prefer to only do that coordinated and after you give it a
try to enable more diagnostic logging, because apparently in the
original logs there wasn't enough information for you.

I also offer to run the test (once or twice) manually and get
information out of the testbed, if you tell me the exact commands you
want me to run in the testbed.

Paul
PS: I propose we drop debian-devel from the replies and continue our
discussion in the bug, but please keep me in CC as I'm not subscribed to
the bug. Be reminded that the BTS doesn't send e-mail to the submitter
unless asked explicitly or unless the bug is closed.

Hi Paul,

In which case, I evidently misunderstood. I had assumed that replying to
the bug mail would do the same.

Please trigger a new run. I can't add more diagnostics until I know where.

What bothers me about this is that these flaky tests do not occur with
the buildd hosts:

https://buildd.debian.org/status/logs.php?pkg=gscan2pdf&arch=all

The only failure took 7 minutes, not 27 hours.

So how do the CI hosts diff from the buildd ones?

Thanks for your help.

Regards

Jeff

Hi Jeff,

Done so a couple of times, we'll first need to hit one that times out of
course. I'll continue to trigger until we find one for a while.

I don't know how buildd's look like, but an obvious delta is that
autopkgtests on our host run inside lxc. Also, as your package is
arch:all only, it always runs on amd64. I think one of the delta's we
have for *several* hosts is that the amount of cores and RAM available
for the test is significantly higher than on buildds. On a lot of our
hosts, we also run multiple debci workers in parallel, so timing wise,
you may be seeing varying performance even on the same host.

Paul

Hi Jeff,

I have triggered several runs (about 15 or so) and they all passed. I
have removed the block and am lowering the severity of this bug for now.

Well, the failure didn't happen that often, so maybe it just didn't
happen on the buildd. Also, maybe it's something relatively new (and you
only had X chances on the buildds).

We'll keep monitoring.

Paul

Hi Paul,

Thanks for this. I'll keep looking at the CI builds and will try to
diagnose the timeouts when they happen.

Regards

Jeff

This morning, I see that for s390x in testing, there was a timeout and
failure (both in 380_cancel_user_defined_with_pids), as well as two passes:

https://ci.debian.net/packages/g/gscan2pdf/testing/s390x/

armel had a failure that was unrelated to gscan2pdf:

https://ci.debian.net/data/autopkgtest/testing/armel/g/gscan2pdf/26567445/log.gz

I also see that all the other architectures have passes, but this is not
reflected by the summary page:

https://ci.debian.net/packages/g/gscan2pdf/

Any idea why?

Given that gscan2pdf is a desktop app, I assume there are no users on
s390x. Hence, I'm not sure of the value of debugging on s390x. According to:

https://www.debian.org/doc/debian-policy/ch-source.html

DEB_HOST_ARCH and DEB_TARGET_ARCH should be set. I'm tempted to test for
s390x in the test target in debian/rules and skip accordingly. Objections?

Given that the packages in stable are, well, stable, and I made no
changes to those parts of gscan2pdf, I wonder what happened in the rest
of the environment between stable and testing. I'd be interested to
confirm this by pushing the stable version of gscan2pdf through testing
s390x a couple of times.

Regards

Jeff

Hi Jeff,
"""
"Too many open files" at t/380_cancel_user_defined_with_pids.t line 40.
"""
Might that be a hint? Maybe a race condition under a high amount of
parallelism built?

Again, the failure rate with 27 hour timeout wasn't that high, but 27
hours is really long... Hmm, autopkgtest should have sane default
timeouts, and now I take a careful look, I notice that the timeout
happens during *building* of your package, even *before* the test is
started. Do you really need the "build-needed" restriction? Our spec [1]
has this:
     Please use this considerately, as for large builds it unnecessarily
     builds the entire project when you only need a tiny subset (like the
     tests/ subdirectory). It is often possible to run ``make -C tests``
     instead, or copy the test code to ``$AUTOPKGTEST_TMP`` and build it
     there with some custom commands. This cuts down the load on the
     Continuous Integration servers and also makes tests more robust as
     it prevents accidentally running them against the built source tree
     instead of the installed packages.
What's more, autopkgtest is really ment to test as-installed packages,
building the binaries again makes it very easy to test the freshly built
binaries *instead of* the installed binaries. (Not saying that's the
case here, but building the binaries is something that I think tests
should hardly ever do.)

This page only shows "pure" runs because showing failures cause by
packages in other suites that are not allowed to migrate is confusing.
We do want to change that page, but UI is difficult and we haven't found
the time and energy to implement it.

autopkgtest also supports an "Architecture" field in the d/t/control
file that takes negation. So, why not use "Architecture: !s390x" if you
just want to disable the test there?

Not sure you can do that, except by using "+really" version number
uploads. Also, see:
https://ci.debian.net/data/autopkgtest/stable/armel/g/gscan2pdf/23364480/log.gz
So stable is not free from the issue.


Paul

[1]
https://salsa.debian.org/ci-team/autopkgtest/-/raw/master/doc/README.package-tests.rst

gscan2pdf is now only a transition package.