#1071563 emacs: Don't let test suite failures block the build

Package:
src:emacs
Source:
src:emacs
Submitter:
Sean Whitton
Date:
2024-05-28 17:57:02 UTC
Severity:
normal
#1071563#5
Date:
2024-05-21 10:20:40 UTC
From:
To:
Hello,

I propose that we make test suite failures non-fatal to the build.

The test suite has got quite flaky in recent years, and uploading Emacs
often requires issuing several giveback requests for random failures.

Patching individual tests to mark them as flaky does not scale.
#1071563#10
Date:
2024-05-22 22:41:58 UTC
From:
To:
Hello,

I think we should move the tests to run under autopkgtest, rather than
during the build.  Debian's autopkgtest infrastructure, with things like
blocking migration, is by now quite sophisticated.

On IRC it's also been suggested that we also

- mark dired-test-bug27243-02 as flaky (thanks Arto Jantunen); and

- in the new autopkgtest, don't run the tests in parallel.
#1071563#15
Date:
2024-05-26 23:02:08 UTC
From:
To:
Sean Whitton <spwhitton@spwhitton.name> writes:

I don't have a good overall sense of what's been failing lately, but I
did wonder if just marking that test (and any others that have been
(unnecessary) trouble) as flaky, and going back to not running the tests
in parallel on the autobuilders might be sufficient.

As long as the package tests restrict transitions, moving the tests
there, if we can, may be fine too, but if we could keep the spurious
test failure rate low enough (pretty low) without *too* much effort,
then I think catching failures during the build is somewhat preferable
(in case the error is serious).

Thanks
#1071563#20
Date:
2024-05-27 10:43:50 UTC
From:
To:
Hello,

I agree with you in principle, but I'm proposing this precisely because
I think that the level of effort has already become too high.  I find
that I have to do multiple givebacks every upload.
#1071563#25
Date:
2024-05-27 19:32:43 UTC
From:
To:
Sean Whitton <spwhitton@spwhitton.name> writes:

Understood, but I was wondering if that might be a more recent
phenomenon, perhaps because of the switch to parallel testing (or
something else we could fix).

Of course if things have just gotten worse for other reasons that are
outside our control, then that's a different matter.
#1071563#30
Date:
2024-05-28 10:43:26 UTC
From:
To:
Hello,

I think it's just that upstream has lots and lots of tests now.
#1071563#35
Date:
2024-05-28 17:53:49 UTC
From:
To:
Sean Whitton <spwhitton@spwhitton.name> writes:

Well, sure, but in the past I don't recall having to (re)try very often,
(a few cases where I had to track down and (most of the time) mark a
test as flaky).

But maybe 29 is worse, or maybe I misremember...