Dear Maintainer,

While working on updating the docker.io package in experimental, I've noticed an
autopkgtest failure on arm64 that did not happen on amd64:

https://ci.debian.net/packages/c/crowdsec/unstable/arm64/51171995/

274s === RUN   TestOneShot
274s     journalctl_test.go:172: Expected log output 'journalctl: invalid option' but got nothing !
274s --- FAIL: TestOneShot (0.05s)


Looking at the code in question, I don't see anything obvious that could
be attributed into changes of the docker.io package:

https://sources.debian.org/src/crowdsec/1.4.6-9/pkg/acquisition/modules/journalctl/journalctl_test.go/?hl=104#L104-L178

Please have a look at the code and form an opinion whether this
indicates a flaky test (in which case disabling it it might be a good
way to proceed), or whether code changes in either the docker.io package
or crowdsec would be appropriate.

Thank you!
-rt

Hallo Reinhard,

Reinhard Tartler <siretart@tauware.de> (2024-09-01):

I'd suggest taking this specific issue out of your todo list before
considering an upload to unstable: I'd rather avoiding diving into
unstable + experimental overlay with different results per arch at
this point, and deal with whatever happens in unstable.

(Of course letting this bug report get an upgraded severity, possibly
resulting in crowdsec's getting out of testing for a little while.)

Would that approach be sufficient to let you make more progress?


Danke dir,

On 2024-09-01 08:34, Cyril Brulebois wrote:

Hey Kibi, thanks for your prompt response!

I'm not really familiar with the crowdsec package. I just found the
article at
https://debamax.com/blog/2023/06/12/packaging-crowdsec-for-debian-bullseye/
and it looks really interesting. I'll have to take a closer look.

As for this issue, I was hoping you as maintainer could help by
forming an opinion on how to proceed.

That'd be a shame if it came to that, because that would also delay
testing migration for other packages (such as podman, etc.)

I currently suspect that this is actually a timing issue indicating a
flaky
test. I've retriggered the autopkgtest to see if that theory holds.

best,
-rt

https://ci.debian.net/packages/c/crowdsec/unstable/arm64/51181261/

Does that indicate the test is flaky in some way?

Unfortunately, the test doesn't provide much diagnostics and I fail to
reproduce it.

Hi!  I noticed this as part of the golang-go.crypto migration, see
random test failure here:

https://ci.debian.net/packages/c/crowdsec/testing/s390x/64965099/

I think this has happened on all archs at some point, right?

For flaky tests, I usually report these upstreams and mask it out of the
Debian build.  I initially thought keeping flaky tests around was
acceptable, but I've realized it causes serious Debian testing migration
pains so I believe we must not have any flaky tests in Debian.

Here is an idea:

override_dh_auto_test:
	dh_auto_test $(DH_BUILD_OPTS) -- -skip=ExampleGExpect_SendSignal
        -dh_auto_test $(DH_BUILD_OPTS)

This allows the failure message to still become available in build logs,
for future references in bug reports.

If it only has happened on a particular arch, here is an idea:

ifneq (,$(filter $(DEB_TARGET_ARCH), ppc64el))
override_dh_auto_test:
	dh_auto_test $(DH_BUILD_OPTS) -- -skip=ExampleGExpect_SendSignal
        -dh_auto_test $(DH_BUILD_OPTS)
endif

For complex workarounds I have used this:

ifneq (,$(filter $(DEB_TARGET_ARCH), i386 ppc64el riscv64 s390x))
override_dh_auto_test:
ifneq (,$(filter $(DEB_TARGET_ARCH), i386))
	dh_auto_test $(DH_BUILD_OPT) -- --skip=TestAllocate
endif
ifneq (,$(filter $(DEB_TARGET_ARCH), ppc64el riscv64 s390x))
	dh_auto_test $(DH_BUILD_OPT) -- --skip='TestTreePersistent|TestBuffer'
endif
	-dh_auto_test $(DH_BUILD_OPT) -- --timeout=20m
endif

/Simon