When a new version of the package under test enters the archive during a
test run, the tests towards the end of the run may use the binaries of
the new version, but debci lists it as a test of the old version.

If these tests fail because the new version contains a regression, this
gives a "fail" entry for the _old_ version, and hence the false
impression that the problem must be somewhere else.

Example: row 2018-09-20 11:04:29 of
https://ci.debian.net/packages/p/pymca/unstable/amd64/
(This is probably #909379 not a pymca regression, but this log isn't
proof it isn't)

Possibly related to #896023 / #902027, but those are about testing vs
unstable, while this is changes within a suite.

This is moderately unlikely, but not hugely so: for pymca (~15min of
tests and a big enough dependency chain that it gets tested ~2x/day),
~2% chance per upload.

I'd guess the best solution is to make this a tmpfail, as testing an
already-superseded version is probably a waste of resources.

Hi Rebecca,

Thanks for reporting this issue. I didn't came across it yet.

A race condition indeed. It's worse than you describe, not all tests are
run with the same binaries.

No, not related. That issue I don't consider a bug anymore, but an
autopkgtest feature. In the Debian archive testing case, we just need to
try harder to detect which version we want to test against. The old
behavior is currently not possible in our setup because we run with the
new --disable-apt-fallback option.

What kind of logic do you have in mind?

Paul

Probably because it's so rare, and easy not to notice if the status
didn't change.

If any binary of the package under test is being installed and isn't the
same version as the source, abort the run as a tmpfail.  (At least in a
standard debci run, as some documented uses probably want to allow such
mismatches:
https://sources.debian.org/src/autopkgtest/5.5/doc/README.running-tests.rst/#L65
)

I don't know if this is actually worth the effort; it also assumes
debci's displayed Version is the source version.

Hi,

As I mentioned in my previous reply, I consider the current behavior a
feature, so I don't like the logic you mention above. I have code laying
around for autopkgtest support of specifying the version one wants to
test. failing (or tmpfail, but I don't think I like that) in the case
where the version isn't matching the one requested makes sense to me.

What autopkgtest outputs is the version of the package it took the test
from, which is still correct. It is however confusing in cases like this
(and in cases like those reported in bug #896023). debci just displays
whatever autopkgtest outputs here.

I guess the point is really that people, understandably, take the
version to mean something else than it really means as in most cases the
version is the same to what they believe it means. debci could stop
showing the version, but I don't think that is actually helpful except
in rare cases.

Paul

What, if anything, do we actually disagree on?  I've already agreed that
it should stay _possible_ to test non-matching source and binary
versions: I just don't want this happening where it is unintended and
confusing.

If the best place to fix this is outside autopkgtest, feel free to reassign.

Why not?  I suggested tmpfail because this isn't an actual problem with
the package under test, and the whole point of this bug is that debci
shouldn't claim that it is.  (Debci fail = code 4, 6 or 12
https://sources.debian.org/src/debci/1.13/bin/debci-generate-index/#L168 ).

However, if we're passing a requested version to autopkgtest we should
probably check that it is updated when retrying a tmpfail, to avoid
endless retry loops.

Agreed that generally hiding the version would be worse.  (Hiding the
version / changing it to something like 'mixed-versions' *only* when
these issues happen would be a valid solution, but may or may not be
practical.)

Hi Rebecca,

Your quote mentions a different case than the one used by debci. I
concluded that you wanted to support that case to test non-matching
versions, but not the case that debci uses. I don't want to limit debci
to actually run these mismatching combinations.

Yes, but defining that is what we are trying to do now. I don't think
defining it unambiguously is easy.

Well, if we can clearly define what needs fixing and how, I bet it is
autopkgtest. And until we have defined what needs fixing, we can't even
reassign.

Yes, but in which exact case do you want the tmpfail? That isn't clear
to me.

Well, these just match autopkgtest, don't they?

Well, that exactly a reason why I don't like the tmpfail. I don't
remember if debci reschedules tmpfails in unstable, but britney2 (the
tests in testing) only reschedules tmpfails if it believes they are
still needed. So from britney2 side tmpfail is fine and probably good in
the case of version mismatch.

debci can only do that if autopkgtest supports that.

Paul

https://ci.debian.net/data/autopkgtest/testing/amd64/p/python-feather-format/3483361/log.gz
appears to have the opposite problem: it finds the new source (-3), but
as this had not yet been built, it used the old binary (-2).

The testing excuses pages for both pandas and python-feather-format
currently list this as a test of -3, which it is not.  The debci summary
page doesn't appear to list it at all - is that intended to fix this bug?

(It failed because -2 is uninstallable due to the Breaks: for #943925,
which is fixed in -3.  A test of actual -3 has since been run, which
passed.)

Hi Rebecca,

The root cause here should be a mirror that is out-of-date, as britney
only schedules the tests when the build is done (so your statement isn't
correct). It's annoying that it can happen.

Further, this is not a huge issue (albeit confusing for people looking
into). It will be retried after one day and all should be fine.

It's the opposite that I worry about, a version X which passes while if
it would have correctly run with X+1 it would show a fail. So, a
regression that goes unnoticed due the issue you mention above.

No, we changed debci a bit to only show "pure" runs in the overview
page, i.e. runs without pinned packages. Follow the link under "britney"
and you'll go to the page with all the runs scheduled by the migration
software.

Paul

The autopkgtest is timestamped 15:22, the amd64 build finish 15:47.
Maybe this test was triggered by something other than
python-feather-format itself?

It now is.

Hi Rebecca,
--pin-packages=unstable=src:pandas,src:python-feather-format,src:python-skbio,src:statsmodels

The first package listed there is the triggering package. The rest come
from (versioned) Depends/Breaks/Conflicts relations.

Paul