nmu rust-async-broadcast_0.7.1-1 . riscv64 . unstable . -m "Rebuild to fix previous broken build"

https://packages.debian.org/sid/riscv64/librust-async-broadcast-dev/filelist
looks really weird, and this is not seen on other architectures.

I suspect this broken binray package is the reason why the package fails on
debci and is holding up the rust-zbus transition

That is not the actual contents of the .deb. Download the .deb and
open it in file-roller or something to check for yourself. Or look at
the build log.

https://buildd.debian.org/status/package.php?p=rust-async-broadcast

Some Rust autopkgtests take a very long time to complete which is
especially noticeable on riscv64 which I think may be the slowest
autopkgtest infrastructure.

Some rust autopkgtests have additional trouble because
1. No riscv64 packages are in Testing yet
2. The Debian Rust team sets skip-not-installable by default. That
means the migration-reference autopkgtest will always show as neutral
since the dependencies are uninstallable when migration-reference is
the only trigger. I think skip-not-installable by default is bad for
other reasons but I think this situation makes it worse.

See for instance
https://ci.debian.net/packages/r/rust-gtk4/testing/riscv64/ which
passes once in a while. But I don't believe riscv64 is delaying the
rust-zbus transition.

Thank you,
Jeremy Bícha

Hi,

That's why the test timeout on riscv64 is double that of other
architectures:
https://salsa.debian.org/ci-team/debian-ci-config/-/blob/master/config/production/nodes.d/riscv64.yaml?ref_type=heads#L11

This is indeed a problem, largely because of 2.

I regret I added skip-not-installable to autopkgtest. It does more harm
than it helps.

Until yesterday at least rust-libseccomp was blocking a lot. I was
hoping that once that migrated (happend yesterday) new tests would have
more chance of succes, as it seems that several tests pass in a pure
unstable environment. So this hints at missing *versioned* dependencies.
But maybe it will resolve itself by time ordering.

Paul

Unfortunately in many cases, the performance difference seems a lot more
than a factor of two.

Take rust-gix for example, looking at the results for testing migration
tests I see

18-28m on amd64
53-59m on arm64
33-47m on armel
33-48m on armhf
1h59m-2h2m on i386
49m-1h37m on ppc64el
23m-55m on s390x

on riscv64 it was timing out after over 8 hours. To unblock things I modified it
to only test a single feature configuration. It then passed in 29 minuites.

If we assume all feature configurations take the same time (very rough) then
that suggests running the full set of tests on riscv64 would take of the order
of 23 hours. Over 10 times worse than the next-slowest architecture.

It seems to me that britney's autopkgtest scheduler sometimes fails to pick
up that stuff needs to go in together. Even though the main part of britney
knows that they do.

The end result of this is frequently that the fallback dependency solver
gets triggered and autopkgtest tries to run the tests from testing against
the package from unstable. This often results in a "crate directory not
found" error. In other cases the fallback depsolver is not invoked or
doesn't succeed and the testsuite fails with a "fail badpkg" error.

Why this is happening I'm not entirely sure. Though I have some theories

In some cases I suspect it happens because the dependencies are indirect.
or even involve going down the tree and back up again.

In some cases I suspect it happens because riscv64 testing is currently
a basket case, so the packages are already uninstallable in testing.

In some cases I suspect it happens due to the special treatment of arch
all packages (rust team packages are arch any, but Jonas's packages
are arch all). For example if we look at the migration page for
rust-event-listener, it seems britney has done the right thing on amd64
and arm64 (where arch all packages are required to be installable) but
not done the right thing on architectures where arch all packages are
not required to be installable.

https://ci.debian.net/packages/r/rust-async-broadcast/testing/riscv64/
being retried over and over again without progress?

In contrast to the rust-gtk5/testing/risc64 example, in
rust-async-broadcast/testing/riscv64 failures have logfiles that indicate
that the actual tests don't even run, still it is being counted as a
failure.

Also, looking at the description of "skip-not-installable":

    This restrictions may cause a test to miss a regression due to
    installability issues, so use with caution. If one only wants to
    skip certain architectures, use the ``Architecture`` field for
    that.

    This test might have test dependencies that can't be fulfilled in
    all suites or in derivatives. Therefore, when apt-get installs the
    test dependencies, it will fail. Don't treat this as a test
    failure, but instead treat it as if the test was skipped.

I need some help to understand how skipped tests lead to delaying the
package migration to testing. My naive understanding is that this flag
would rather allow issues to go through to testing?

Changing that would be a one-liner to
https://salsa.debian.org/rust-team/debcargo/-/blob/master/src/debian/control.rs?ref_type=heads#L149,
but before proposing that, I'd need to understand your concern better.


My question is basically: what needs to be done so that
https://tracker.debian.org/pkg/rust-event-listener can actually migrate
testing?

On Mon, 5 Aug 2024 12:57:27 +0100 Peter Green  <plugwash-urgent@p10link.net> wrote:
 > > That's why the test timeout on riscv64 is double that of other
architectures:
 >
 > Unfortunately in many cases, the performance difference seems a lot more
 > than a factor of two.
 >
 > Take rust-gix for example, looking at the results for testing migration
 > tests I see
 >
 > 18-28m on amd64
 > 53-59m on arm64
 > 33-47m on armel
 > 33-48m on armhf
 > 1h59m-2h2m on i386
 > 49m-1h37m on ppc64el
 > 23m-55m on s390x
 >
 > on riscv64 it was timing out after over 8 hours. To unblock things I
modified it
 > to only test a single feature configuration. It then passed in 29
minuites.
 >
 > If we assume all feature configurations take the same time (very
rough) then
 > that suggests running the full set of tests on riscv64 would take of
the order
 > of 23 hours. Over 10 times worse than the next-slowest architecture.
 >
 > > So this hints at missing *versioned* dependencies.
 >
 > It seems to me that britney's autopkgtest scheduler sometimes fails
to pick
 > up that stuff needs to go in together. Even though the main part of
britney
 > knows that they do.
 >
 > The end result of this is frequently that the fallback dependency solver
 > gets triggered and autopkgtest tries to run the tests from testing
against
 > the package from unstable. This often results in a "crate directory not
 > found" error. In other cases the fallback depsolver is not invoked or
 > doesn't succeed and the testsuite fails with a "fail badpkg" error.
 >
 > Why this is happening I'm not entirely sure. Though I have some theories
 >
 > In some cases I suspect it happens because the dependencies are indirect.
 > or even involve going down the tree and back up again.
 >
 > In some cases I suspect it happens because riscv64 testing is currently
 > a basket case, so the packages are already uninstallable in testing.
 >
 > In some cases I suspect it happens due to the special treatment of arch
 > all packages (rust team packages are arch any, but Jonas's packages
 > are arch all). For example if we look at the migration page for
 > rust-event-listener, it seems britney has done the right thing on amd64
 > and arm64 (where arch all packages are required to be installable) but
 > not done the right thing on architectures where arch all packages are
 > not required to be installable.
 >

Converting the librust-* packages to arch:all is wrong anyway since this
can break crosscompling in some cases, but that's only part of the
problem here.

 From #debian-release:

"jochensp    werdahias: rust-async-channel breaks
rust-async-executor/1.12.0-3 and rust-mpris-server/0.7.0-1 and probably
more, if I get it right"

rust-libseccomp migrated but rust-async-broadcast can't migrate due to
rust-event-listener which seems to be missing a B+R.


best,

werdahias

Hi

What skipping means isn't up to autopkgtest to determine, but up to the
consumers of the test results. For migration to testing, that britney2
that's owned by the Release Team. We, the Release Team, want britney2 to
prevent migration:
a) the autopkgtest of a package doesn't fail in testing, but the test
fails with $something from unstable (regression). That $something should
be blocked.
b) the test is new to testing, but it fails (weird definition of
regression).

Good question. It seems that britney2 did schedule some combination
"correctly", but not all. That is probably due to the order in which
packages were uploaded. I.e. if a package needs some other package,
britney2 will schedule the test for both triggers and if it passes, it
counts for both. However, if the dependent-on package is later updated
again, britney2 doesn't see the relation and will not schedule the
combination. If the test than fails, the package is blocked. If there
would be a versioned Breaks (not sure if the package is really broken,
or merely the test, then a Breaks is a bit overkill), than britney2
would again trigger the combination.

Does this help to tell the right answer? If this is only a test issue,
and not really a versioned Breaks missing, I'm willing to manually
trigger the right test combination. But then people need to hold off
with uploading, otherwise the results might come too late.

Paul

So I think I managed to find the right combination of packages.
Basically, we need all of
- rust-async-channel
- rust-async-process
- rust-event-listener
- rust-async-broadcast

to migrate together. When triggering with all of these four sources from
unstable, I end up with a successful autopkgtest:
https://ci.debian.net/data/autopkgtest/testing/riscv64/r/rust-async-broadcast/50019012/log.gz

How to convey that to britney/debci?

Is that something that could be added as a hint by the release team, or
does that require adding what Breaks to what package exactly?

Hi,

Because they break each other, or merely because the test fail?

Ack, thanks.

Depending on the answer above, I'll need to manually trigger those jobs,
just like you did, but then with britney2 credentials.
need to figure out which package Breaks which package. I can't say from
this distance, because I can't distinguish package breakage from test
breakage.

I suddenly realize that rust packages are special in Debain. Do the
involved packages actually do anything besides providing source? Can
other packages actually break due to packages that provide source? I
recall rust packages are mostly meant to be used to build Debian
packages, and not intended to be used by users. Am I correct in that
understanding? Than, what happens if package A would migrate and package
B not: would it mean FTBFS in testing of reverse Build-Depends package C
until B migrates too? In the latter case, I think package A should have
a versioned Breaks on package B in testing.

Paul

Hi,

Wait, I think I now understand what you tried to say earlier (sorry,
it's very exciting at $dayjob and I'm not always as sharp at home as I
normally am). Apparently librust-async-io stopped providing
librust-async-io-2+default-dev (and others), or something along those
lines. I think we're running into limitations of britney2 here where it
isn't aware of how to deal with situations where only one package
Provides something else during the excuses phase. It's already for a
long time on my todo list, but alas.

Under the assumption that the above assessment above is correct, the
packages will migrate together because otherwise they become
not-installable and the second phase of britney2 protects against that.
Let me schedule the tests.

Paul