package: jenkins.debian.org
severity: wishlist

hi,

in https://salsa.debian.org/qa/jenkins.debian.net/-/merge_requests/163
James Addison suggested to use --max-container-depth=3 (or 5) for
when diffscope runs into a timeout on a package. (or rather not then,
but always, which why this MR wasnt accepted.)

However this lead to the following discussion on irc:

<h01ger> jayaddison: thanks for closing MR163!
<jayaddison> no probs!  thought it might confuse/distract someone later otherwise :)
<h01ger> jayaddison: thinking about it, we could probably run diffoscope with the reduced depth option on those packages we know diffoscope timeouts, eg all packages listed more than 4 times on https://tests.reproducible-builds.org/debian/index_breakages.html
<h01ger> the breakage job could generated that list, and then reproducible_build.sh could consume it and act accordingly
<jayaddison> interesting idea.  that was going to be my next question (how to maintain the list of relevant packages)
<jayaddison> I'm slightly averse to adaptive testing because it becomes a maintenance/debug challenge over time, but am thinking about it
<jayaddison> what's the effect of a diffoscope timeout at the moment?  no output at all?
< h01ger> thinking about it, this could cause an interesting jojo effect: git-annex is tested, diffoscope runs into a timeout. this happens more than 4 times -> git-annex is put on the list of packages which diffoscope is run on with reduced depth. thus, (hopefullly! now) diffoscope doesnt run into a timeout
          anymore. this happens so often that git-annex appears less than 4 times on the breakages page ->
<h01ger> git-annex is tested without a timeout again, rinse repeat, until diffoscope or git-annex is improved
<h01ger> yes, its basically empty, see eg https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/git-annex.html
<h01ger> basically true for any link on https://tests.reproducible-builds.org/debian/index_breakages.html
<jayaddison> ok.  I guess 'in theory' it'd be nice for diffoscope to diff to depth 1, then write output, then proceed to depth 2, rewrite output, etc.
<h01ger> nah, depth 3/4/5 should be fine, i hope
<jayaddison> first few levels are generally .deb, data/xz or something, and... I forget.  but fast/inexpensive compute-wise
<h01ger> yeah
<jayaddison> if we could write output after each layer, there should always be 'something' to show on the results page even if it later fails at depth 8 or whatever
<jayaddison> s/fails/times-out
<h01ger> i'd just go with layer X for a start, i doubt its useful to make this more sophisticated, esp from the start. also its sophisticated enough to only invoke this for a few timeouting packages

or maybe even simpler: first run diffoscope normally, then if that runs into a timeout,
run with --max-container-depth=3 (or 5). I think this would be acceptable with
only 286 suite/arch/package combinations currently which run into a timeout.

I'd still like to store/record somewhere (for later reading, eg to then semiautomatically
add this information to notes.git) that diffoscope ran into a timeout on this package,
but for the moment I don't have a plan how to that exactly.

Package: jenkins.debian.org
Followup-For: Bug #1067232
X-Debbugs-Cc: holger@layer-acht.org
complexity.

In reproducible_build.sh it looks like there are some IRC notifications: it
could make sense to suppress the retried-diffoscope-results, or emit a
noticably different message for those to distinguish them from the initial
attempt?

no need for that cc:, i'm subscribed to the package.

indeed.

yes. (and I apologize to your eyes for having seen that IRC notification
"logic"... I've looked at it yesterday for too long and only came up with
a small improvement while that whole thing could use an overhaul.)

It also occured to me that we then could diffoscope with a (way) lower timeout,
eg 60min instead of 155min, because hardly never diffoscope runs longer than
30min, and if it does, it usually runs 155min until it gets killed.

I'm not sure we're collecting data on this, so far this is just an educated guess. :)