glib2.0 fails to build from source on amd64. This happens both on the
buildd
https://buildd.debian.org/status/fetch.php?pkg=glib2.0&arch=amd64&ver=2.54.2-2&stamp=1513222982&raw=0
and on the reproducible infrastructure
https://tests.reproducible-builds.org/debian/logs/unstable/amd64/glib2.0_2.54.2-2.build2.log.gz.
Though reproducible experiences more failures (including gwakeup and
gmenumodel). It seems unlikely that this fixes itself somehow.

Helmut

Control: clone -1 -2
Control: clone -1 -3
Control: clone -1 -4
Control: clone -1 -5
Control: retitle -1 glib2.0: FTBFS on amd64 buildd: gdbus-peer test: assertion 'source->ref_count > 0' failed
Control: retitle -2 glib2.0: sometimes FTBFS on reproducible-builds: gwakeup, gwakeup-fallback tests terminated by SIGALRM
Control: found -2 2.50.3-2
Control: found -2 2.54.1-1
Control: retitle -3 glib2.0: sometimes FTBFS on reproducible-builds: gdbus-threading test: assertion failed (elapsed_msec < 8000): (8220 < 8000)
Control: found -3 2.50.3-2
Control: retitle -4 glib2.0: sometimes FTBFS on reproducible-builds: gmenumodel test: assertion failed (items_changed_count == 1): (0 == 1)
Control: retitle -5 glib2.0: sometimes FTBFS on reproducible-builds: tar: ./usr/share/locale/en_??/LC_MESSAGES/glib20.mo/: Cannot savedir: Not a directory
Control: found -5 2.54.1-1

I don't think these have the same root cause. r-b doesn't seem to get
the assertion failure seen on the buildd, but it does get different
test failures. Please use separate bugs for what appear to be separate
issues.

Let's use #884654 for the failure on the buildd, and its clones for the
failures on reproducible-builds. All of these might need to be downgraded
to non-RC if they can't be reproduced elsewhere or understood, but I'll
leave them all RC for now.

We are unlikely to be able to get anywhere with most of these test
failures unless someone can reproduce them in an environment that leaves
core dumps, or at least captures backtraces.

The failure on the buildd is new, although I think I might have seen it
before in local testing (but never reproducibly). It's probably some
rarely-hit race condition?

Clone -2 is not new. It has also happened (more often for
gwakeup-fallback) in earlier versions:
https://tests.reproducible-builds.org/debian/rbuild/stretch/amd64/glib2.0_2.50.3-2.rbuild.log
https://tests.reproducible-builds.org/debian/rbuild/buster/amd64/glib2.0_2.54.1-1.rbuild.log
https://tests.reproducible-builds.org/debian/logs/buster/armhf/glib2.0_2.54.1-1.build2.log.gz
Presumably it doesn't happen on the real buildds because the reproducible
build workers are more heavily-loaded, or have more or fewer CPUs, or
some other factor. It isn't the build vs. build2 variation, because this
test failure has been seen in both.

Clone -3 can also be seen in
https://tests.reproducible-builds.org/debian/rbuild/buster/amd64/glib2.0_2.54.1-1.rbuild.log

Clone -4 might be new, or just rare.

Clone -5 (a build failure inside dpkg-builddeb, not a test failure)
I don't know what is going on, and it doesn't seem particularly likely
to be a GLib bug - GLib just puts files in a tree like any other package,
so I'm not sure how it would trigger this particular failure. It can be
seen in these logs:
https://tests.reproducible-builds.org/debian/rbuild/buster/i386/glib2.0_2.54.1-1.rbuild.log
https://tests.reproducible-builds.org/debian/rbuild/unstable/armhf/glib2.0_2.54.2-2.rbuild.log
(not build2, so we presumably can't blame disorderfs either).

    smcv

(Re-sending to the correct bug, sorry for the noise. Please reply to
this copy, not the previous one.)

It looks as though the failing test is always /gwakeup/threaded or
/gwakeup/threaded-fallback (we see /gwakeup/semantics{,-fallback} pass
before the test fails, and threaded{,-fallback} is the only other test
in this executable).

That's a multi-threaded stress-test for GWakeup:

- set an alarm for 60 seconds, so that on failure we don't block forever
- create 50 threads
- create 5 "tokens", each with a TTL of 100K
- each token is owned by one thread at a time
- each thread is woken up when it owns at least one token, and at that
  point it will do:
  - for each token:
    - decrement the TTL
    - if the TTL is nonzero, choose a random thread and send the token
      to that thread
- when all tokens' TTLs are zero, wake up the main thread, which exits

It reliably passes on the buildds (even those for weaker architectures
like mips). Are the reproducible-builds machines perhaps doing other
things in parallel with running the glib2.0 test suite, such as building
another package?

The 60 second alarm is completely arbitrary (it was increased from
30 because some ARM systems took more than 30 seconds to finish). The
thread count, number of tokens and token TTL are probably also completely
arbitrary, and could be reduced, at the cost of being less likely to
detect bugs (unfortunately this sort of threading stress-test is usually
probabilistic).

    smcv

In 2.54.2-3 I've reduced the number of threads and the TTL, both by a
factor of 10. I don't think this is really a correct solution (it's now
less likely to find bugs), but it'll let us see what happens. If the
builds succeed I'll drop this bug down to a non-RC severity.

    smcv

Control: severity -1 important

This mitigation seems to have been successful.

I haven't sent the bug report upstream yet - I have no objection to some
other GNOME team member upstreaming it, if someone gets there before I do.

    smcv