The log files included in /usr/share/doc/binutils/tests/ include many
variable things (timestamps, timings, PIDs, temporary filenames,
usernames, addresses) that vary between builds.

I'm not sure what the rationale for including these test logs in the
package is, but from a reproducible builds perspective, ideally these
would simply not be included at all.

When running the build with DEB_BUILD_OPTIONS=nocheck, the build logs
are excluded, and results in a reproducible build. Perhaps you could
create a build profile that doesn't include these logs, and use it in
builds in which you want the logs?

If the logs must be included in the packages, the attached patch
attempts to sanitize the log file by removing everything that varies
between builds; I'm not sure if this removes too much data to be useful,
but it should make binutils reproducible at least in bullseye
(unstable/experimental also varies build path in the reproducible builds
test infrastructure, which triggers other issues). This approach does
feel a bit prone to playing whack-a-mole, so long-term this may not be
viable.

Please consider removing or sanitizing the logs so that this part of the
essential build toolchain can itself be built reproducibly!


Thanks for maintaining binutils!


live well,
  vagrant

that's not an option.  this is all useful information for debugging purposes,
and you'll find that in the GCC packages as well.  Having it turned off by
default defeats the purpose.  I think I filed a bug report that builds should be
able to generate their own artifacts, as the autopkg tests are doing that,
however that probably will take a while to implement.

Or you could add a override database for files which are expected to differ.

Matthias

So, my attempt at sanitizing them removed too much information; ok.

Why can't these test logs be output to the build log instead of being
embedded in the package? What's the use-case that needs them to be in
the package itself?

From what I recall, binutils was reproducible in debian testing for a
while before these logs were added. While GCC was not ever reproducible
thus far, it is another core part of the toolchain that I would hope
could one day be made reproducible.

I would be very interested in this bug report; against what package?

This is considerably more complicated than running a checksum on the
resulting .deb files and is another opportunity for bugs to lead to
incorrect reproducibility results... which I think has actually happened
when trying this kind of approach in the past, though I don't have a
reference off the top of my head.

Exploring avenues to put files like this into some separate artifact for
things that are not reproducible might be one avenue; I know that the
debian-installer packages ship some artifacts which are not
.deb/.dbgsym/.udeb... but this still makes it more difficult to compare
the resulting objects... worried about how to get that right, but maybe
we, as reproducible builds, need to explore that an an option?


live well,
  vagrant

Vagrant Cascadian wrote:

I would very much underline Vagrant's hesitation regarding a
centralised database. Such overrides would get out of date (or at
least out of sync) amongst many many other concerns including it,
albeit at a slight stretch, being a possible attack vector.

The ability to check reproducibility with no other knowledge or tools
other than cmp(1) or sha256sum(1) etc. does not seem to be that
important as it might initially appear but it extremely valuable as it
is so simple, engendering trust, lowering the barrier to entry,
reducing mistakes, etc. etc.

(Vagrant, are you perchance thinking of RPM? If I recall correctly, the
signatures in question there are embedded in the .rpm itself so you
need a special tool to even extract them.)


Best wishes,

cat'ing to stdout is possible, yes, however that will add to the log size.
gcc-N-test-results is a 13MB compressed package. This doesn't scale.

hmm, that was Ubuntu only:
https://bugs.launchpad.net/launchpad/+bug/1845159

Feel free to forward that. sbuild, pbuilder?

There is already the BYHAND (and automatic BYHAND) mechanisms for files
that get installed outside of pool/ in the Debian apt repository. Each
one needs support from dak too though.

https://salsa.debian.org/ftp-team/dak/-/tree/master/scripts/debian/
https://codesearch.debian.net/search?q=byhand+path%3Adebian&literal=0

The other option would be to put the files in a test results .deb file,
but that would still mean repro-builds folks would compare them, unless
there were a naming convention that could be used to ignore them.

It strikes me that these files are most similar to .buildinfo or the
build logs in that they are data *about* the builds. I've wanted
maintainers to be able to also upload build logs with their binary
builds and started a WIP patch for that.

https://salsa.debian.org/pabs/dak/-/commits/maintainer-build-logs

It looks like dpkg-genchanges can already add more files through the
debian/files input file. I managed to get my build log included in my
.changes file using this mechanism. I then attempted to upload it to
the Debian archive, queued gladly accepted the upload but dak rejected
it saying that it looks like a BYHAND package:

   whowatch_1.8.6-2_amd64.changes: whowatch_1.8.6-1_amd64.build.log looks like a byhand package, but is in section build

I suggest that the dpkg-dev maintainer and the ftp-masters should be
talked to about this topic. Probably the right mechanism is to have a
convention in debian/files similar to how dbgsyms are represented and
similar to byhand but the files go into the pool like .deb files do.

   whowatch_1.8.6-1_amd64.build build optional automatic=yes
   whowatch-dbgsym_1.8.6-2_amd64.deb debug optional automatic=yes

Hi Paul,
[..]

My gut feeling is that this is the avenue we want to explore. Having a
separate mechanism to capture this build-specific metadata would be an
elegant solution and, as you imply, having the logs would have QA
advantages as well as permit reproducible builds. The system could be
generic enough for future use-cases that we cannot envisage too.

This sort of thinking always makes me a little uneasy. We have taken
great pains over the years that no special knowledge, tools or tricks
are required to compare the artifacts of a Debian build with respect
to reproducibility.

All you need right now is sha1sum(1) or any cmp(1)-like command-line
tool. This is partly due to aesthetics but I have further observed
that distributions and platforms that have not adopted this fail-safe
approach are always at a marked disadvantage. This would appear to be
a superficial niggle, but it makes a huge difference in mindset with
many 1st and 2nd effects.


Best wishes,

Agreed that this is the best option. In order to standardise the
naming, structure and organisation of the data in order to make it work
across all Debian derivatives, probably a new conversation needs to be
started between ftp-master and the dpkg maintainer.

I'm not involved in repro builds enough so your statement leads me to
wonder how you deal with ignoring the inevitable differences between
the buildinfo files, which would record the inevitable differences in
build environment between different builders. I'm guessing you just
ignore all differences in buildinfo files and would have to add to that
ignoring differences between build logs and other build metadata?

I thought I also had filed a bug report for Debian, but there's only one in LP,
describing build artifacts for both successful and failing builds. See
https://bugs.launchpad.net/launchpad/+bug/1845159

Any idea which package this should be filed for?

Matthias

I think this is going to require a bunch of different changes in
various places including dpkg-dev, debhelper, reprepro, dak, launchpad
and any other apt repos that have an incoming system. As long as the
files are mentioned in the .changes files dupload/dput won't change. So
probably there needs to be a discussion on debian-devel CCed to the
debian-dpkg list and to the ftp-masters and other folks.