#1147469 reproducible-builds: must not set 'nocheck' on only one build

#1147469#5
Date:
2026-09-12 04:39:18 UTC
From:
To:
I’ve recently found out that reproducible-builds nowadays fails
packages which produce different output when 'nocheck' is set vs.
unset.

This is wrong.

If (if!) a 'nocheck' build *profile* is supported by the package,
then, if that build *profile* is set, the binary packages may not
differ.

But the 'nocheck' build option *is* allowed to have binary
differences, and this is something important for various corner
cases.

So, please add it to DEB_BUILD_OPTIONS on either no or all builds,
and investigate into setting the 'nocheck' build *profile* on only
one build *iff* the package supports it. I’m not exactly sure how
to actually test for that, but if it only gets added to the variable
DEB_BUILD_PROFILES but not DEB_BUILD_OPTIONS, then it can just be
ignored by packages not supporting the 'nocheck' *profile*.

#1147469#10
Date:
2026-09-13 13:51:43 UTC
From:
To:
hi Thorsten,

just to be clear, are you talking about https://reproduce.debian.net/
or https://tests.reproducible-builds.org/debian/ ?

(the diff is explained on top of https://reproduce.debian.net/)

i'm not sure you are correct here (you might well be!), so i'm
wondering: where is this documented?

ack.

#1147469#13
Date:
2026-09-13 14:23:02 UTC
From:
To:
Re: Thorsten Glaser

This sounds like you have a whole lot of packages with that problem.
Care to share concrete cases?

I believe testing for that is very valuable and it should stay.

Unless I missed something, packages do not announce profiles they
support. You could only look at debian/rules and guess which profiles
make any difference.

Even if that is true, I think t.rb.o should still test for the
problem. In most cases, the difference should be unintentional and
then people can work on fixing it.

So the fix is to just set DEB_BUILD_PROFILES=nocheck along and then
the complaint is gone? :)

Christoph

#1147469#18
Date:
2026-09-14 01:23:26 UTC
From:
To:
Hi!

Long reply with many facets. I’d appreciate if all of it could
be read, even if it’s 231 lines (with quotes and spacing).

… uhm. The one that tests for Debian… i.e. that shows up on DDPO.
That would be the latter given the link from DDPO.
something to the users that has the best quality. To explain,
I’ll need to describe how and what the mksh source package builds
for normal native builds.


First, an mksh binary is built for /bin/mksh using the system glibc.

Then, mksh-static binaries are built for all other supported libcs
(dietlibc, musl, klibc) if they exist, otherwise one with glibc
(basically only on the Hurd; trust me you do NOT want to build this
statically against glibc if you can at all avoid it).

These are *all* installed, but the ones that test badly are chmod -x’d.
Having them still present is *extremely* useful in tracking down bugs
in mksh, GCC, binutils, the libc in question, qemu, and linux. (I’m not
kidding here.) These are installed into subdirectories.

The musl build also has to run small test executables in order to
find out whether the combination of musl version, architecture and
kernel, *and* toolchain and compiler, supports static-PIE yet and
enables it if found, to improve security.

Then, the “best” libc from that is selected according to an order
(currently, IIRC, I prefer musl over klibc, dietlibc last, as that
produces the most reliable one), and then an lksh binary (basically
mksh but with “long” as arithmetic type, a printf(1) builtin for
early boot, and auto-enabling POSIX mode when called as sh) is built
with that. (If none are found, lksh is built dynamically against
glibc.)

The mksh-static from the “best” libc is also linked as mksh-static
into the PATH.

And finally, an mksh-g is built against system glibc with ASan/UBSan.


In a cross or nocheck build, you get all of mksh, mksh-static, lksh
and mksh-g built against glibc, which for mksh-static is… not nice.


But also, see below: it is very common for packages to include long
full testsuite output in their binaries, and this is a very useful
property to have.

In both cases, the difference in binary packages does not impact
the “package API”: the presence of test results does not change the
way things behave, and the various mksh-static and lksh binaries
built with different libcs also behave the same (in normal circum‐
stances; the glibc-built static ones may need the very identical
nss solibs at runtime when 'cd ~user/' is used, which is why they
are less optimal for mksh-static); the build profile wiki page
mentions the exposed surface of packages as a driving concern.

I do agree that packages whose exposed “API” significantly differs
(e.g. leaving out an executable or library functions) if nocheck is
given are buggy, but the cases I showed don’t do that.

The not being allowed to have binary differences is a thing for
build profiles and defined in the build profiles spec. It is not
defined for the nocheck DEB_BUILD_OPTIONS, whose spec in Policy
merely says it’s to not run the package testsuite, and which is
commonly understood to facilitate cross builds as well. But it
has always permitted binary package differences, such as storing
the testsuite results into a binary package so the maintainer
can inspect the *full* logs as opposed to the summary shown in
the buildd log.

So it’s not so much a documentation of “nocheck DEB_BUILD_OPTIONS
is permitted to have differing binaries” but instead a “the nocheck
profile is explicitly specified to not allow that, but profiles and
DEB_BUILD_OPTIONS are in different spaces, and the DEB_BUILD_OPTIONS
does not specify that for these packages, and it is currently in use
and has always been and is in use for very good reasons”.


That… probably won’t do. If you build any of the affected packages
without and with that profile *and they do not support the profile*,
then the build without will include check-dependent files (mksh-klibc
in mksh’s case, test logs in others). I think we should figure out a
way to make this work for all packages.

Oh, hmm. This is… interesting.

I do not even disagree here: I do concur that a difference for the
nocheck DEB_BUILD_OPTIONS can be unintentional and that it’d be good
to check for that.

However we need to do this in a way that packages that do make use
of the being allowed to differ in binaries can continue to do so,
*especially* in the face of build reproducibility becoming a testing
migration requirement¹.

① I’d like to see if this allows me to bump prerequisite bugs, like
  lack of reproducibility in cmake-generated .tar/.zip, to RC…

The probably-easiest way would be an allowlist on the reprotest side,
but that wouldn’t scale and it would age badly.

I could work with having to touch packages that need to keep the
distinction allowed somehow. There are several ways we could do that:

• a field in d/control, e.g. “XS-nocheck-option-differs: yes”, and
  if this is present, the nocheck DEB_BUILD_OPTIONS is not set by
  reprotest

  ⇒ we would need to figure out how to signal not-support for the
    nocheck *profile*, too

• a new DEB_BUILD_OPTIONS with a long name that reprotest sets
  that allows d/rules to avoid the non-differing ones:

  ⓐ it could make the presence of the new option equivalen to
    nocheck and thus omit the check-dependent things from the
    first run as well; for this, it would need to be set in
    *both* runs

  ⓑ it would make the presence of the new option mean it should
    ignore(!) the presence of nocheck in the second run (and thus
    needs to be set in the second run only)

  Out of these two, to be honest I’d prefer option b because I
  believe that that gives us overall the better QA/test coverage,
  even though this may affront sensibilities. My reasoning here
  is that this new option in variant b signals to the package
  that it is being reprotested, and that the presence of nocheck
  does not have the standard Policy meaning for this build.

  Option b also has the benefit that packages cannot just make
  amends for the presence of that new flag for the first run and
  do other things that result in the package being reproducible
  only when that option is set but not in normal buildd builds.

  So, reprotest could add 'nocheck reprotest-do-check-if-you-must'
  to DEB_BUILD_OPTIONS for the second run only, and packages can
  then detect reprotest-do-check-if-you-must and, if present, drop
  nocheck from DEB_BUILD_OPTIONS *iff* they actually need it.

  Hm. Could even vary between 'nocheck reprotest-do-check-if-you-must'
  for the second run only and 'reprotest-prepare-for-nocheck' in
  both runs, so that some reprotest runs test with nocheck and some
  without, if this is deemed important. As long as affected packages
  have a way to run both consistently while other packages vary.

I’d be fine with either of the two approaches, no preference
between 1 and 2b, or if you figure out something else, we can
discuss it.

⚠ BUT ⚠ could we please, at least while this discussion is still
underfood, revert the addition of nocheck until we find a solution,
so that the testing migration of packages is not yet affected?

As to build profiles… hm. I think we need to consult the people
driving that spec with regards to profile discoverability, and
not set the profile until then (the profile requires the option)
or in case of 1 (XS-nocheck-option-differs), only set the profile
if not present, or in case of 2b let packages ignore the profile
as well… that would even work, because if a package has <!nocheck>
B-Ds, it’s signalling support for the profile, so dpkg assuming
the profile is present would not matter.

Thanks for your consideration,
//mirabilos

#1147469#23
Date:
2026-09-14 08:02:21 UTC
From:
To:
both are relevant for Debian, please read the text on https://reproduce.debian.net/
quoted here for your convinience:

reproduce.debian.net
Attempts to bit-for-bit identically rebuild each Debian binary package found in the distribution archive, using the .buildinfo file produced when the buildd originally built the package.

For each distributed package, rebuilderd calls debrebuild that calls debootsnap, mmdebstrap and finally sbuild to build that package within a user namespace.

    buildinfos.debian.net provides .buildinfo files for each distributed binary package. (see #763822, #929397)
    debootsnap uses snapshot.debian.org and metasnap.debian.net to retrieve the exact same version of each build dependency package (which was used for the "original build"). (see #1096129)
    The goal is to replicate the same build process that is used by Debian during package publication -- not to seek out additional sources of variance.
    Variance testing, used to find factors that can prevent packages from rebuilding reproducibly, will continue at https://tests.reproducible-builds.org/debian/reproducible.html.
    If you are wondering why we are doing this at all, then hopefully the Reproducible Builds website will explain why this is useful.
    reproduce.debian.net is currently testing all release architectures except s390x.
    Better navigation to experimental, unstable, forky, trixie, trixie-security, trixie-updates, trixie-proposed-updates, trixie-backports would be nice.


will reply to the rest of your mail hopefully later this afternoon,
got a busy morning now...

#1147469#28
Date:
2026-09-14 08:29:07 UTC
From:
To:
Hmm, the one that shows up in DDPO then, as that’s where I saw this.

No worries, have a stress-free morning.

bye,
//mirabilos