Hi!
Long reply with many facets. I’d appreciate if all of it could
be read, even if it’s 231 lines (with quotes and spacing).
… uhm. The one that tests for Debian… i.e. that shows up on DDPO.
That would be the latter given the link from DDPO.
something to the users that has the best quality. To explain,
I’ll need to describe how and what the mksh source package builds
for normal native builds.
First, an mksh binary is built for /bin/mksh using the system glibc.
Then, mksh-static binaries are built for all other supported libcs
(dietlibc, musl, klibc) if they exist, otherwise one with glibc
(basically only on the Hurd; trust me you do NOT want to build this
statically against glibc if you can at all avoid it).
These are *all* installed, but the ones that test badly are chmod -x’d.
Having them still present is *extremely* useful in tracking down bugs
in mksh, GCC, binutils, the libc in question, qemu, and linux. (I’m not
kidding here.) These are installed into subdirectories.
The musl build also has to run small test executables in order to
find out whether the combination of musl version, architecture and
kernel, *and* toolchain and compiler, supports static-PIE yet and
enables it if found, to improve security.
Then, the “best” libc from that is selected according to an order
(currently, IIRC, I prefer musl over klibc, dietlibc last, as that
produces the most reliable one), and then an lksh binary (basically
mksh but with “long” as arithmetic type, a printf(1) builtin for
early boot, and auto-enabling POSIX mode when called as sh) is built
with that. (If none are found, lksh is built dynamically against
glibc.)
The mksh-static from the “best” libc is also linked as mksh-static
into the PATH.
And finally, an mksh-g is built against system glibc with ASan/UBSan.
In a cross or nocheck build, you get all of mksh, mksh-static, lksh
and mksh-g built against glibc, which for mksh-static is… not nice.
But also, see below: it is very common for packages to include long
full testsuite output in their binaries, and this is a very useful
property to have.
In both cases, the difference in binary packages does not impact
the “package API”: the presence of test results does not change the
way things behave, and the various mksh-static and lksh binaries
built with different libcs also behave the same (in normal circum‐
stances; the glibc-built static ones may need the very identical
nss solibs at runtime when 'cd ~user/' is used, which is why they
are less optimal for mksh-static); the build profile wiki page
mentions the exposed surface of packages as a driving concern.
I do agree that packages whose exposed “API” significantly differs
(e.g. leaving out an executable or library functions) if nocheck is
given are buggy, but the cases I showed don’t do that.
The not being allowed to have binary differences is a thing for
build profiles and defined in the build profiles spec. It is not
defined for the nocheck DEB_BUILD_OPTIONS, whose spec in Policy
merely says it’s to not run the package testsuite, and which is
commonly understood to facilitate cross builds as well. But it
has always permitted binary package differences, such as storing
the testsuite results into a binary package so the maintainer
can inspect the *full* logs as opposed to the summary shown in
the buildd log.
So it’s not so much a documentation of “nocheck DEB_BUILD_OPTIONS
is permitted to have differing binaries” but instead a “the nocheck
profile is explicitly specified to not allow that, but profiles and
DEB_BUILD_OPTIONS are in different spaces, and the DEB_BUILD_OPTIONS
does not specify that for these packages, and it is currently in use
and has always been and is in use for very good reasons”.
That… probably won’t do. If you build any of the affected packages
without and with that profile *and they do not support the profile*,
then the build without will include check-dependent files (mksh-klibc
in mksh’s case, test logs in others). I think we should figure out a
way to make this work for all packages.
Oh, hmm. This is… interesting.
I do not even disagree here: I do concur that a difference for the
nocheck DEB_BUILD_OPTIONS can be unintentional and that it’d be good
to check for that.
However we need to do this in a way that packages that do make use
of the being allowed to differ in binaries can continue to do so,
*especially* in the face of build reproducibility becoming a testing
migration requirement¹.
① I’d like to see if this allows me to bump prerequisite bugs, like
lack of reproducibility in cmake-generated .tar/.zip, to RC…
The probably-easiest way would be an allowlist on the reprotest side,
but that wouldn’t scale and it would age badly.
I could work with having to touch packages that need to keep the
distinction allowed somehow. There are several ways we could do that:
• a field in d/control, e.g. “XS-nocheck-option-differs: yes”, and
if this is present, the nocheck DEB_BUILD_OPTIONS is not set by
reprotest
⇒ we would need to figure out how to signal not-support for the
nocheck *profile*, too
• a new DEB_BUILD_OPTIONS with a long name that reprotest sets
that allows d/rules to avoid the non-differing ones:
ⓐ it could make the presence of the new option equivalen to
nocheck and thus omit the check-dependent things from the
first run as well; for this, it would need to be set in
*both* runs
ⓑ it would make the presence of the new option mean it should
ignore(!) the presence of nocheck in the second run (and thus
needs to be set in the second run only)
Out of these two, to be honest I’d prefer option b because I
believe that that gives us overall the better QA/test coverage,
even though this may affront sensibilities. My reasoning here
is that this new option in variant b signals to the package
that it is being reprotested, and that the presence of nocheck
does not have the standard Policy meaning for this build.
Option b also has the benefit that packages cannot just make
amends for the presence of that new flag for the first run and
do other things that result in the package being reproducible
only when that option is set but not in normal buildd builds.
So, reprotest could add 'nocheck reprotest-do-check-if-you-must'
to DEB_BUILD_OPTIONS for the second run only, and packages can
then detect reprotest-do-check-if-you-must and, if present, drop
nocheck from DEB_BUILD_OPTIONS *iff* they actually need it.
Hm. Could even vary between 'nocheck reprotest-do-check-if-you-must'
for the second run only and 'reprotest-prepare-for-nocheck' in
both runs, so that some reprotest runs test with nocheck and some
without, if this is deemed important. As long as affected packages
have a way to run both consistently while other packages vary.
I’d be fine with either of the two approaches, no preference
between 1 and 2b, or if you figure out something else, we can
discuss it.
⚠ BUT ⚠ could we please, at least while this discussion is still
underfood, revert the addition of nocheck until we find a solution,
so that the testing migration of packages is not yet affected?
As to build profiles… hm. I think we need to consult the people
driving that spec with regards to profile discoverability, and
not set the profile until then (the profile requires the option)
or in case of 1 (XS-nocheck-option-differs), only set the profile
if not present, or in case of 2b let packages ignore the profile
as well… that would even work, because if a package has <!nocheck>
B-Ds, it’s signalling support for the profile, so dpkg assuming
the profile is present would not matter.
Thanks for your consideration,
//mirabilos