Hi,

the binary package bash 5.2.15-2+b3 was uploaded to the archive twice. Once to
bookworm and once to sid but with differing content. Here is a diff of their
d/changelog:

@@ -1,6 +1,6 @@
-bash (5.2.15-2+b3) bookworm; urgency=low, binary-only=yes
+bash (5.2.15-2+b3) sid; urgency=low, binary-only=yes

   * Binary-only non-maintainer upload for arm64; no source changes.
   * Rebuild for outdated Built-Using (glibc/2.36-9)

- -- arm Build Daemon (arm-ubc-03) <buildd_arm64-arm-ubc-03@buildd.debian.org>  Fri, 29 Mar 2024 13:22:36 +0000
+ -- arm Build Daemon (arm-ubc-02) <buildd_arm64-arm-ubc-02@buildd.debian.org>  Thu, 13 Jul 2023 09:12:53 +0000

This is not only confusing for apt (it will not be able to figure out which of
the two is installed because dpkg does not keep track of package hashes) but it
is also problematic for reproducible builds because the buildinfo file records
packages by their name/architecture/version tuple and relies on those to be
unique throughout the history of Debian.

This example with bash is especially problematic since bash is Essential:yes so
there will now be a large portion of buildinfo files where it is not possible
to figure out with which of the two differing bash packages the sources were
compiled.

snapshot.d.o also shows the issue and could probably be used to get an idea how
many packages are affected:

http://snapshot.debian.org/package/bash/5.2.15-2/#bash_5.2.15-2:2b:b3

Maybe this issue is blocked by #620356?

Thanks!

cheers, josch

Hi,

very "nice" find, josch!

I've checked all .buildinfo files to see how big the impact of this problem
is:

holger@ionos7-amd64:/var/ … tent/reproducible/debian/ftp-master.debian.org/buildinfo $ for i in amd64 arm64 armel armhf i386 mips64el ppc64el riscv64 s390x ; do echo -n "$i: " ; grep -c "$i.buildinfo: bash" ~/dak-bug-1072205 ; done
holger@ionos7-amd64:/var/ … tent/reproducible/debian/ftp-master.debian.org/buildinfo $ for i in amd64 arm64 armel armhf i386 mips64el ppc64el riscv64 s390x ; do echo -n "$i: " ; grep -c "$i.buildinfo: bash" ~/dak-bug-1072205 ; done
amd64: 1322
arm64: 22
armel: 0
armhf: 0
i386: 10
mips64el: 0
ppc64el: 1
riscv64: 92
s390x: 0
holger@ionos7-amd64:/var/ … tent/reproducible/debian/ftp-master.debian.org/buildinfo 1 $

So, not too bad. And luckily all of these can be fixed with binNMUs.

Hi,
to get some hard data about this problem. My idea was the following: parse all
Packages files for all suites, all architectures and all components for all
timestamps stored on snapshot.d.o and find packages with the same
name/arch/version tuple that have a different checksum. To this end, I slightly
(less than 1000 lines of diff) patched the tooling at
https://salsa.debian.org/metasnap-team/metasnap.git with the patch that I
attached to this mail on top of 1dadf2575160caf9467c4e21aa6c0a31ac10ffc2.

After running that script for 3 months and downloading 189 GB of data in 3.5
Million requests (about 2 seconds for every request), we had a database
(actually a git repository) of 48 GB that we can use to find duplicates. It
took another 2 months to go through that data. I attached a graph which shows
the number of duplicate name/arch/version triplets per timestamp. Please note
the logarithmic y-axis. The total number of duplicates from 2005 until 2024 is
334335.

Problem solved? Not so fast. Processing all Packages files will *not* find the
original problem with bash. Why? Because according to the Packages files from
snapshot.debian.org only one version of bash:arm64=5.2.15-2+b3 exists, namely:

MD5sum: 01ee4cfa3df78e7ff0dc156ff19e2c88
SHA1:   1a0b12419b69a983bf22ac1d3d9f8bec725487b1
SHA256: 828ce0b4445921fff5b6394e74cce8296f3038d559845a3e82435b55ca6fcaa8

The other version never ended up in a Packages file even though it was found in
the  /pool/main/b/bash directory in the snapshot of 2023-07-13 21:11:09 nearly
one year before the other version popped up.

How can a package be in the pool directory but not in a Packages file? No idea
but it shows that my method from above does not find a certain class of
problems. We could find those by creating a fitting query against the
snapshot.d.o database. Apparently lw07 is DD accessible and has a
snapshot-guest service. So this is on my TODO list and Nicolas Dandrimont
already offered to help with constructing an appropriate SQL query during
MiniDebConf Hamburg this year.

Lastly there is the problem of packages in incoming. Those packages will be
used to build other packages that end up in the archive but they might never
end up in the archive themselves. Thus, we might never know whether one of
these packages violated the idea that the packagename/architecture/version
triplet uniquely identifies a Debian binary package in the archive...

Thanks!

cheers, josch

Hi,

regarding your analysis: I think you could just scan the .changes files
as they list all *.deb files uploaded. Though very old changes only
have MD5 hashes.

They can be found in
file://mirror.ftp-master.debian.org/srv/ftp-master.debian.org/queue/done

Regarding your observation regarding bash not showing up in any
Packages index: that can happen for (at least) two reasons. The
snapshot service does not retrieve all Packages files. Or the package
could have been superseded by a newer version before it was ever
published in a dinstall run.

Regards,
Ansgar

On Wed, 2025-04-02 at 15:26 +0200, Johannes Schauer Marin Rodrigues wrote:

Hi Ansgar,

thank you for chiming in!

Quoting Ansgar 🙀 (2025-04-02 21:41:39)

are those the changes files uploading *.debs produced by the buildds?

Does that include *.changes files from package builds which only ever affected
incoming because there was a new build of the same package before the next
dinstall?

Looking at those .changes files, it indeed seems that the binNMUs were made by
the buildds, so those are not maintainer changes files.

You mean the snapshot service was buggy and forgot to retrieve an arm64
Packages file? Is that likely? Lets have a look:

http://snapshot.debian.org/package/bash/5.2.15-2/#bash_5.2.15-2:2b:b3

We have two different copies of bash:arm64=5.2.15-2+b3:

20230713T211109Z until 20230813T092534Z
01ee4cfa3df78e7ff0dc156ff19e2c88
597e85b2fdc7fe9da3f3215030e7c3dead3f464c
828ce0b4445921fff5b6394e74cce8296f3038d559845a3e82435b55ca6fcaa8
http://snapshot.debian.org/archive/debian/20230713T211109Z/pool/main/b/bash/bash/bash_5.2.15-2%2Bb3_arm64.deb

20240331T210805Z until 20240402T025937Z
2f8db1d53e5b2e6292b3c408d55ae5de
1a0b12419b69a983bf22ac1d3d9f8bec725487b1
58000b621cceafd717ed0bf4cb428ba1220c0e1993620bde8ad8a3ccc765780b
http://snapshot.debian.org/archive/debian/20240331T210805Z/pool/main/b/bash/bash_5.2.15-2%2Bb3_arm64.deb

I had a look at the changes files stored in
/srv/ftp-master.debian.org/queue/done/2024/03.tar.xz and indeed there I found
03/31/bash_5.2.15-2+b3_arm64-buildd.changes which is the binNMU to Bookworm.
The same tarball contains another changes file for bash from eight hours later
called 03/31/bash_5.2.15-2+b7_arm64-buildd.changes. Maybe it is indeed possible
that snapshot.d.o skipped the one Packages for Bookworm file that should've
contained the former version?

But hashes of packages that were never published by a dinstall should be part
of the changes file on coccia, right? That would be really nice! A year of
changes files seems to be just around 2.6 GB which is really tiny compared to
the data I was processing before.

Is this really the silver bullet that solves all my problems with finding these
hash collisions?

Can I just scp those tar.xz files from coccia to my local box and process them
to my heart's content? Is there a catch?

Thanks!

cheers, josch

Hi,

All regular uploads to the archive should come with a .changes files
and these should be all, that is, for both maintainer uploads and
buildd uploads. (Note that historic buildd uploads were also signed by
a DD key.)

It does not cover uploads to the security archive, but packages synced
from the security archive to the main archive should be recorded there.

It also does not include rejected packages (e.g., for policy queues
before *-proposed-updates, new or even before). But for your use case
these are probably not relevant.

There is a mechanism in DAK to import packages without .changes files,
but that shouldn't be used much. The only regular use case is adding
source packages for Built-Using to the security archive.

There is no technical mechanism to ensure files end up in snapshot.d.o.
If the service does not import them on time, they will not end up in
snapshot.d.o; the archive does currently not retain files until the
snapshot service picked them up.

There have been times in the past where snapshot.d.o was not always
reliably importing files, see for example [1].

  [1]: https://lists.debian.org/msgid-search/169097599149.3513878.11315997455981479167@localhost

I've not checked if this was the case for the particular instance you
mentioned.

Only the .changes files might be even less data: the last years also
include .buildinfo files.

I think that is fine.

The equivalent data from the security archive is not on a DD-accessible
mirror, but if you find it useful, we could probably make them
available to you (at least provide a one-time dump). As this is only
about published packages, there is no need for confidentiality.

Ansgar

Hi,

would it help to use +b12.1, +b12.2, ... for binNMUs to bookworm, and
similarly for future releases?

    Simon