Dear Maintainer,

Running Discover on a system that uses my apt-cacher-ng instance
resulted in the following message:

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://security.debian.org/debian-security trixie-security InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Message has been manipulated Verifying signature:            Message has been manipulated
E: http://security.debian.org/debian-security trixie-security InRelease is not (yet) available (Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Message has been manipulated Verifying signature:            Message has been manipulated)

Requesting sqv on the apt-cacher-ng host to validate the file:

# sqv --keyring=/etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc --message /var/cache/apt-cacher-ng/secdeb/dists/trixie-security/InRelease --output /tmp/1
Missing key B0CAB9266E8C3929798B3EEEBDE6D2B9216EC7A8, which is needed to verify signature.
Verifying signature:
           Message has been manipulated

Removing the offending InRelease file and requesting Discover to refresh
results in an InRelease file that passes SQV validation.

The bad file has a filesystem timestamp of August 6th, 14:03.

Diffing the bad and good Inrelease files show that the Date: is the same
for both files (Thu, 06 Aug 2026 09:33:27 UTC) but various SHA256 hashes
and the signature at the end of the file are different. For example:

- 9838bcde033a51b3d214a99cc75399f14df0415b79279248ae8866a6b2dd2281  1594893 main/binary-amd64/Packages
- 109d397f3295ffcd096cf334105cfbb3a00281ee033041efc5097807b1752fb1   231820 main/binary-amd64/Packages.xz
+ f66f17e77f4b35f6c04daf302dafa615d80517a247b7f8c3669ee02d25395d74  1596949 main/binary-amd64/Packages
+ d58ff282c3f5534c3fb906a6e447e892219780feb04d9cde10f5c39552556687   232072 main/binary-amd64/Packages.xz

This is not the first instance of this happening. Exactly the same issue
occurred with the trixie-backports InRelease file on July 11th. The
date field in that file was Fri, 10 Jul 2026 20:06:31 UTC.

As far as I can see, apt-cacher-ng does not record where it fetched its
files from, so if this is being caused by a bad debian mirror, there
seems to be no way to identify which mirror may be at fault. Is there
such a way?

The same problem has occurred again today:

W: An error occurred during the signature verification. The repository is not
updated and the previous index files will be used. OpenPGP signature verification
failed: http://deb.debian.org/debian trixie-backports InRelease: Sub-process /usr/
bin/sqv returned an error code (1), error message is: Verifying signature:
Message has been manipulated Verifying signature:            Message has been
manipulated
E: http://deb.debian.org/debian trixie-backports InRelease is not (yet) available (Sub-
process /usr/bin/sqv returned an error code (1), error message is: Verifying
signature:            Message has been manipulated Verifying signature:            Message
has been manipulated)

The symptoms are the same - the bad InRelease file has changed sha256 hashes,
and a different signature to the good InRelease file, and the bad
file does not pass the sqv validation.

I have now setup a crontab to save all apt-cacher-ng InRelease files at midnight,
4am and 9am in an attempt to capture the past history of these cached files, so I
can see whether the latter half of the file and signature is the same as the previous
version, which would indicate that a Debian mirror is updating InRelease files in-
place, leading to this kind of corruption.

Russell: David (penguin42) pinged me on Matrix about this and I've done
some digging.

I used your 2026-08-06  trixie-security  bad/good and compared against
the copies on snapshot.debian.org [0] because snapshot no longer
collects debian-backports so unable to compare those.

The obvious issue is the SHA's seen in the 'bad' file are actually from
the previous day but something has altered the header fields - that
explains the failed signature.

$ diff -u InRelease.snapshot.20260805-215856.txt  InRelease.bad
--- InRelease.snapshot.20260805-215856.txt      2026-08-11 
17:39:44.590066986 +0000
+++ InRelease.bad       2026-08-11 17:38:51.632581770 +0000
@@ -6,8 +6,8 @@
  Suite: stable-security
  Version: 13
  Codename: trixie-security
-Date: Wed, 05 Aug 2026 21:58:30 UTC
-Valid-Until: Wed, 12 Aug 2026 21:58:30 UTC
+Date: Thu, 06 Aug 2026 09:33:27 UTC
+Valid-Until: Thu, 13 Aug 2026 09:33:27 UTC
  Acquire-By-Hash: yes
  Architectures: amd64 arm64 armel armhf i386 ppc64el riscv64 s390x
  Components: updates/main updates/contrib updates/non-free-firmware
updates/non-free

I do not see anything in apt-cacher-ng source that could be doing this
so I suspect this is being done by an upstream server. If
/etc/apt-cacher-ng/acng.conf is unchanged and has:

$ grep 'Remap-secdeb' /etc/apt-cacher-ng/acng.conf
Remap-secdeb: security.debian.org security.debian.org/debian-security
deb.debian.org/debian-security /debian-security
cdn-fastly.deb.debian.org/debian-security ;
deb.debian.org/debian-security security.debian.org
cdn-fastly.deb.debian.org/debian-security

Then the list after the  semi-colon is the remote servers contacted, and
according to the code and docs the first is used until it returns a
vital error (but not 404s) and moves on to the next

I wonder if enabling (un-commenting) Debug will add sufficient data to
/var/log/apt-cacher-ng/*.{log,err} to identify the target server (by IP
address).

$ grep 'Debug' /etc/apt-cacher-ng/acng.conf
# it's a convenience alias for the Debug option, see below for details.
# Debug:3

It rather looks like a Fastly issue since all the IP addresses are
Fastly regardless of the name (security doesn't CNAME but has the
addresses as A/AAAA resource records):

$ resolvectl query -t CNAME deb.debian.org
deb.debian.org IN CNAME debian.map.fastlydns.net

$ resolvectl query deb.debian.org
deb.debian.org: 151.101.194.132                -- link: SUNNY
                 151.101.130.132                -- link: SUNNY
                 151.101.2.132                  -- link: SUNNY
                 151.101.66.132                 -- link: SUNNY
                 2a04:4e42::644                 -- link: SUNNY
                 2a04:4e42:400::644             -- link: SUNNY
                 2a04:4e42:200::644             -- link: SUNNY
                 2a04:4e42:600::644             -- link: SUNNY
                 (debian.map.fastlydns.net)

$ resolvectl query  security.debian.org
security.debian.org: 2a04:4e42::644            -- link: SUNNY
                      2a04:4e42:200::644        -- link: SUNNY
                      2a04:4e42:400::644        -- link: SUNNY
                      2a04:4e42:600::644        -- link: SUNNY
                      151.101.2.132             -- link: SUNNY
                      151.101.66.132            -- link: SUNNY
                      151.101.130.132           -- link: SUNNY
                      151.101.194.132           -- link: SUNNY

I'm not quite sure who to talk to about this or how to prove for sure
how the CDN is doing this.

[0]
https://snapshot.debian.org/archive/debian-security/20260805T215856Z/dists/trixie-security/

Hi,

After the last problem, I set up a cronjob to save all the InRelease files that apt-
cacher-ng has at midnight, 4am and 9am UK time (UTC+1), which appears to cover
the time that Debian mirrors update, and also the time that my local machines'
unattended-upgrade scripts run.

Looking at just trixie-backports, I now have plenty of instances where the only
change to the InRelease file is the Date: and Valid-Until: headers:

Please see the HTML version of my above
message - it seems I've triggered a kmail bug
as kmail appears to have chopped some of
the plain text. I guess there's a use for HTML
email after all! :D

Having seen apt-cacher-ng decide to download files, Debug 3 isn't going to help - it
doesn't identify the IP address it's fetching from. What does seem to be suspicious
though is that there is one "started" line and three "finished" lines.

1787313164|M|Download of secdeb/dists/trixie-security/InRelease started
1787313164|M|Download of secdeb/dists/trixie-security/InRelease finished
1787313164|M|Download of secdeb/dists/trixie-security/InRelease finished
1787313164|M|Download of secdeb/dists/trixie-security/InRelease finished

and for trixie-backports, it's a similar story:

1787313164|M|Download of debrep/dists/trixie-backports/InRelease started
1787313164|M|Download of debrep/dists/trixie-backports/InRelease finished
1787313164|M|Download of debrep/dists/trixie-backports/InRelease finished
1787313164|M|Download of debrep/dists/trixie-backports/InRelease finished

Nothing else in the log files seem to relate to these downloads. apt-cacher.dbg and
apt-cacher.err are empty.