- Package:
- apt-cacher-ng
- Source:
- apt-cacher-ng
- Description:
- caching proxy server for software repositories
- Submitter:
- Russell King
- Date:
- 2026-09-07 06:58:03 UTC
- Severity:
- normal
Dear Maintainer,
Running Discover on a system that uses my apt-cacher-ng instance
resulted in the following message:
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://security.debian.org/debian-security trixie-security InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature: Message has been manipulated Verifying signature: Message has been manipulated
E: http://security.debian.org/debian-security trixie-security InRelease is not (yet) available (Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature: Message has been manipulated Verifying signature: Message has been manipulated)
Requesting sqv on the apt-cacher-ng host to validate the file:
# sqv --keyring=/etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc --message /var/cache/apt-cacher-ng/secdeb/dists/trixie-security/InRelease --output /tmp/1
Missing key B0CAB9266E8C3929798B3EEEBDE6D2B9216EC7A8, which is needed to verify signature.
Verifying signature:
Message has been manipulated
Removing the offending InRelease file and requesting Discover to refresh
results in an InRelease file that passes SQV validation.
The bad file has a filesystem timestamp of August 6th, 14:03.
Diffing the bad and good Inrelease files show that the Date: is the same
for both files (Thu, 06 Aug 2026 09:33:27 UTC) but various SHA256 hashes
and the signature at the end of the file are different. For example:
- 9838bcde033a51b3d214a99cc75399f14df0415b79279248ae8866a6b2dd2281 1594893 main/binary-amd64/Packages
- 109d397f3295ffcd096cf334105cfbb3a00281ee033041efc5097807b1752fb1 231820 main/binary-amd64/Packages.xz
+ f66f17e77f4b35f6c04daf302dafa615d80517a247b7f8c3669ee02d25395d74 1596949 main/binary-amd64/Packages
+ d58ff282c3f5534c3fb906a6e447e892219780feb04d9cde10f5c39552556687 232072 main/binary-amd64/Packages.xz
This is not the first instance of this happening. Exactly the same issue
occurred with the trixie-backports InRelease file on July 11th. The
date field in that file was Fri, 10 Jul 2026 20:06:31 UTC.
As far as I can see, apt-cacher-ng does not record where it fetched its
files from, so if this is being caused by a bad debian mirror, there
seems to be no way to identify which mirror may be at fault. Is there
such a way?
The same problem has occurred again today: W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://deb.debian.org/debian trixie-backports InRelease: Sub-process /usr/ bin/sqv returned an error code (1), error message is: Verifying signature: Message has been manipulated Verifying signature: Message has been manipulated E: http://deb.debian.org/debian trixie-backports InRelease is not (yet) available (Sub- process /usr/bin/sqv returned an error code (1), error message is: Verifying signature: Message has been manipulated Verifying signature: Message has been manipulated) The symptoms are the same - the bad InRelease file has changed sha256 hashes, and a different signature to the good InRelease file, and the bad file does not pass the sqv validation. I have now setup a crontab to save all apt-cacher-ng InRelease files at midnight, 4am and 9am in an attempt to capture the past history of these cached files, so I can see whether the latter half of the file and signature is the same as the previous version, which would indicate that a Debian mirror is updating InRelease files in- place, leading to this kind of corruption.
Russell: David (penguin42) pinged me on Matrix about this and I've done
some digging.
I used your 2026-08-06 trixie-security bad/good and compared against
the copies on snapshot.debian.org [0] because snapshot no longer
collects debian-backports so unable to compare those.
The obvious issue is the SHA's seen in the 'bad' file are actually from
the previous day but something has altered the header fields - that
explains the failed signature.
$ diff -u InRelease.snapshot.20260805-215856.txt InRelease.bad
--- InRelease.snapshot.20260805-215856.txt 2026-08-11
17:39:44.590066986 +0000
+++ InRelease.bad 2026-08-11 17:38:51.632581770 +0000
@@ -6,8 +6,8 @@
Suite: stable-security
Version: 13
Codename: trixie-security
-Date: Wed, 05 Aug 2026 21:58:30 UTC
-Valid-Until: Wed, 12 Aug 2026 21:58:30 UTC
+Date: Thu, 06 Aug 2026 09:33:27 UTC
+Valid-Until: Thu, 13 Aug 2026 09:33:27 UTC
Acquire-By-Hash: yes
Architectures: amd64 arm64 armel armhf i386 ppc64el riscv64 s390x
Components: updates/main updates/contrib updates/non-free-firmware
updates/non-free
I do not see anything in apt-cacher-ng source that could be doing this
so I suspect this is being done by an upstream server. If
/etc/apt-cacher-ng/acng.conf is unchanged and has:
$ grep 'Remap-secdeb' /etc/apt-cacher-ng/acng.conf
Remap-secdeb: security.debian.org security.debian.org/debian-security
deb.debian.org/debian-security /debian-security
cdn-fastly.deb.debian.org/debian-security ;
deb.debian.org/debian-security security.debian.org
cdn-fastly.deb.debian.org/debian-security
Then the list after the semi-colon is the remote servers contacted, and
according to the code and docs the first is used until it returns a
vital error (but not 404s) and moves on to the next
I wonder if enabling (un-commenting) Debug will add sufficient data to
/var/log/apt-cacher-ng/*.{log,err} to identify the target server (by IP
address).
$ grep 'Debug' /etc/apt-cacher-ng/acng.conf
# it's a convenience alias for the Debug option, see below for details.
# Debug:3
It rather looks like a Fastly issue since all the IP addresses are
Fastly regardless of the name (security doesn't CNAME but has the
addresses as A/AAAA resource records):
$ resolvectl query -t CNAME deb.debian.org
deb.debian.org IN CNAME debian.map.fastlydns.net
$ resolvectl query deb.debian.org
deb.debian.org: 151.101.194.132 -- link: SUNNY
151.101.130.132 -- link: SUNNY
151.101.2.132 -- link: SUNNY
151.101.66.132 -- link: SUNNY
2a04:4e42::644 -- link: SUNNY
2a04:4e42:400::644 -- link: SUNNY
2a04:4e42:200::644 -- link: SUNNY
2a04:4e42:600::644 -- link: SUNNY
(debian.map.fastlydns.net)
$ resolvectl query security.debian.org
security.debian.org: 2a04:4e42::644 -- link: SUNNY
2a04:4e42:200::644 -- link: SUNNY
2a04:4e42:400::644 -- link: SUNNY
2a04:4e42:600::644 -- link: SUNNY
151.101.2.132 -- link: SUNNY
151.101.66.132 -- link: SUNNY
151.101.130.132 -- link: SUNNY
151.101.194.132 -- link: SUNNY
I'm not quite sure who to talk to about this or how to prove for sure
how the CDN is doing this.
[0]
https://snapshot.debian.org/archive/debian-security/20260805T215856Z/dists/trixie-security/
Hi, After the last problem, I set up a cronjob to save all the InRelease files that apt- cacher-ng has at midnight, 4am and 9am UK time (UTC+1), which appears to cover the time that Debian mirrors update, and also the time that my local machines' unattended-upgrade scripts run. Looking at just trixie-backports, I now have plenty of instances where the only change to the InRelease file is the Date: and Valid-Until: headers:
Please see the HTML version of my above message - it seems I've triggered a kmail bug as kmail appears to have chopped some of the plain text. I guess there's a use for HTML email after all! :D
Having seen apt-cacher-ng decide to download files, Debug 3 isn't going to help - it doesn't identify the IP address it's fetching from. What does seem to be suspicious though is that there is one "started" line and three "finished" lines. 1787313164|M|Download of secdeb/dists/trixie-security/InRelease started 1787313164|M|Download of secdeb/dists/trixie-security/InRelease finished 1787313164|M|Download of secdeb/dists/trixie-security/InRelease finished 1787313164|M|Download of secdeb/dists/trixie-security/InRelease finished and for trixie-backports, it's a similar story: 1787313164|M|Download of debrep/dists/trixie-backports/InRelease started 1787313164|M|Download of debrep/dists/trixie-backports/InRelease finished 1787313164|M|Download of debrep/dists/trixie-backports/InRelease finished 1787313164|M|Download of debrep/dists/trixie-backports/InRelease finished Nothing else in the log files seem to relate to these downloads. apt-cacher.dbg and apt-cacher.err are empty.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://security.debian.org/debian-security trixie-security InRelease: Sub- process /usr/bin/sqv returned an error code (1), error message is: Verifying signature: Message has been manipulated Verifying signature: Message has been manipulated E: http://security.debian.org/debian-security trixie-security InRelease is not (yet) available (Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature: Message has been manipulated Verifying signature: Message has been manipulated) In this instance, the problem appears to have occurred between 9am yesterday and midnight, and hasn't been resolved in the InRelease files captured at 4am nor 9am today.--- InRelease-20260826-09 2026-08-26 05:01:32.567389425 +0100 +++ InRelease-20260827-00 2026-08-26 20:42:56.654039495 +0100 @@ -6,8 +6,8 @@ Suite: stable-security Version: 13 Codename: trixie-security -Date: Wed, 26 Aug 2026 03:20:06 UTC -Valid-Until: Wed, 02 Sep 2026 03:20:06 UTC +Date: Wed, 26 Aug 2026 18:13:34 UTC +Valid-Until: Wed, 02 Sep 2026 18:13:34 UTC Acquire-By-Hash: yes Architectures: amd64 arm64 armel armhf i386 ppc64el riscv64 s390x Components: updates/main updates/contrib updates/non-free-firmware updates/ non-free
I suspect the three "finished" could be due to chunked transfer... and that makes me wonder if that is the cause of this. If the archive server isn't correctly handling the chunks (maybe a faulty front-end cache or similar) it *could* explain the symptoms.
As this is happening with secdeb in the standard configuration, this means apt- cacher-ng should be downloading from: deb.debian.org/debian-security security.debian.org cdn-fastly.deb.debian.org/debian-security which, according to host, resolve to: deb.debian.org is an alias for debian.map.fastlydns.net. debian.map.fastlydns.net has address 199.232.58.132 debian.map.fastlydns.net has IPv6 address 2a04:4e42:4b::644 (which is the same for cdn-fastly.deb.debian.org) security.debian.org has address 151.101.194.132 security.debian.org has address 151.101.2.132 security.debian.org has address 151.101.66.132 security.debian.org has address 151.101.130.132 security.debian.org has IPv6 address 2a04:4e42::644 security.debian.org has IPv6 address 2a04:4e42:200::644 security.debian.org has IPv6 address 2a04:4e42:400::644 security.debian.org has IPv6 address 2a04:4e42:600::644 security.debian.org mail is handled by 10 mitropoulos.debian.org. security.debian.org mail is handled by 10 mailly.debian.org. security.debian.org mail is handled by 10 muffat.debian.org.
Looking back at the report this is all using HTTP endpoints not HTTPS. That makes me wonder if there's a (transparent) HTTP proxy in the circuit. It used to be a common issue with ISPs operating them and altering response bodies. I'd recommend testing with HTTPS URLs. I use HTTP and have never experienced this and we don't have other similar reports so it is unlikely to be a general Debian infrastructure issue.
#http://ftp.uk.debian.org/debian/ https://debian.hands.com/debian/ which is the same machine. The former doesn't pass certificate validation when requesting https - broken cert? Secondly, for trixie-security, I changed acng.conf thusly: Remap-secdeb: security.debian.org security.debian.org/debian-security deb.debian.org/debian-security /debian-security cdn-fastly.deb.debian.org/debian- security ; https://deb.debian.org/debian-security https://security.debian.org https:// cdn-fastly.deb.debian.org/debian-security Then I restarted apt-cacher-ng via systemd restart ... and I can indeed confirm that the presently running apt-cacher-ng post-dates the modifications to these files. However, the log files give no clue (as I've already pointed out) what URL(s) are being used to fetch from, so I have no way to validate that apt-cacher-ng is fetching using https rather than http. Assuming that it is using https as it's been told, the problem still persists. Warning: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://security.debian.org/debian-security trixie-security InRelease: Sub- process /usr/bin/sqv returned an error code (1), error message is: Verifying signature: Message has been manipulated Verifying signature: Message has been manipulated
... It's a real pain that it doesn't report the protocol and without recompiling I don't see any way to do it directly. However, I've tested an alternatively locally, with manual browser access to the sites, and it logs connections so I wonder if you can run a background monitor service that simply does: ss --events --processes --tcp state established 'dport = :https and ( dst = debian.hands.com or dst = security.debian.org )' For my IPv6 only network when I access the two sites from Firefox it logs the expected addresses: Recv-Q Send-Q Local Address:Port Peer Address:Port Process 0 1 [2a0d:3344:379:31f0::ff]:41926 [2a04:4e42:600::644]:https 0 25 [2a0d:3344:379:31f0::ff]:44470 [2001:1b40:5600:ff80:f8ee::1]:https 0 1 [2a0d:3344:379:31f0::ff]:33396 [2a04:4e42:200::644]:https
So that does seem to show that https is being used:
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
1 0 [xxxx::xxxx]:46886
[2001:1b40:5600:ff80:f8ee::1]:https
25 1 [xxxx::xxxx]:45606
[2001:1b40:5600:ff80:f8ee::1]:https
25 1 [xxxx::xxxx]:60508
[2001:1b40:5600:ff80:f8ee::1]:https
That rules out a transparent proxy at the ISP (which is unlikely, I'm on a business
class connection, not a consumer class.)
I'm not sure there's much point in pursuing this bug any further. I've found that
other people have reported this same bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1121610[1]
and within that bug (which I think is for a slightly different problem) it seems that
the issue with corrupted files is well known:
"The data races are down to concurrent downloads. They result in a
corrupted file in acng's cache and the only fix is to delete the corrupt
file and try again. Release files update often enough that wait is also
an option, but a corrupted kernel deb, for example, will stay broken for
ages."
There's also:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1003865[2]
which looks like exactly the same problem as my bug, which was reported four
years ago and remains unfixed.
Other distributions also have the same bug, e.g.:
https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/1998865[3]
As pointed out in the Ubuntu bug, this is a security issue because the use of apt-
cacher-ng results in security fixes being delayed due to the frequeny corruption of
InRelease files.
As apt-cacher-ng currently stands, it is basically unsafe to have it in the package
delivery path for any public facing machine as it *will* delay the application of
Debian security fixes.
Lastly, if we look at the changelog and changelog.Debian in the current stable
package, the last entry there is July 2024, which suggests that this package is no
longer maintained, especially when there are outstanding "Important" class bugs -
that makes it sound like apt-cacher-ng is basically unmaintained.
Thus, even if we did get to the bottom of this, I have zero confidence that there will
be a fixed version of apt-cacher-ng released.
Thus, my conclusion is that apt-cacher-ng is a piece of buggy and dead software
that should not be used. I will be removing it, and seeking a replacement.
Tj - Thanks for having a look at this problem, but I feel that it has been a waste of
both of our time.
--------
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1121610
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1003865
[3] https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/1998865
On the contrary I think your proving the problem lies within the code is an important step. The codebase isn't too complex so I intend to: a) create a simple corruption reproducer in order to accurately trace the code b) fix it I suspect a reproducer, when restricted to a single thread, will not corrupt files. If that is correct the fix may be as simple as adding mutex at critical points.