Dear Maintainer,

Running Discover on a system that uses my apt-cacher-ng instance
resulted in the following message:

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://security.debian.org/debian-security trixie-security InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Message has been manipulated Verifying signature:            Message has been manipulated
E: http://security.debian.org/debian-security trixie-security InRelease is not (yet) available (Sub-process /usr/bin/sqv returned an error code (1), error message is: Verifying signature:            Message has been manipulated Verifying signature:            Message has been manipulated)

Requesting sqv on the apt-cacher-ng host to validate the file:

# sqv --keyring=/etc/apt/trusted.gpg.d/debian-archive-trixie-security-automatic.asc --message /var/cache/apt-cacher-ng/secdeb/dists/trixie-security/InRelease --output /tmp/1
Missing key B0CAB9266E8C3929798B3EEEBDE6D2B9216EC7A8, which is needed to verify signature.
Verifying signature:
           Message has been manipulated

Removing the offending InRelease file and requesting Discover to refresh
results in an InRelease file that passes SQV validation.

The bad file has a filesystem timestamp of August 6th, 14:03.

Diffing the bad and good Inrelease files show that the Date: is the same
for both files (Thu, 06 Aug 2026 09:33:27 UTC) but various SHA256 hashes
and the signature at the end of the file are different. For example:

- 9838bcde033a51b3d214a99cc75399f14df0415b79279248ae8866a6b2dd2281  1594893 main/binary-amd64/Packages
- 109d397f3295ffcd096cf334105cfbb3a00281ee033041efc5097807b1752fb1   231820 main/binary-amd64/Packages.xz
+ f66f17e77f4b35f6c04daf302dafa615d80517a247b7f8c3669ee02d25395d74  1596949 main/binary-amd64/Packages
+ d58ff282c3f5534c3fb906a6e447e892219780feb04d9cde10f5c39552556687   232072 main/binary-amd64/Packages.xz

This is not the first instance of this happening. Exactly the same issue
occurred with the trixie-backports InRelease file on July 11th. The
date field in that file was Fri, 10 Jul 2026 20:06:31 UTC.

As far as I can see, apt-cacher-ng does not record where it fetched its
files from, so if this is being caused by a bad debian mirror, there
seems to be no way to identify which mirror may be at fault. Is there
such a way?

The same problem has occurred again today:

W: An error occurred during the signature verification. The repository is not
updated and the previous index files will be used. OpenPGP signature verification
failed: http://deb.debian.org/debian trixie-backports InRelease: Sub-process /usr/
bin/sqv returned an error code (1), error message is: Verifying signature:
Message has been manipulated Verifying signature:            Message has been
manipulated
E: http://deb.debian.org/debian trixie-backports InRelease is not (yet) available (Sub-
process /usr/bin/sqv returned an error code (1), error message is: Verifying
signature:            Message has been manipulated Verifying signature:            Message
has been manipulated)

The symptoms are the same - the bad InRelease file has changed sha256 hashes,
and a different signature to the good InRelease file, and the bad
file does not pass the sqv validation.

I have now setup a crontab to save all apt-cacher-ng InRelease files at midnight,
4am and 9am in an attempt to capture the past history of these cached files, so I
can see whether the latter half of the file and signature is the same as the previous
version, which would indicate that a Debian mirror is updating InRelease files in-
place, leading to this kind of corruption.

Russell: David (penguin42) pinged me on Matrix about this and I've done
some digging.

I used your 2026-08-06  trixie-security  bad/good and compared against
the copies on snapshot.debian.org [0] because snapshot no longer
collects debian-backports so unable to compare those.

The obvious issue is the SHA's seen in the 'bad' file are actually from
the previous day but something has altered the header fields - that
explains the failed signature.

$ diff -u InRelease.snapshot.20260805-215856.txt  InRelease.bad
--- InRelease.snapshot.20260805-215856.txt      2026-08-11 
17:39:44.590066986 +0000
+++ InRelease.bad       2026-08-11 17:38:51.632581770 +0000
@@ -6,8 +6,8 @@
  Suite: stable-security
  Version: 13
  Codename: trixie-security
-Date: Wed, 05 Aug 2026 21:58:30 UTC
-Valid-Until: Wed, 12 Aug 2026 21:58:30 UTC
+Date: Thu, 06 Aug 2026 09:33:27 UTC
+Valid-Until: Thu, 13 Aug 2026 09:33:27 UTC
  Acquire-By-Hash: yes
  Architectures: amd64 arm64 armel armhf i386 ppc64el riscv64 s390x
  Components: updates/main updates/contrib updates/non-free-firmware
updates/non-free

I do not see anything in apt-cacher-ng source that could be doing this
so I suspect this is being done by an upstream server. If
/etc/apt-cacher-ng/acng.conf is unchanged and has:

$ grep 'Remap-secdeb' /etc/apt-cacher-ng/acng.conf
Remap-secdeb: security.debian.org security.debian.org/debian-security
deb.debian.org/debian-security /debian-security
cdn-fastly.deb.debian.org/debian-security ;
deb.debian.org/debian-security security.debian.org
cdn-fastly.deb.debian.org/debian-security

Then the list after the  semi-colon is the remote servers contacted, and
according to the code and docs the first is used until it returns a
vital error (but not 404s) and moves on to the next

I wonder if enabling (un-commenting) Debug will add sufficient data to
/var/log/apt-cacher-ng/*.{log,err} to identify the target server (by IP
address).

$ grep 'Debug' /etc/apt-cacher-ng/acng.conf
# it's a convenience alias for the Debug option, see below for details.
# Debug:3

It rather looks like a Fastly issue since all the IP addresses are
Fastly regardless of the name (security doesn't CNAME but has the
addresses as A/AAAA resource records):

$ resolvectl query -t CNAME deb.debian.org
deb.debian.org IN CNAME debian.map.fastlydns.net

$ resolvectl query deb.debian.org
deb.debian.org: 151.101.194.132                -- link: SUNNY
                 151.101.130.132                -- link: SUNNY
                 151.101.2.132                  -- link: SUNNY
                 151.101.66.132                 -- link: SUNNY
                 2a04:4e42::644                 -- link: SUNNY
                 2a04:4e42:400::644             -- link: SUNNY
                 2a04:4e42:200::644             -- link: SUNNY
                 2a04:4e42:600::644             -- link: SUNNY
                 (debian.map.fastlydns.net)

$ resolvectl query  security.debian.org
security.debian.org: 2a04:4e42::644            -- link: SUNNY
                      2a04:4e42:200::644        -- link: SUNNY
                      2a04:4e42:400::644        -- link: SUNNY
                      2a04:4e42:600::644        -- link: SUNNY
                      151.101.2.132             -- link: SUNNY
                      151.101.66.132            -- link: SUNNY
                      151.101.130.132           -- link: SUNNY
                      151.101.194.132           -- link: SUNNY

I'm not quite sure who to talk to about this or how to prove for sure
how the CDN is doing this.

[0]
https://snapshot.debian.org/archive/debian-security/20260805T215856Z/dists/trixie-security/

Hi,

After the last problem, I set up a cronjob to save all the InRelease files that apt-
cacher-ng has at midnight, 4am and 9am UK time (UTC+1), which appears to cover
the time that Debian mirrors update, and also the time that my local machines'
unattended-upgrade scripts run.

Looking at just trixie-backports, I now have plenty of instances where the only
change to the InRelease file is the Date: and Valid-Until: headers:

Please see the HTML version of my above
message - it seems I've triggered a kmail bug
as kmail appears to have chopped some of
the plain text. I guess there's a use for HTML
email after all! :D

Having seen apt-cacher-ng decide to download files, Debug 3 isn't going to help - it
doesn't identify the IP address it's fetching from. What does seem to be suspicious
though is that there is one "started" line and three "finished" lines.

1787313164|M|Download of secdeb/dists/trixie-security/InRelease started
1787313164|M|Download of secdeb/dists/trixie-security/InRelease finished
1787313164|M|Download of secdeb/dists/trixie-security/InRelease finished
1787313164|M|Download of secdeb/dists/trixie-security/InRelease finished

and for trixie-backports, it's a similar story:

1787313164|M|Download of debrep/dists/trixie-backports/InRelease started
1787313164|M|Download of debrep/dists/trixie-backports/InRelease finished
1787313164|M|Download of debrep/dists/trixie-backports/InRelease finished
1787313164|M|Download of debrep/dists/trixie-backports/InRelease finished

Nothing else in the log files seem to relate to these downloads. apt-cacher.dbg and
apt-cacher.err are empty.

W: An error occurred during the signature verification. The repository is not
updated and the previous index files will be used. OpenPGP signature verification
failed: http://security.debian.org/debian-security trixie-security InRelease: Sub-
process /usr/bin/sqv returned an error code (1), error message is: Verifying
signature:            Message has been manipulated Verifying signature:            Message
has been manipulated
E: http://security.debian.org/debian-security trixie-security InRelease is not (yet)
available (Sub-process /usr/bin/sqv returned an error code (1), error message is:
Verifying signature:            Message has been manipulated Verifying signature:
Message has been manipulated)

In this instance, the problem appears to have occurred between 9am yesterday and
midnight, and hasn't been resolved in the InRelease files captured at 4am nor 9am
today.

--- InRelease-20260826-09       2026-08-26 05:01:32.567389425 +0100
+++ InRelease-20260827-00       2026-08-26 20:42:56.654039495 +0100
@@ -6,8 +6,8 @@
 Suite: stable-security
 Version: 13
 Codename: trixie-security
-Date: Wed, 26 Aug 2026 03:20:06 UTC
-Valid-Until: Wed, 02 Sep 2026 03:20:06 UTC
+Date: Wed, 26 Aug 2026 18:13:34 UTC
+Valid-Until: Wed, 02 Sep 2026 18:13:34 UTC
 Acquire-By-Hash: yes
 Architectures: amd64 arm64 armel armhf i386 ppc64el riscv64 s390x
 Components: updates/main updates/contrib updates/non-free-firmware updates/
non-free

I suspect the three "finished" could be due to chunked transfer... and
that makes me wonder if that is the cause of this. If the archive server
isn't correctly handling the chunks (maybe a faulty front-end cache or
similar) it *could* explain the symptoms.

As this is happening with secdeb in the standard configuration, this means apt-
cacher-ng should be downloading from:

deb.debian.org/debian-security
security.debian.org
cdn-fastly.deb.debian.org/debian-security

which, according to host, resolve to:

deb.debian.org is an alias for debian.map.fastlydns.net.
debian.map.fastlydns.net has address 199.232.58.132
debian.map.fastlydns.net has IPv6 address 2a04:4e42:4b::644

(which is the same for cdn-fastly.deb.debian.org)

security.debian.org has address 151.101.194.132
security.debian.org has address 151.101.2.132
security.debian.org has address 151.101.66.132
security.debian.org has address 151.101.130.132
security.debian.org has IPv6 address 2a04:4e42::644
security.debian.org has IPv6 address 2a04:4e42:200::644
security.debian.org has IPv6 address 2a04:4e42:400::644
security.debian.org has IPv6 address 2a04:4e42:600::644
security.debian.org mail is handled by 10 mitropoulos.debian.org.
security.debian.org mail is handled by 10 mailly.debian.org.
security.debian.org mail is handled by 10 muffat.debian.org.

Looking back at the report this is all using HTTP endpoints not HTTPS.
That makes me wonder if there's a (transparent) HTTP proxy in the
circuit. It used to be a common issue with ISPs operating them and
altering response bodies.

I'd recommend testing with HTTPS URLs.

I use HTTP and have never experienced this and we don't have other
similar reports so it is unlikely to be a general Debian infrastructure
issue.

#http://ftp.uk.debian.org/debian/
https://debian.hands.com/debian/

which is the same machine. The former doesn't pass certificate validation when
requesting https - broken cert?

Secondly, for trixie-security, I changed acng.conf thusly:

Remap-secdeb: security.debian.org security.debian.org/debian-security
deb.debian.org/debian-security /debian-security cdn-fastly.deb.debian.org/debian-
security ; https://deb.debian.org/debian-security https://security.debian.org https://
cdn-fastly.deb.debian.org/debian-security


Then I restarted apt-cacher-ng via systemd restart ... and I can indeed confirm that
the presently running apt-cacher-ng post-dates the modifications to these files.

However, the log files give no clue (as I've already pointed out) what URL(s) are
being used to fetch from, so I have no way to validate that apt-cacher-ng is fetching
using https rather than http.

Assuming that it is using https as it's been told, the problem still persists.

Warning: An error occurred during the signature verification. The repository is not
updated and the previous index files will be used. OpenPGP signature verification
failed: http://security.debian.org/debian-security trixie-security InRelease: Sub-
process /usr/bin/sqv returned an error code (1), error message is: Verifying
signature:            Message has been manipulated Verifying signature:            Message
has been manipulated

...
It's a real pain that it doesn't report the protocol and without
recompiling I don't see any way to do it directly.

However, I've tested an alternatively locally, with manual browser
access to the sites, and it logs connections so I wonder if you can run
a background monitor service that simply does:

ss --events --processes --tcp state established 'dport = :https and (
dst = debian.hands.com or dst = security.debian.org )'

For my IPv6 only network when I access the two sites from Firefox it
logs the expected addresses:

Recv-Q                     Send-Q          Local Address:Port   Peer
Address:Port                     Process
0                       1 [2a0d:3344:379:31f0::ff]:41926
[2a04:4e42:600::644]:https
0                      25  [2a0d:3344:379:31f0::ff]:44470
  [2001:1b40:5600:ff80:f8ee::1]:https
0                      1 [2a0d:3344:379:31f0::ff]:33396      
[2a04:4e42:200::644]:https

So that does seem to show that https is being used:

Recv-Q    Send-Q       Local Address:Port       Peer Address:Port    Process
1           0                 [xxxx::xxxx]:46886
            [2001:1b40:5600:ff80:f8ee::1]:https
25          1                 [xxxx::xxxx]:45606
            [2001:1b40:5600:ff80:f8ee::1]:https
25          1                 [xxxx::xxxx]:60508
            [2001:1b40:5600:ff80:f8ee::1]:https

That rules out a transparent proxy at the ISP (which is unlikely, I'm on a business
class connection, not a consumer class.)

I'm not sure there's much point in pursuing this bug any further. I've found that
other people have reported this same bug:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1121610[1]

and within that bug (which I think is for a slightly different problem) it seems that
the issue with corrupted files is well known:

"The data races are down to concurrent downloads. They result in a
corrupted file in acng's cache and the only fix is to delete the corrupt
file and try again. Release files update often enough that wait is also
an option, but a corrupted kernel deb, for example, will stay broken for
ages."
There's also:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1003865[2]

which looks like exactly the same problem as my bug, which was reported four
years ago and remains unfixed.

Other distributions also have the same bug, e.g.:

https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/1998865[3]

As pointed out in the Ubuntu bug, this is a security issue because the use of apt-
cacher-ng results in security fixes being delayed due to the frequeny corruption of
InRelease files.

As apt-cacher-ng currently stands, it is basically unsafe to have it in the package
delivery path for any public facing machine as it *will* delay the application of
Debian security fixes.

Lastly, if we look at the changelog and changelog.Debian in the current stable
package, the last entry there is July 2024, which suggests that this package is no
longer maintained, especially when there are outstanding "Important" class bugs -
that makes it sound like apt-cacher-ng is basically unmaintained.

Thus, even if we did get to the bottom of this, I have zero confidence that there will
be a fixed version of apt-cacher-ng released.

Thus, my conclusion is that apt-cacher-ng is a piece of buggy and dead software
that should not be used. I will be removing it, and seeking a replacement.

Tj - Thanks for having a look at this problem, but I feel that it has been a waste of
both of our time.

--------
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1121610
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1003865
[3] https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/1998865

On the contrary I think your proving the problem lies within the code is
an important step. The codebase isn't too complex so I intend to:

a) create a simple corruption reproducer in order to accurately trace
the code

b) fix it

I suspect a reproducer, when restricted to a single thread, will not
corrupt files. If that is correct the fix may be as simple as adding
mutex at critical points.

Additional reproduction evidence for Debian bug #1143813
(apt-cacher-ng / InRelease signature verification)

On October 13, 2026, I reproduced this issue on two Debian 13 (Trixie)
systems sharing the same apt-cacher-ng instance:

- Raspberry Pi 5 (arm64): apt-cacher-ng on port 3142; APT proxy
http://127.0.0.1:3142
- Debian desktop (amd64): APT HTTP proxy http://192.168.50.1:3142
- Affected repository: http://deb.debian.org/debian trixie-updates
- Cache entry: /var/cache/apt-cacher-ng/debrep/dists/trixie-updates/InRelease

Both machines reported:
  Sub-process /usr/bin/sqv returned an error code (1)
  Verifying signature: Message has been manipulated

Findings:
1. The bad InRelease file in apt-cacher-ng's cache had SHA-256:
   706c13b5902fb74271db07319a366e929c58e9e3523edaae9ff48bf2a57079f5
   The Raspberry Pi's failed APT download had this same checksum.
2. The good copy, downloaded directly from Debian, had SHA-256:
   7348aa608c589b8d6421e8311f69deba685235a87681abe1879981f846f9122d
3. A diff showed identical repository metadata, with differences
confined to the ASCII-armored PGP signature block. The bad copy failed
independent sqv verification; the good copy verified successfully
against Debian's archive keys.
4. On the Pi, moving aside the local failed APT partial file and
bypassing the proxy allowed apt update to succeed. On the desktop,
moving aside the failed local partial file and retrying through the
proxy continued to fail.
5. I stopped apt-cacher-ng, moved the cache's InRelease and
InRelease.head files into a backup directory, and restarted the
service. Normal apt update then succeeded on both the Pi and desktop
(after moving aside the desktop's failed partial file).

The shared cache demonstrably held the invalid metadata, but I have
not established how the mismatched signature entered the cache.

The attached apt-cacher-ng-evidence-20261013.tar.xz contains
bad.InRelease and good.InRelease with the SHA-256 hashes above. The
original cache header is preserved separately but not attached.

I hope these files help diagnose the underlying issue.

This follows my October 13 report, which included
apt-cacher-ng-evidence-20261013.tar.xz with the failing and valid
trixie-updates InRelease files.

The issue affected both my Raspberry Pi 5 (Debian 13, arm64, running
apt-cacher-ng) and a Debian 13 amd64 desktop using it as a proxy. The
corrupted cached InRelease failed sqv verification on both clients.
Moving the cached InRelease and its .head file aside and allowing a
fresh retrieval resolved the issue.

I subsequently examined the original two files, preserved at:

/home/user/acng-bad-20261013-042256/InRelease
/home/user/acng-bad-20261013-042256/InRelease.head

The complete preserved InRelease.head contents were:

    HTTP/1.1 200 OK
    Content-Length: 47340
    Last-Modified: Tue, 13 Oct 2026 08:13:05 GMT
    X-Original-Source:
http://mirrors.xtom.com/debian/dists/trixie-updates/InRelease

The header records mirrors.xtom.com as the upstream source for this
cache entry. This does NOT establish that the mirror itself delivered
a bad file or caused the signature mismatch.

Relevant filesystem timestamps reported by stat (America/Boise,
UTC-06:00 on that date):

InRelease (47,340 bytes; owner apt-cacher-ng):
    Birth:  2026-08-11 12:29:24.172711360 -0600
    Modify: 2026-10-13 03:35:56.595302998 -0600
    Change: 2026-10-13 04:23:11.462275208 -0600

InRelease.head (170 bytes; owner apt-cacher-ng):
    Birth:  2026-06-26 22:37:53.206584491 -0600
    Modify: 2026-10-13 04:16:33.042699234 -0600
    Change: 2026-10-13 04:23:11.486275425 -0600

The 04:23 ctimes coincide with moving the files into the backup
directory. The old birth times reflect reused cache inodes, not
necessarily when these particular file contents were obtained.

Chronology (October 13, Boise local time):
- 02:13:05: Upstream Last-Modified timestamp (08:13:05 UTC). The
failed APT partial file also showed approximately 02:13 as mtime.
- 03:35:56: Cached InRelease last modified (09:35:56 UTC).
- Approximately 03:37: First observed signature verification error on the Pi.
- 04:16:33: Cached InRelease.head last modified (10:16:33 UTC), around
a failed desktop update.
- Approximately 04:23: I preserved both cached files by moving them aside.

Additional observation: I had taken the Pi to my mother's house on
another Internet connection for several days and ran apt update there
without signature errors. The failures began after I returned home. I
cannot determine whether the bad file was acquired on that other
network or after returning; the timestamps are suggestive of activity
at home but do not prove the download time or cause. In particular,
HTTP Last-Modified is not a client retrieval timestamp.

Previous attachment SHA-256 values:
- bad.InRelease:
706c13b5902fb74271db07319a366e929c58e9e3523edaae9ff48bf2a57079f5
- good.InRelease:
7348aa608c589b8d6421e8311f69deba685235a87681abe1879981f846f9122d

The repository metadata was byte-for-byte identical; the PGP signature
blocks differed, and independent sqv verification rejected the bad
copy.

I have kept the original 170-byte InRelease.head file and can provide
it if helpful. I hope this new source and timestamp evidence assists
the investigation without presuming the mirror or either network was
at fault.