On Debian sid, apt update is broken for some days now

$ sudo apt update
Get:1 http://deb.debian.org/debian sid InRelease [165 kB]
Err:1 http://deb.debian.org/debian sid InRelease
  The following signatures were invalid: BADSIG 648ACFD622F3D138 Debian
Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
Get:2 http://deb.debian.org/debian experimental InRelease [75.4 kB]
Err:2 http://deb.debian.org/debian experimental InRelease
  The following signatures were invalid: BADSIG 648ACFD622F3D138 Debian
Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
Hit:3 https://packages.riot.im/debian sid InRelease
Hit:4 https://people.debian.org/~praveen/nheko unstable InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1369 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: An error occurred during the signature verification. The repository
is not updated and the previous index files will be used. GPG error:
http://deb.debian.org/debian sid InRelease: The following signatures
were invalid: BADSIG 648ACFD622F3D138 Debian Archive Automatic Signing
Key (10/buster) <ftpmaster@debian.org>
W: An error occurred during the signature verification. The repository
is not updated and the previous index files will be used. GPG error:
http://deb.debian.org/debian experimental InRelease: The following
signatures were invalid: BADSIG 648ACFD622F3D138 Debian Archive
Automatic Signing Key (10/buster) <ftpmaster@debian.org>
W: Failed to fetch http://deb.debian.org/debian/dists/sid/InRelease The
following signatures were invalid: BADSIG 648ACFD622F3D138 Debian
Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
W: Failed to fetch
http://deb.debian.org/debian/dists/experimental/InRelease The following
signatures were invalid: BADSIG 648ACFD622F3D138 Debian Archive
Automatic Signing Key (10/buster) <ftpmaster@debian.org>
W: Some index files failed to download. They have been ignored, or old
ones used instead.

Control: tags -1 + moreinfo unreproducible

Based on the fact that autobuilding of unstable is still continuing and
there haven't been any reports of this issue on either #d-devel or
debian-devel@lists, it seems unlikely that there's a general issue
here, rather than something more local to your environment.

In any case, if there were an issue, it would either be with the
archive itself (which is ftp.d.o territory) or with deb.d.o (which is
mirror team / DSA territory), not with either debian-archive-keyring
(which simply ships the public keys) or APT.

Regards,

Adam

2022, ജനുവരി 19 12:23:06 AM IST, "Adam D. Barratt" <adam@adam-barratt.org.uk>ൽ എഴുതി

I have seen at least one forum posting on the same error when searching for it so its likely more common
https://www.nixcraft.com/t/signatures-were-invalid-badsig-648acfd622f3d138-debian-archive-automatic-signing-key-10-buster/4025

May be related to apt-cacher-ng (though I tried with apt-cacher-ng disabled without fixing this issue).

Agreed. I initially thought it could be because of wrong expiry dates on keys.

sudo rm -rf /var/lib/apt/*

fixed the issue (sharing in case someone someone searches for the same
error)

sudo rm -rf /var/lib/apt/*

fixed the issue (sharing in case someone someone searches for the same
error)

When I encounter similar errors from time to time (once a year or so) I
consider them as "caching artefacts" and fix them by having apt
"reinitialize" the corresponding package source: first comment the line
in sources.list (or rename the snippet in sources.list.d to *.list.off),
run apt-get update (or whatever you like instead) to "forget" the
source, enable it again, apt-get update again and the error is gone.

Actually I cannot remember having ever seen that as a piuparts failure
(and that does a lot of apt-get update), only once in a while on my main
machine which has everything from oldoldstable to experimental with 4
foreign architectures available ...

Andreas

Control: reassign -1 apt-cacher-ng

I have seen this on another sid chroot. So I had to remove
/var/lib/apt/* inside the chroot and debrepo/dists directory in
apt-cacher-ng's cache directory. I'm not sure if it can be fixed in
apt-cacher-ng.

On Fri, 28 Jan 2022 20:45:00 +0530 Pirate Praveen  <praveen@onenetbeyond.org> wrote:
 >
 > I have seen this on another sid chroot. So I had to remove
 > /var/lib/apt/* inside the chroot and debrepo/dists directory in
 > apt-cacher-ng's cache directory. I'm not sure if it can be fixed in
 > apt-cacher-ng.

This seems to be happening very often, I had to fix this manually today
as well. Wondering if something needs to be done at apt-cacher-ng side
to invalidate the cached version?

Today I also faced this issue.
I tried with and without apt-cacher-ng.

moreover (I do not know if I check properly)
================
%gpg --keyserver keyring.debian.org --recv-key 0x648ACFD622F3D138
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
============

Also checked:

rsync -az --progress keyring.debian.org::keyrings/keyrings/ .
for q in *; do gpg --list-packets <$q ; done |grep 648ACFD622F3D138

[...empty...]

I'm having/seeing this issue for ~1 week now and it very much looks like it is
caused by apt-cacher-ng.
I don't know why it popped up now as usually/normally it works.

# debootstrap --arch arm64 sid /home/diederik/tmp/debootstrap/sysbase-sid-arm64 http://<acng-machine>:3142/deb.debian.org/debian
I: Target architecture can be executed
I: Retrieving InRelease
I: Checking Release signature
E: Invalid Release signature (key id 648ACFD622F3D138)

# debootstrap --arch arm64 sid /home/diederik/tmp/debootstrap/sysbase-sid-arm64 http://deb.debian.org/debian
I: Target architecture can be executed
I: Retrieving InRelease
I: Checking Release signature
I: Valid Release signature (key id A7236886F3CCCAAD148A27F80E98404D386FA1D9)

Interestingly enough, this only happens when specifying 'sid':

# debootstrap --arch arm64 testing /home/diederik/tmp/debootstrap/sysbase-bookworm-arm64 http://<acng-machine>:3142/deb.debian.org/debian
I: Target architecture can be executed
I: Retrieving InRelease
I: Checking Release signature
I: Valid Release signature (key id A7236886F3CCCAAD148A27F80E98404D386FA1D9)

Do you do that on the host machine or on the machine running apt-cacher-ng?

I have a device running acng in my LAN which all (other) devices use,
but I also have one running on my main PC when repeatedly building images.

I just stumbled in the same problem.

I had to follow these step to actually fix this and it seems none can be
skipped:

1 remove offending sources from /etc/apt/source.list
2 zap /var/lib/apt/lists
3 disable the proxy
4 apt update
5 add back the offending sources from /etc/apt/source.list
6 apt update
7 enable the proxy

I had to put together all the receipt I found on the net to succeed.

Removing the offending source, apt update, reenabling the offending
source didn't work. (1 4 5 6)

disabling the proxy alone didn't help 3.

2 3 4 5 didn't work either

I had the chance to fix this on 2 machines and on the second one I tried
to anticipate to re-enable the proxy just after putting back the sources
(6) and it didn't work.

I hope it helps to find the problem.

Adding more data...

I have been seeing this on bookworm (both the machine running "apt
update" and the machine running apt-cacher-ng):

W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: The following signatures were invalid: BADSIG 0E98404D386FA1D9 Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.

The above workaround has fixed it for me... for now.

peace & happiness,
martin

Hi,

since I updated the host that my apt-cacher-ng runs on to bookworm, I
see this happen about once a week for all machines that get their
updates via this instance of apt-cacher-ng:

[2/4650]mh@banana:~ $ sudo apt update
Hit:1 http://debian-security.debian.zugschlus.de/debian-security bookworm-security InRelease
Hit:2 http://security.debian.org bookworm-security InRelease
Hit:3 http://zg20150.debian.zugschlus.de/zg20150 bookworm-zg-stable InRelease
Hit:4 http://zg20150.debian.zugschlus.de/zg20150 bookworm-zg-unstable InRelease
Hit:5 http://debian.debian.zugschlus.de/debian bookworm InRelease
Hit:6 http://zg20150.debian.zugschlus.de/zg20150 sid-zg-stable InRelease
Get:7 http://debian.debian.zugschlus.de/debian experimental InRelease [109 kB]
Hit:8 http://zg20150.debian.zugschlus.de/zg20150 sid-zg-unstable InRelease
Get:9 http://debian.debian.zugschlus.de/debian sid InRelease [210 kB]
Err:7 http://debian.debian.zugschlus.de/debian experimental InRelease
  The following signatures were invalid: BADSIG 0E98404D386FA1D9 Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
Err:9 http://debian.debian.zugschlus.de/debian sid InRelease
  The following signatures were invalid: BADSIG 0E98404D386FA1D9 Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
Fetched 319 kB in 11s (29,1 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
29 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://debian.debian.zugschlus.de/debian experimental InRelease: The following signatures were invalid: BADSIG 0E98404D386FA1D9 Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://debian.debian.zugschlus.de/debian sid InRelease: The following signatures were invalid: BADSIG 0E98404D386FA1D9 Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
W: Failed to fetch http://debian.debian.zugschlus.de/debian/dists/experimental/InRelease  The following signatures were invalid: BADSIG 0E98404D386FA1D9 Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
W: Failed to fetch http://debian.debian.zugschlus.de/debian/dists/sid/InRelease  The following signatures were invalid: BADSIG 0E98404D386FA1D9 Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>
W: Some index files failed to download. They have been ignored, or old ones used instead.
[3/4650]mh@banana:~ $

debian.debian.zugschlus.de points to the IPv4 address of my apt-cacher-ng,
the relevant configuration line is
Remap-debian: debian ; debian.apt-cache.zugschlus.de/debian
and debian.apt-cache.zugschlus.de is a CNAME to ftp2.de.debian.org

On my machines on the Internet that don't use apt-cacher-ng,
debian.debian.zugschlus.de is directly a CNAME to ftp2.de.debian.org and
that works flawlessly even during the time when my apt-cacher-ng is
acting up.

I can fix the issue without deleting any files on my host, I call up my
apt-cacher-ng webpage /acng-report.html, click on "Start Scan and/or
Expiration" while keeping the defaults, tell acng to override the sanity
check and to go ahead anyway ("Override this check now"), wait, click on
"Check all", and then on "Delete selected files", and then "Delete now".

This makes the issue pretty clearly an acng issue.

Greetings
Marc

also here the error occurs with all Linux installations that use apt-cacher-ng as apt proxy.
As a workaround, it helps to deactivate the caching of the InRelease files so that they are always fetched directly.

##################
/etc/apt-cacher-ng/acng.conf
DontCacheResolved: .*InRelease
##################

Regards
Klaus

Hi,
I have the same problems and I think a can describe the problem a bit
more closely now. The problem is the conditional request of APT towards
the apt-cacher-ng. The conditional request can be simulated by curl for
example. The problem is, that APT doesn't need to fetch the InRelease
file, because it is not newer than its already stored
(/var/lib/apt/lists/...), but in time between the apt-cacher-ng started
to fetch some chunks of the InRelease file and it is storing it to its
cache file. But as long as it realize the requestor (APT) no longer
needs the data, send RST on connection to its backend mirror and left
incomplete data in cache file. This can be repeated several times and
apt-cacher-ng fetches and stores more and more of InRelease file, until
it is complete.

My hypothesis: if the InRelease file have changed, the apt-cacher-ng
doesn't realize, the data changed and continue in refetching the
InRelease file resulting in inconsistent file and failed GPG signature
finally. The InRelease file is then built by apt-cacher-ng from chunks of
different versions of this file from backend mirror.

Short experimenting can demonstrate the potential  problem. I have
configured

  deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# cat /etc/apt-cacher-ng/local.conf
  CacheDir: /srv/cache/apt-cacher-ng
  Port: 9999
  Remap-hwraidrep: /hwraid        ; https://hwraid.le-vert.net/debian/
  Remap-debsecrep: /security      ; https://security.debian.org/debian-security/
  Remap-debsecrep: /debian-security ; https://security.debian.org/debian-security/
  LocalDirs: debian-icz /srv/debian-icz

  deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# rm InRelease*

  deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# apt update
  Get:1 http://deb:9999/debian-icz icz-bookworm InRelease [3,938 B]
  Hit:2 http://deb:9999/debian bookworm InRelease
  Hit:3 http://deb:9999/debian bookworm-updates InRelease
  Hit:4 http://deb:9999/debian bookworm-backports InRelease
  Hit:5 http://deb:9999/security bookworm-security InRelease
  Fetched 3,938 B in 1s (3,860 B/s)
  Reading package lists... Done
  Building dependency tree... Done
  Reading state information... Done
  All packages are up to date.

  deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# ll InRelease*
  -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 10519 Mar 26 18:21 InRelease
  -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng   166 Mar 26 18:21 InRelease.head


  deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# cat InRelease.head
  HTTP/1.1 200 OK
  Content-Length: 151073
  Last-Modified: Sat, 15 Mar 2025 09:18:33 GMT
  X-Original-Source: http://ftp.cz.debian.org/debian/dists/bookworm/InRelease

  deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# rm InRelease*
  deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# for x in {1..20}; do curl  --header 'If-Modified-Since: Sat, 15 Mar 2025 09:18:33 GMT' http://localhost:9999/debian/dists/bookworm/InRelease; stat -c "%10s %n" InRelease; done
        2431 InRelease
       12896 InRelease
       22012 InRelease
       35172 InRelease
       41592 InRelease
       52056 InRelease
       65217 InRelease
       70290 InRelease
       74015 InRelease
       80436 InRelease
       89553 InRelease
      100018 InRelease
      113178 InRelease
      119598 InRelease
      130062 InRelease
      131090 InRelease
      141554 InRelease
      150671 InRelease
      151073 InRelease
      151073 InRelease

  deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# gpg --verify --keyring /usr/share/keyrings/debian-archive-keyring.gpg InRelease
  gpg: Signature made Sat 15 Mar 2025 10:10:27 AM CET
  gpg:                using RSA key A7236886F3CCCAAD148A27F80E98404D386FA1D9
  gpg: Good signature from "Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>" [unknown]
  gpg: WARNING: This key is not certified with a trusted signature!
  gpg:          There is no indication that the signature belongs to the owner.
  Primary key fingerprint: 1F89 983E 0081 FDE0 18F3  CC96 73A4 F27B 8DD4 7936
       Subkey fingerprint: A723 6886 F3CC CAAD 148A  27F8 0E98 404D 386F A1D9
  gpg: Signature made Sat 15 Mar 2025 10:10:28 AM CET
  gpg:                using RSA key 4CB50190207B4758A3F73A796ED0E7B82643E131
  gpg: Good signature from "Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>" [unknown]
  gpg: WARNING: This key is not certified with a trusted signature!
  gpg:          There is no indication that the signature belongs to the owner.
  Primary key fingerprint: B8B8 0B5B 623E AB6A D877  5C45 B7C5 D7D6 3509 47F8
       Subkey fingerprint: 4CB5 0190 207B 4758 A3F7  3A79 6ED0 E7B8 2643 E131
  gpg: Signature made Sat 15 Mar 2025 10:13:02 AM CET
  gpg:                using EDDSA key 4D64FEC119C2029067D6E791F8D2585B8783D481
  gpg: Good signature from "Debian Stable Release Key (12/bookworm) <debian-release@lists.debian.org>" [unknown]
  gpg: WARNING: This key is not certified with a trusted signature!
  gpg:          There is no indication that the signature belongs to the owner.
  Primary key fingerprint: 4D64 FEC1 19C2 0290 67D6  E791 F8D2 585B 8783 D481


In this case, the GPG signarute is OK, because the InRelease file was static
in upstream mirror. I have strong suspicion, that apt-cacher-ng doesn't detect the
upstream file change in this partially stored file. I don't inspect the code.

I'm pretty sure this is the same bug as #1022043.

There's a patch on that bug that resolves this. The race happens when
two (or more) clients request the same file at the same time that apt
does not have cached.