#1022043 apt-cacher-ng: sometimes fails to download some index files - maybe a race condition #1022043
- Package:
- apt-cacher-ng
- Source:
- apt-cacher-ng
- Description:
- caching proxy server for software repositories
- Submitter:
- Václav Ovsík
- Date:
- 2025-03-27 07:15:02 UTC
- Severity:
- normal
- Tags:
Package: apt-cacher-ng Version: 3.7.4-1~bpo11+1 Severity: normal Dear Maintainer, I have errors while updating apt database with several concurrent requests. I usually run apt update using Fabric framework on many servers in the same moment and apt-cacher-ng has a trouble with this. Apt shows that some Release file has invalid signature or some other problem with index files. I found this errors in error log: ns1:/srv/cache/apt-cacher-ng/debrep/dists/bullseye# tail /var/log/apt-cacher-ng/apt-cacher.err Wed Oct 19 13:47:18 2022|Error creating file item for debrep/dists/bullseye-updates/InRelease -- check file permissions! Wed Oct 19 13:47:18 2022|Failure to move file /srv/cache/apt-cacher-ng/debrep/dists/bullseye-backports/InRelease out of the way or cannot create /srv/cache/apt-cacher-ng/debrep/dists/bullseye-backports/InRelease.1666180038 - errno: File exists Wed Oct 19 13:47:18 2022|Error creating file item for debrep/dists/bullseye-backports/InRelease -- check file permissions! Wed Oct 19 13:47:18 2022|Failure to move file /srv/cache/apt-cacher-ng/debsecrep/dists/bullseye-security/InRelease out of the way or cannot create /srv/cache/apt-cacher-ng/debsecrep/dists/bullseye-security/InRelease.1666180038 - errno: File exists Wed Oct 19 13:47:18 2022|Error creating file item for debsecrep/dists/bullseye-security/InRelease -- check file permissions! Wed Oct 19 13:47:18 2022|/srv/cache/apt-cacher-ng/adoptopenjdk/dists/bullseye/InRelease.1666180038 storage error [Checked size beyond EOF], check file AND directory permissions, last errno: File exists Wed Oct 19 13:47:18 2022|Failure to move file /srv/cache/apt-cacher-ng/debsecrep/dists/bullseye-security/InRelease out of the way or cannot create /srv/cache/apt-cacher-ng/debsecrep/dists/bullseye-security/InRelease.1666180038 - errno: File exists Wed Oct 19 13:47:18 2022|Error creating file item for debsecrep/dists/bullseye-security/InRelease -- check file permissions! Wed Oct 19 13:47:18 2022|Failure to move file /srv/cache/apt-cacher-ng/debrep/dists/bullseye/InRelease out of the way or cannot create /srv/cache/apt-cacher-ng/debrep/dists/bullseye/InRelease.1666180038 - errno: File exists Wed Oct 19 13:47:18 2022|Error creating file item for debrep/dists/bullseye/InRelease -- check file permissions! ns1:/srv/cache/apt-cacher-ng/debrep/dists/bullseye# ls -la total 3684 drwxr-xr-x 5 apt-cacher-ng apt-cacher-ng 4096 Oct 19 13:47 . drwxr-xr-x 15 apt-cacher-ng apt-cacher-ng 4096 Oct 13 18:35 .. drwxr-xr-x 5 apt-cacher-ng apt-cacher-ng 4096 Oct 14 06:25 contrib -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 19 13:47 InRelease -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Sep 27 16:49 InRelease.1664901121 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 4 22:03 InRelease.1664963050 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 5 11:44 InRelease.1664967465 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 32768 Oct 14 06:25 InRelease.1664967466 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 5 13:07 InRelease.1664968046 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 16384 Oct 14 06:25 InRelease.1664968047 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 5 14:27 InRelease.1664976362 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 5 15:26 InRelease.1664987237 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 5 18:27 InRelease.1665135746 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 7 11:42 InRelease.1665143341 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 7 14:27 InRelease.1665146115 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 7 14:35 InRelease.1665423845 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 16384 Oct 14 06:25 InRelease.1665423846 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 10 19:44 InRelease.1665917759 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 16 12:55 InRelease.1665917760 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 16 12:56 InRelease.1665922621 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 16384 Oct 16 14:17 InRelease.1665922622 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 16 14:17 InRelease.1665937102 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 16 18:18 InRelease.1665937103 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 16 18:18 InRelease.1665943973 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 16 20:12 InRelease.1665943974 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 16 20:12 InRelease.1666009621 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 17 14:35 InRelease.1666037442 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 17 22:10 InRelease.1666037443 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 17 22:55 InRelease.1666075608 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 18 08:46 InRelease.1666075609 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 32768 Oct 18 08:46 InRelease.1666080465 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 18 10:07 InRelease.1666168744 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 19 10:39 InRelease.1666168745 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 19 10:39 InRelease.1666169228 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 19 10:47 InRelease.1666169229 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 19 11:03 InRelease.1666176606 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 115947 Oct 19 12:50 InRelease.1666180038 -rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 163 Oct 19 14:09 InRelease.head drwxr-xr-x 5 apt-cacher-ng apt-cacher-ng 4096 Oct 13 10:32 main drwxr-xr-x 5 apt-cacher-ng apt-cacher-ng 4096 Oct 14 06:25 non-free ns1:/srv/cache/apt-cacher-ng/debrep/dists/bullseye# df /srv Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/ns1-srv 15375304 1673112 12899376 12% /srv Is it possible, that many concurrent requests go to apt-cacher-ng and it tries to download the new InRelease file several times and to do filesystem move operation in the same moment? Thanks for your work!
I can confirm this behaviour on Ubuntu 22.04. I see the creation of an Inrelease.timestamp also with 1 client. My apt-cacher-ng installation is upgraded from 3.3.1 to the version mentioned in this report. Unfortunately I couldn't find any helpfull extra information. I tried making some changes to the config file, which don't seem to help. MaxConThreads: 300 DlMaxRetries: 4 PrecacheFor: uburep/dists/jammy/InRelease KeepExtraVersions: 1 FreshIndexMaxAge: 300 VfileUseRangeOps: -1 PassThroughPattern: ^(.*):443$ ResponseFreezeDetectTime: 30
I've managed to reproduce this issue when many hosts hit apt-cacher-ng at the same time. Attached a patch which fixes it for me - this is a quick and hacky patch! Sent debug logs of a run that reproduces this problem and a run with this patch applied directly to the maintainer.
I've cleaned up the patch and got rid of a lot of the cruft that got added while I was debugging. Also found one more race condition in my original solution that I believe is now fixed. And I believe it's now safe even if different urls map to the same cache file which the previous patch didn't get right.
Try again - without the old patch at the top.
Hi Tim, thanks for the provided patch! We see the same issue here, so I included it in a locally built package to give it a try on our infrastructure. So far it looks quite promising, no errors up to now . Best regards, Andi
That's great! I've been running it for a few weeks now and also seen no errors since I started using it. I believe it's possibly fixed a second "buggette" too although I never confirmed that this really existed and I haven't confirmed that it's definitely gone away either. If you simultaneously install a big package on multiple machines at the same time (e.g. a kernel upgrade) then I believe that apt-cacher-ng used to download it from upstream multiple times. Only once it had a copy did it serve that up. This was noticable to me in the past because I was on a relatively slow download (c 2MB/s) so kernels would take a noticable amount of time to download and 10 in parallel could take a lot longer than a single download should have taken. I used to work around this by triggering the download on a single machine and then only triggering the rest once that one had completed downloading. Since applying this fix I've done another mass kernel upgrade. I'm on a much faster connection now so I didn't really think about triggering the download in advance but it only downloaded the kernel once. zgrep linux-image-5.10.0-28 * access.log.2.gz:1708183270.671 17321 <apt-cacher-ng address> TCP_MISS/200 55667185 GET http://ftp.debian.org/debian/pool/main/l/linux-signed-amd64/linux-image-5.10.0-28-amd64_5.10.209-2_amd64.deb - HIER_DIRECT/2a04:4e42:4b::644 application/vnd.debian.binary-package Next time I'll try and take more care to watch what really happens here (although with my much faster connection a kernel downloads in a few seconds anyway) Tim.
Hi Tim, Same here, still no errors! I guess this really should be addressed in the package, perhaps even in a point release. Unfortunatelly it looks like apt-cacher-ng is unmaintained for the time being, that's a pitty . :-( Best Regards, Andi
Hello Tim, I have created a new version of apt-cacher-ng for our infrastructure using the last version of your patch and can confirm that all observed issues are now resolved, too. Our infrastructure currently consists of 38 locations. At each location there is a server with apt-cacher-ng installed. Each server is accessed by 5 - 40 apt-clients in the middle of the night in the same second. As a result, the race conditions occurred regularly (daily) for us. Now everything runs without problems. Without this patch, apt-cacher-ng was unusable for us. Different errors occurred every day (from apt-client hangs to signature errors). I wouldn't be surprised if this patch also fixes other reported bugs. It's really a pitty that apt-cacher-ng is currently not maintained. Best Regards, Arndt
Hello Eduard, I ported version 3.7.5-1 to Bookworm and also have problems with this version without this patch. There are currently more than 750 Debian hosts in my environment that download their packages via apt-get from 38 apt-cacher-ng hosts. If all hosts run an apt-get update at the same time (cron job), the logs are full of the following Messages (generated by apt-get): W: Tried to start delayed item http://ftp.de.debian.org/debian bookworm InRelease, but failed W: Tried to start delayed item http://ftp.de.debian.org/debian bookworm InRelease, but failed W: Tried to start delayed item http://ftp.de.debian.org/debian bookworm InRelease, but failed W: Tried to start delayed item http://ftp.de.debian.org/debian bookworm InRelease, but failed . . . Most of the time, the download of the InRelease files does not work and apt-get terminates with an error. However, I have also sporadically observed problems downloading packages when the download was started from multiple client hosts at the same time. If I apply this patch to version 3.7.5-1 and install the newly built package on the 38 apt-chacher-ng hosts, I no longer see any problems. Therefore, in my opinion, this patch (or a similar solution) should be included in the upstream version (and in Trixie). Can you please take a look at this issue/patch? Is there anything I can do to help with this? I can test and develop in C/C++ and would really like to help make apt-cacher-ng more stable. Best Regards, Arndt
Hi,
I also did a rebuild of the version 3.7.5-1 with the T. Woodall's patch
and still have a problem. The problem is with the InRelease file
signature. I don't understand where can be the cause of the problem.
I still suspect the problem is concurent downloads.
ns1:/srv/cache/apt-cacher-ng/debrep/dists/bookworm-updates# ll
total 76
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 55443 Feb 4 16:05 InRelease
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 170 Feb 4 21:59 InRelease.head
drwxr-xr-x 5 apt-cacher-ng apt-cacher-ng 4096 Jan 24 06:26 contrib
drwxr-xr-x 5 apt-cacher-ng apt-cacher-ng 4096 Jan 24 06:26 main
drwxr-xr-x 5 apt-cacher-ng apt-cacher-ng 4096 Jan 24 06:26 non-free
drwxr-xr-x 5 apt-cacher-ng apt-cacher-ng 4096 Jan 24 06:26 non-free-firmware
ns1:/srv/cache/apt-cacher-ng/debrep/dists/bookworm-updates# cat InRelease.head
HTTP/1.1 200 OK
Content-Length: 55443
Last-Modified: Tue, 04 Feb 2025 14:32:02 GMT
X-Original-Source: https://ftp.debian.cz/debian/dists/bookworm-updates/InRelease
ns1:/srv/cache/apt-cacher-ng/debrep/dists/bookworm-updates# curl --head https://ftp.debian.cz/debian/dists/bookworm-updates/InRelease
HTTP/1.1 200 OK
Date: Tue, 04 Feb 2025 21:00:06 GMT
Server: Apache/2.4.62 (Debian)
Last-Modified: Tue, 04 Feb 2025 14:32:02 GMT
ETag: "d893-62d51df5099d3"
Accept-Ranges: bytes
Content-Length: 55443
ns1:/srv/cache/apt-cacher-ng/debrep/dists/bookworm-updates# wget -O /tmp/InRelease https://ftp.debian.cz/debian/dists/bookworm-updates/InRelease
--2025-02-04 22:01:21-- https://ftp.debian.cz/debian/dists/bookworm-updates/InRelease
Resolving ftp.debian.cz (ftp.debian.cz)... 78.128.211.127, 2001:718:1:1f:50:56ff:feee:127
Connecting to ftp.debian.cz (ftp.debian.cz)|78.128.211.127|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 55443 (54K)
Saving to: ‘/tmp/InRelease’
/tmp/InRelease 100%[=============================================================================>] 54.14K --.-KB/s in 0.005s
2025-02-04 22:01:21 (11.7 MB/s) - ‘/tmp/InRelease’ saved [55443/55443]
ns1:/srv/cache/apt-cacher-ng/debrep/dists/bookworm-updates# md5sum InRelease /tmp/InRelease
42a3992a340220eaedbe41f3ea03fb6b InRelease
1e1c8c8a3eb793e0994a5b30150cc02f /tmp/InRelease
The content of both files /tmp/InRelease and InRelease from
apt-cacher-ng's cache is the same, except the signature at the end.
ns1:/srv/cache/apt-cacher-ng/debrep/dists/bookworm-updates# gpg --verify --keyring /usr/share/keyrings/debian-archive-keyring.gpg /tmp/InRelease
gpg: Signature made Tue 04 Feb 2025 03:31:25 PM CET
gpg: using RSA key A7236886F3CCCAAD148A27F80E98404D386FA1D9
gpg: Good signature from "Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 1F89 983E 0081 FDE0 18F3 CC96 73A4 F27B 8DD4 7936
Subkey fingerprint: A723 6886 F3CC CAAD 148A 27F8 0E98 404D 386F A1D9
gpg: Signature made Tue 04 Feb 2025 03:31:57 PM CET
gpg: using RSA key 4CB50190207B4758A3F73A796ED0E7B82643E131
gpg: Good signature from "Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: B8B8 0B5B 623E AB6A D877 5C45 B7C5 D7D6 3509 47F8
Subkey fingerprint: 4CB5 0190 207B 4758 A3F7 3A79 6ED0 E7B8 2643 E131
ns1:/srv/cache/apt-cacher-ng/debrep/dists/bookworm-updates# gpg --verify --keyring /usr/share/keyrings/debian-archive-keyring.gpg InRelease
gpg: Signature made Tue 04 Feb 2025 09:19:26 AM CET
gpg: using RSA key A7236886F3CCCAAD148A27F80E98404D386FA1D9
gpg: BAD signature from "Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>" [unknown]
gpg: Signature made Tue 04 Feb 2025 09:19:56 AM CET
gpg: using RSA key 4CB50190207B4758A3F73A796ED0E7B82643E131
gpg: BAD signature from "Debian Archive Automatic Signing Key (12/bookworm) <ftpmaster@debian.org>" [unknown]
ns1:/srv/cache/apt-cacher-ng/debrep/dists/bookworm-updates# diff -u /tmp/InRelease InRelease
--- /tmp/InRelease 2025-02-04 15:32:02.000000000 +0100
+++ InRelease 2025-02-04 16:05:01.597595082 +0100
@@ -496,29 +496,29 @@
bd8fb630f74032553aa5942fc3ca546102d4a1b86dae248e2b0931170bb6a3d7 1544 non-free/source/Sources.xz
-----BEGIN PGP SIGNATURE-----
-iQIzBAEBCAAdFiEEpyNohvPMyq0Uiif4DphATThvodkFAmeiJL0ACgkQDphATThv
-odli6Q/+PTI18TtKuTsvER3kTXgnrAoFE6RohHJQPxbN2fky51I19fTw1aipXaJ0
-/R9INFF6KVYcO8VLmPqgbZItpt7EJEUve6eqLEsiVa0QzvRRIZfF8hYoGlPc9FLF
-vy2YiPqbgNo/eYvjocfvw5wywm8HKEWmPH6AIn9PU9M69vaT1atZ0+8LXDWIdV2D
-xcIL7H6CE6lceD96F3+9nrn6IZFkwQlr3oJtCgO9e5TU8a55x0J2UEkje4mm4kS0
-FraRwnhRXRTrONkweQKpMQ7Dp+DUMeH7OtEfEPNo0UMbYPC+/XMJnm39vr26p5Ar
-Bc4pgRgKsIeGltMetuTy4yupVifNNgYS+Tm0Vp5A8AmizKScnNhHCIfPFrDF+vzf
-g6cC6Bdf1fgWBK5w7uiR4V7v8uukrORM9m2fBoNy4tY+8LRO01lyP9PdISEVSihd
-fzZolCP+/13TsYzko9cFT6dRcabiQE+sa/ZuAkU6FYfMtbJ0YqpEDtiyFK+br7ad
-4lPA78xaDrfOFZpqRvQr2zIT71qC+LTNEZj3WtwGhSt2YOjjEdwn7/f3ql4NBBO3
-/H5xsEk0G8cFhyrUhIeXJ1jSHPygGjaj5EVA+wM1ECGZZxMv6I8BO5+9Ly++67qd
-PyzepebNAtEPUzO2XcY93Aky1hMdY6kz/zLxqmRXbyA9dQaHLHqJAjMEAQEIAB0W
-IQRMtQGQIHtHWKP3Onlu0Oe4JkPhMQUCZ6Ik3QAKCRBu0Oe4JkPhMVMcEADQKd9q
-TtdMsMUhvjCmYc40AXBtEgUkOurd9+M58pLuAdqCa2ApJA0DXjQlAVFM0BV95b2O
-yyoyc9AhGjTSdFB2tPlp6RzIBrrOh3V1mszt99oVkGKtBx3HBa967GmfgXRaXp0e
-ZDEpzfWUlNN7MxY+vNXvIbWYxUBh0R/5xfITv6Cbb8bC0mcaoIso836wHiATBkqh
-iwNCyNEaXoQv1hdpJg2k7AIi9dvIGMXubkq+QhPaSVvHzZXAH/UKODubBd5U+voQ
-Q2+3/PpXCcnzYq/jbtr8Yhz8tUaBinCCnYhW0j/+pHZAQQEH6wErjKMfjJa5wE7I
-MlxBddsP+NMQyI07UhXk0rTulMsbZMfkrOD6FYoWESAt8OkiMMoihwBYRK/h9Mcl
-Xl9tfO+u99hg31bnClPV3zazMM0NoRaBLF9xq84EjJd8K7RhbAxdaTAkud8LV4lr
-vgumCLvAARSDZU8qbhaeApE91V99GSiSNGMYHRRWNZnRciOBVv3sspqkCJ91WYin
-ozYKI4LKPFkYqAWBs5KNR9Llrih3ZBcxB/fugeBraFmvPQC5k5za5ONonn+ST3k/
-Rb7mjubThamBiov0IG76iGDTi18nhsdyIiZWDbOqXS+/6Vbqwsb4VgA8eiD52820
-MlYTzAPp1G8D+qK+r/1ClvfRGZYLgscv+4nltg==
-=8ep+
+iQIzBAEBCAAdFiEEpyNohvPMyq0Uiif4DphATThvodkFAmehzY4ACgkQDphATThv
+odlnRw//deKBlDUh+2yvG88wZ7qAVnZmvMZixtULJoQo/Wv4MQEHygsmeVyZhV9m
+yKCdJIL9cpJHOwghwRtvF1n5wMFBiMMOlfnCAucrTyvMdIY+jf2VolsLmI6IfXl3
+3SAdeK0NL7V8Ll4IqSpdD0kJ6aoyV3hfOD2FH8CCv605ayUxifXXWutK34bpsfNh
+QWCqiJkxax/ZihD9Qt2rZNi8W4oOtvihYZILV5NSQCzZP33i4QXnAkMSje4XRue1
+7RuBQXZA6myfdGzrdMBnc/is+81VkbgduZujfzUd94jB5/P8oI/OfcR8c9GnK9pY
+0Sto6GcwP6wmvFJcYCPqZSRkFSxpqWswhMrg4web/YQV5DBU+1xIHVyoJ8TxL2nh
+LwnHe8WHgsAJdCNdFWElLfXAPGGBuwOITP5lPiTgniHXl0mCO2e2Vhtxjhy/H7hI
+oo7tvecT32CmvJYWFzndmWbl2ClYnmTxl9MAwKgW+sJJ6IJi2dpIzGtaNr7zMsPw
+lW2bi+kaj/255HBsIIgkTx8xOSciDTDBldXOgyeZEurOu3cxDlLV06zPn3s6B98J
+XUwWA2u7Cr2huwRJgC836krIIoNGGqt6qzHTcGpB1wpYZEQtephx89XWAEqZyD2y
+POZBL3FdwuX7pS0RpSSjshl8w63lqnMdhMIsc4uyWin+goDIG+mJAjMEAQEIAB0W
+IQRMtQGQIHtHWKP3Onlu0Oe4JkPhMQUCZ6HNrAAKCRBu0Oe4JkPhMQnqEADQQzCR
+p1RngkK/1Ab1+c9nWqUvo+SZREeBHcSo1LgOUZY46hek5mCSlmdt5EUZ9CmW0I5X
+y4yJn1c/8vnMYQOuBhTq0ue5j5sx7wuGTMnU4HbWM9rbLOwE35oJj7SZKc3AXVZC
+4K0aczgowT0i05KP9k6YRP+OvpYptTOrmS1Oxl+ykA+MyFTrXAOedMj2L51Bwjpo
+HUugPRIzY5/B+6CN9deVUp9ggdUdrmG70/vEEGyM5EmMjZBiR/Ul0aJ5NUDKzntz
+bhAZ3wrdtEzqkliRTvZmfwSecUqMUbd9c1n7owA6LtOM24BaLdPOsO4fLCja8rLf
+dnqKjRNq7RfnuGPTqR0kBHm3B2LQGLUS5pOvHr7kd/gQRWoafCcZrgCp/pKlqjQ4
+alXPIL5QKHvFbvIq8nKFnxAivfwkuEs5yMmU5B/zXnNl0opu9CZGKWpmIa7eM378
+pyyvD7Cde6BtRjeWndkzDoYf76gXMloKyvAhnf8OuWqWUirK8sFCvCwmJvX1Qakl
+0/tKj0ShumVWuSeBwvnZa8Y4Ha11YBbiN7EZLLQf1yDifSZfBtW6v6bMqESbVMrp
+WTkKRoRtHmVnb0d7X7jwfMbnzq7JbGO49IFxNFCUc8PQslmRcWaCsJwiB5oA7vP7
+wyKI29gK78fI21qOAUSbXH6W26vveDsEWT3dig==
+=gvGF
-----END PGP SIGNATURE-----
I can delete InRelease file from cache and do an apt update from some
client and the problem is for a moment solved, but after some time it
is here again. :-/
I'm confused by this strange behavior.
Hi,
the story of inconsistencies of InRelease files moved to a new point.
I discovered, that when I remove the InRelease file from cache then:
1) If I try to download the InRelease file through cache using
curl, the full length of the file is stored into cache storage.
2) If I tried to do an apt update run with the cache as backend,
only a small portion of the file is stored 8-O. Repeated uses of
the InRelease file by APT extends the file in cache by about an
8KB new portions.
ad 1) Test using curl
====================
deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# rm InRelease*
deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# curl http://localhost:9999/debian/dists/bookworm/InRelease >/dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 147k 100 147k 0 0 2773k 0 --:--:-- --:--:-- --:--:-- 2783k
deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# ll InRelease*
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 151073 Mar 26 16:08 InRelease
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 163 Mar 26 16:08 InRelease.head
ad 2) Test using APT
====================
deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# rm InRelease*
deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# apt update
Get:1 http://deb:9999/debian-icz icz-bookworm InRelease [3,938 B]
Hit:2 http://deb:9999/debian bookworm InRelease
Hit:3 http://deb:9999/debian bookworm-updates InRelease
Hit:4 http://deb:9999/debian bookworm-backports InRelease
Get:5 http://deb:9999/security bookworm-security InRelease [48.0 kB]
Fetched 51.9 kB in 1s (76.9 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# ll
total 152
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 7927 Mar 26 15:45 InRelease
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 163 Mar 26 15:45 InRelease.head
Obviously the file returned to APT is the full length of InRelease file,
apt would complain otherwise. But stored is only a small chunk.
ad 2) APT but with removed corresponding APT list file
======================================================
deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# rm InRelease*
deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# rm /var/lib/apt/lists/deb\:9999_debian_dists_bookworm_InRelease
deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# apt update
Get:1 http://deb:9999/debian-icz icz-bookworm InRelease [3,938 B]
Get:2 http://deb:9999/debian bookworm InRelease [151 kB]
Hit:3 http://deb:9999/debian bookworm-updates InRelease
Hit:4 http://deb:9999/debian bookworm-backports InRelease
Hit:5 http://deb:9999/security bookworm-security InRelease
Hit:6 http://deb:9999/debian bookworm/main amd64 Packages
Hit:7 http://deb:9999/debian bookworm/main Translation-en
Hit:8 http://deb:9999/debian bookworm/contrib amd64 Packages
Hit:9 http://deb:9999/debian bookworm/contrib Translation-en
Hit:10 http://deb:9999/debian bookworm/non-free amd64 Packages
Hit:11 http://deb:9999/debian bookworm/non-free Translation-en
Hit:12 http://deb:9999/debian bookworm/non-free-firmware amd64 Packages
Hit:13 http://deb:9999/debian bookworm/non-free-firmware Translation-en
Fetched 155 kB in 1s (163 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
deb:/srv/cache/apt-cacher-ng/debrep/dists/bookworm# ll InRelease*
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 151073 Mar 26 16:13 InRelease
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 163 Mar 26 16:13 InRelease.head
I tried to downgrade apt-cacher-ng to current stable version 3.7.4-1+b2
and this behavior is the same. I conclude from this, that the patch of
Tim Woodall is fine. Many thanks for it. Sorry for my previous update to
this bugreport.
The problem above is completely a new bug.
I will try to capture some packets and maybe I fill another bug against
apt-cacher-ng. I think that more frequently changing InRelease file
(bootworm-updates or bootkworm-security) can change while apt-cacher-ng
downloads the full length of InRelease file by 8KB chunks during
subsequent request for this file and then corrupted version is finally
stored into cache. Maybe. More analyzes needed.
FYI my last two messages are about bug #1003865 probably https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1003865