Dear Maintainer,

Since the update to 3.7.5-1, package import stopped working.

I 'followed the trail' for an example deb:
* its checksums (md5, sha1 and sha512) are correctly recorded in _impkeycache;
* the relevant index file (debrep/dists/trixie/main/binary-amd64/Packages.xz)
  is available, up to date and of course has correct md5 and sha256 checksums;
* the import log says
  "Parsing metadata in debrep/dists/trixie/main/binary-amd64/Packages.xz";
* but the import doesn't happen, the deb file is among the files "left behind"
  (which are counted three times, but that's another matter).

The logs don't seem to contain anything out of the ordinary, even with debug=7
and verboselog=1.  Is there anything I could provide to help diagnosis?

Thanks for maintaining!
-Steven

I installed trixie from scratch with the 13.2 netinst just over
two weeks ago, my second such installation. To save bandwidth,
I copied the trixie packages saved from my first installation
into /var/cache/apt/archives: it's now a 6.7GB .deb collection.
As this laptop runs pretty much 24/7, I decided to install
apt-cacher-ng on it, rather than my usual workaday desktop.

Following the acng guide, I attempted to import my packages,
and got the same symptoms as reported here at #5. The opening
words, "Since the update to 3.7.5-1," gave me hope, though;
so I carried out the following operations:

1. Re-moved the package collection out of /v/c/acng/_import/
   back into a safer location,

2. Removed apt-cacher-ng and its three dependencies that were
   recently installed (/only/ purging the dependencies), but
   see step 8,

3. Manually downloaded bookworm's version of apt-cacher-ng,

4: Attempted to install it, which required libc-ares2 in
   addition as a dependency,

5. Manually downloaded (trixie's) libc-ares2, and installed
   apt-cacher-ng and libc-ares2 together,

6. Moved the package collection back into /v/c/acng/_import/,

7. Import now worked.

I now have 5.2GB under debrep, and 1.7GB left behind in _import,
and the latter seems reasonable with there being 8 old chromiums,
4 old firefoxes, one old libreoffice suite and a linux-image
among the detritus. So,

8. apt-get upgrade   which upgraded apt-cacher-ng. I left
   libc-ares2 installed though it's currently redundant.
   (Removing the three dependencies at step 2 was unnecessary.
   Had this procedure not worked, I was going to remove acng
   for the time being, and use a USB stick instead.)

As this bug has been open for 17 months, I hope this post
might be of use to somebody.

Cheers,
David.

(Adding Tim Woodall in CC as he appears to have already started a patch
collection for ACNG.)

I think I have found the root cause preventing the import of packages
from the local file system.

When running a package import operation, ACNG computes cryptographic
hashes ("checksums") for each package encountered. In the current
(broken) version, it uses MD5, SHA1 and SHA512.

After computing these hashes, ACNG scans the index files. When
processing the content of Packages.xz, it keeps the last cryptographic
hash specified for each package record in that file. At present (Debian
Trixie), the cryptographic hashes found in Packages.xz are MD5 and
SHA256, with the SHA256 usually coming after the MD5 hash, so SHA256 is
the (only!) hash ACNG will remember for the given package.

Unfortunately, that means that there is no common cryptographic hash
ACNG could use to match a package from the local file system to a record
in Packages.xz, causing imports to fail.

This behavior was introduced when support for parsing SHA256 hashes was
added to the processing of Packages.xz files. In ACNG's GitLab
repository, the corresponding commit is ccdc24ae, "Part of #1026848
(SHA256 checksum support)" [1]. Prior to this commit, only the MD5 hash
would have been parsed from the Packages.xz file, which would ensure a
match with the hashes computed for the package files in the local file
system.

(I also got the impression that this change of adding SHA256 does not
really integrate nicely into the existing code base, which appears to
assume only a single cryptographic hash is present in each record from
Packages.xz. Support for multiple hashes should probably be implemented
in a slightly different way than commit ccdc24ae does - not overriding
the single hash with a new value, but creating a list of hash values or
similar.)

My current workaround, which allows me to use ACNG as documented, is to
add the calculation of an SHA256 hash when scanning local package files.
This appears to work well for my purposes (I was able to import a batch
of 3k packages using it); however, I think a proper fix allowing package
records to store all provided hash values (instead of just the last one
encountered) would be the better long-term fix.

I have attached the patch to this mail; in case it gets lost, the
important part of it is also reproduced below.

(Apologies in case the format of this mail's text is broken; I am
fighting with the settings of a new instance of Thunderbird, and cannot
get it to properly produce lines of limited length.)

---
--- apt-cacher-ng-3.7.5.orig/src/pkgimport.cc
+++ apt-cacher-ng-3.7.5/src/pkgimport.cc
@@ -73,7 +73,11 @@ bool pkgimport::ProcessRegular(const mst
               if(m_precachedList.find(sPath)!=m_precachedList.end())
                       return true; // have that already, somewhere...
 
-              for (CSTYPES ctp : { CSTYPE_MD5, CSTYPE_SHA1,
CSTYPE_SHA512 } )
+              // BUG: When scanning Packages.xz, we only keep the last
checksum encountered for
+              //      a given package. At present, this is typically
SHA256. We therefore must
+              //      include SHA256 checksums in the list here as
well, otherwise we will never
+              //      see a successful hit.
+              for (CSTYPES ctp : { CSTYPE_MD5, CSTYPE_SHA1,
CSTYPE_SHA256, CSTYPE_SHA512 } )
               {
 
                       // get the most likely requested contents id

Kind regards,
Alexander Schulze

----
[1]
https://salsa.debian.org/blade/apt-cacher-ng/-/commit/ccdc24ae5247d56e6cab1ea7c8df8e0e5e04ec85