#999592 apt-cacher-ng: apt update getting "Failed to fetch Sources.xz Hash Sum mismatch" using http proxy

Package:
apt-cacher-ng
Source:
apt-cacher-ng
Description:
caching proxy server for software repositories
Submitter:
David Crook
Date:
2021-11-12 23:09:03 UTC
Severity:
important
#999592#5
Date:
2021-11-12 23:06:11 UTC
From:
To:
Dear Maintainer,

when doing `apt update` with apt-cacher-ng proxy configured in apt, often am getting
error "E: Failed to fetch .../main/source/Sources.xz  Hash Sum mismatch". then apt
exits with an error: "E: Failed to fetch .../main/binary-armhf/Packages.xz E: Some
index files failed to download. They have been ignored, or old ones used instead."

Once it is encountered, any subsequent `apt update` fails with same error, after again
downloading the affected file(s) from the proxy/cache.

Bypassing the proxy (e.g., `sudo apt update -o Acquire::http::proxy=''`) is successful.

Inspecting /var/lib/apt/lists/partial/ there will be a ..._Sources.xz.FAILED. Manually
replacing this file (Sources.xz) with one downloaded directly from source will also
allow the `apt update` to be successful.

re: apt client, occurs on Raspberry Pi OS (both buster and bullseye), Debian 11 (bullseye,
amd64) and Debian testing.

occuring on recent dist versions of apt-cacher-ng (both bullseye and bookworm) running
from VMs on amd64 host.

Doing some investigation, found some interesting results:

- checksum from /var/lib/apt/lists/partial/ .FAILED file matched the reported mismatching
  checksum reported in apt update error

- checksum from using wget to download directly from source matched the expected checksum

- inpecting file in /var/cache/apt-cacher-ng/... on apt-cacher-ng host server matches
  the EXPECTED checksum, meaning that it doesn't seem to be a race

- file lengths are the same (always)

- binary diff of original compressed files shows huge chunks of the expected file vs.
  mismatched file are common. and then large continuous chunks mismatch, and then re-sync
  (are same in both) for huge spans and this repeats.

  - observed this by generating text representation of the binary/compressed generated
    using xxd and side-by-side diffs using sdiff

EXAMPLE Mismatch Error message

E: Failed to fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/main/source/Sources.xz  Hash Sum mismatch
   Hashes of expected file:
    - Filesize:12223504 [weak]
    - SHA256:ed08efd74766e9d76d73f00b200f37ce41e65a51de927114a1e00a3e1f57cc7c
    - SHA1:6399b9bc363c2a0d50ede8c871d92bc6d235c9bb [weak]
    - MD5Sum:3148fbb97feeb4b5fa6b7f1646ab701d [weak]
   Hashes of received file:
    - SHA256:89f42ada72c8c8157b78b8ed173f0feedc9d788e8bcf543ddc447dbda6bf6b26
    - SHA1:d835932deab7abe06bd0ee3edbf3a7fea415b2aa [weak]
    - MD5Sum:af435d409c99ba2950364c21886ea710 [weak]
    - Filesize:12223504 [weak]
   Last modification reported: Thu, 11 Nov 2021 23:20:36 +0000
   Release file created at: Thu, 11 Nov 2021 23:22:43 +0000
E: Failed to fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/main/binary-armhf/Packages.xz
E: Some index files failed to download. They have been ignored, or old ones used instead.