Existing files under http://archive.debian.org/debian/pool/main/
(bullseye) intermittently return 404 Not Found when a request misses
the CDN cache. The 404 page is generated by the origin Apache, and the
CDN then caches it. The same URL returns 206 or 404 on successive
requests.
Impact
apt in our CI (sbuild, bullseye chroots) fetches bullseye main from
archive.debian.org. Between 2026-10-14 09:21 UTC and at least
2026-10-15 11:44 UTC, 8 builds failed with errors like:
E: Failed to fetch
http://archive.debian.org/debian/pool/main/p/python-wrapt/python3-wrapt_1.12.1-4%2bb1_amd64.deb
404 Not Found [IP: 151.101.66.132 80]
A different file failed in each build. Seven distinct files were
affected: python3-wrapt, python3-pystemd, python3-zipp, libelf1,
libpython3-stdlib, libfile-stripnondeterminism-perl and autotools-dev.
Retries sometimes fail again.
Evidence
All seven files exist: requests that were answered from cache returned
206 every time (455 of 455, all cache hits, from two CI hosts and one
other machine).
To force requests past the cache, I added a unique query string to the
same URL (1-byte range requests, 7 files). 27 of 105 requests returned
404 and 78 returned 206. The 404s came from two CI hosts in different
datacentres and from a third machine, via many different edge servers
(cache-fra-*), and every one of the 7 files returned both 404 and 206.
Sending Cache-Control: no-cache or Pragma: no-cache does not bypass
the cache (35 of 35 requests were cache hits), so the query string is
the only bypass I found.
A 404 response, in full:
HTTP/1.1 404 Not Found
Server: Apache
Content-Type: text/html; charset=iso-8859-1
Content-Length: 304
Age: 0
Date: Thu, 15 Oct 2026 14:45:52 GMT
Via: 1.1 varnish
X-Served-By: cache-fra-eddf8230227-FRA
X-Cache: MISS
X-Cache-Hits: 0
The body is the standard Apache page ending "Apache Server at
archive.debian.org Port 80". The cache missed and the origin answered
404.
The CDN also caches the 404. For 4 unique URLs that had just returned
404, I repeated the same request three times: every repeat returned
404 with X-Cache: HIT and a rising Age (1 to 3 seconds), served by
different edge servers each time. I only tested this on unique
query-string URLs. I did not request the real URLs repeatedly, to
avoid caching a 404 for them. If the same happens for real URLs, a
file that hits a bad back end once could keep returning 404 until the
cached entry expires, which would match the repeat failures we saw.
Reproduction (bash; about 15% of requests return 404, and each request
is a single byte):
url=http://archive.debian.org/debian/pool/main/p/python-wrapt/python3-wrapt_1.12.1-4%2bb1_amd64.deb
for i in $(seq 1 20); do
curl -s -o /dev/null -r 0-0 -w '%{http_code} ' "$url?cb=$RANDOM$RANDOM"
done; echo
Example output: 206 206 206 206 206 206 206 404 206 206 206 404 206
206 206 206 206 206 206 404
Question
This looks like some back-end servers behind archive.debian.org
missing parts of the bullseye pool, or not being in sync with the
others. #1147093 (closed 2026-10-04) covers bullseye-security on
archive.debian.org and mentions back-end servers missing files; I
don't know whether the same servers are involved. Could someone check
that all back ends serve the same bullseye main pool content? I can
run further tests if useful.
Michael Bennett
Staff DevOps Engineer
Release and Deployment
IONOS Cloud GmbH | Revaler Str. 28-31 | 10245 Berlin | Germany
E-Mail: michael.bennett@cloud.ionos.com | Web: cloud.ionos.de
General information about the company can be found here.
A member of the IONOS Group