Dear Maintainer,

I have reason to fetch some old files from buster snapshots and have found that
some of the urls result in a HTTP 504 error.

eg:

$ curl -L
https://snapshot.debian.org/archive/debian/20250712T143640Z/extrafiles

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>503 first byte timeout</title>
  </head>
  <body>
    <h1>Error 503 first byte timeout</h1>
    <p>first byte timeout</p>
    <h3>Error 54113</h3>
    <p>Details: cache-lcy-egml8630079-LCY 1793964983 3095964062</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>

Talking direct to port 80 I get:

$curl -L http://snapshot.debian.org/archive/debian/20250712T143640Z/extrafiles
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<title>504 Gateway Timeout</title>
</head><body>
<h1>Gateway Timeout</h1>
<p>The gateway did not receive a timely response
from the upstream server or application.</p>
<hr>
<address>Apache Server at snapshot.debian.org Port 80</address>
</body></html>

I'm aware of the problems (503 and 504 errors). I tried to fix some
yesterday evening (migrating to a materialized view), but it was not
sufficient. Ultimately I think it's because the scraper load (random
access on /dists/ to resolve files) is just too high, coupled with some
inefficient queries that take too long to process. All of our three
beefy machines are hammered and have loads way higher than the amount of
CPU threads we have - with the requests stuck in postgres.

It might still take a few days to resolve, unfortunately.

Kind regards
Philipp Kern

I'm aware of the problems (503 and 504 errors). I tried to fix some
yesterday evening (migrating to a materialized view), but it was not
sufficient. Ultimately I think it's because the scraper load (random
access on /dists/ to resolve files) is just too high, coupled with some
inefficient queries that take too long to process. All of our three
beefy machines are hammered and have loads way higher than the amount of
CPU threads we have - with the requests stuck in postgres.

It might still take a few days to resolve, unfortunately.

Kind regards
Philipp Kern

I think it's fixed now. If there are any unexpected replies (e.g. 404 of
URLs that worked before), please let me know.

Kind regards
Philipp Kern

I think it's fixed now. If there are any unexpected replies (e.g. 404 of
URLs that worked before), please let me know.

Kind regards
Philipp Kern