package: security-tracker
x-debbugs-cc: Raphael Geissert <geissert@debian.org>

Hi Raphael,

httpredir as used for security-tracker.debian.org has some problems updating
some Packages files, _sometimes_. IOW: i've seen this working on my laptop,
but not when deployed on soler. The url exists... (or rather it does as Packages.gz,
so it seems apt-update-file has some magic here too...) but it repeatedly
failed like this. (or similar for other targets too)

sectracker@soler:/srv/security-tracker.debian.org/website/secure-testing$ make update-oldstable
set -e ; for rel in wheezy ; do \
        for archive in main contrib non-free ; do \
        python bin/apt-update-file \
http://httpredir.debian.org/debian//dists/$rel/$archive/source/Sources \
                data/packages/${rel}__${archive}_Sources ; \
        done ; \
        for arch in amd64 armel armhf i386 ia64 mips mipsel powerpc s390 s390x sparc kfreebsd-i386 kfreebsd-amd64 ; do \
          for archive in main contrib non-free ; do \
          python bin/apt-update-file \
http://httpredir.debian.org/debian//dists/$rel/$archive/binary-$arch/Packages \
            data/packages/${rel}__${archive}_${arch}_Packages ; \
          done ; \
        done ; \
done
error: in download of 'http://httpredir.debian.org/debian//dists/wheezy/contrib/binary-sparc/Packages' to 'data/packages/wheezy__contrib_sparc_Packages':
Traceback (most recent call last):
  File "bin/apt-update-file", line 29, in <module>
    debian_support.updateFile(sys.argv[1], sys.argv[2])
  File "/srv/security-tracker.debian.org/website/secure-testing/lib/python/debian_support.py", line 346, in updateFile
    return downloadFile(remote, local)
  File "/srv/security-tracker.debian.org/website/secure-testing/lib/python/debian_support.py", line 309, in downloadFile
    lines = downloadGunzipLines(remote + '.gz')
  File "/srv/security-tracker.debian.org/website/secure-testing/lib/python/debian_support.py", line 292, in downloadGunzipLines
    data = urllib2.urlopen(remote, timeout=TIMEOUT)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 469, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 656, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found
Makefile:106: recipe for target 'update-oldstable' failed
make: *** [update-oldstable] Error 1

So for now I have switched the security-tracker to use ftp.de.debian.org again.


cheers,
	Holger

Hi,

The URL does not exist but Apache's configuration make it exist on most
mirrors:
$ curl -I http://ftp.de.debian.org/debian/dists/wheezy/contrib/binary-sparc/Packages
HTTP/1.1 200 OK
Date: Wed, 10 Jun 2015 19:28:42 GMT
Server: Apache/2.2.22 (Debian)
Content-Location: Packages.bz2

This is Apache's "MultiViews" making the URL work. I guess that you are
redirected on a mirror that might not use Apache or where this option is
not enabled.

$ curl -I http://ftp.debian.org/debian/dists/wheezy/contrib/binary-sparc/Packages
HTTP/1.1 404 Not Found
Date: Wed, 10 Jun 2015 19:35:23 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1

I would suggest to always use the Packages.gz / Sources.gz URL directly.

Cheers,

Hi,

That's correct. The uncompressed versions of the files have not been
publicly available for years, so the fix is indeed to use the .gz,
.bz2, or .xz version - only .gz being available for all releases.

Moreover, if a request for a Packages file, uncompressed, from the
redirector does not always return 404 it is a bug that I'd like to
know about. I.e. it has code that explicitly 404s Packages files,
though Sources aren't.

Cheers,