#563737 smarter cleaning of /var/lib/apt/lists

Package:
apt
Source:
apt
Description:
commandline package manager
Submitter:
martin f krafft
Date:
2010-01-20 18:12:02 UTC
Severity:
wishlist
#563737#5
Date:
2010-01-04 22:51:12 UTC
From:
To:
Moving around a lot, I often switch my Debian mirrors.
geo.debian.org is not really an option as that doesn't allow me to
use internal mirrors. Whenever I switch mirrors in sources.list,
however, /var/lib/apt/lists gets cleaned and everything has to be
redownloaded.

Maybe APT could check the *Release files in that directory and
reuse existing files if they're found to be official Debian
Package/Source lists? This would save me having to do the same s///
replace on the directory contents that I do in sources.list, which
is arguably not that much trouble, but it would be nice if APT did
it for me.
#563737#10
Date:
2010-01-06 15:43:30 UTC
From:
To:
Am Dienstag, den 05.01.2010, 11:51 +1300 schrieb martin f krafft:
based on hostname, port and path.

Options for you might be:
a) local apt-cacher-ng serving from localhost:3142/debian/
   - the real source configured in /etc/apt-cacher-ng/backends_debian
b) using ftp.debian.org in sources.list and hacking /etc/hosts to make
   it point to your favorite mirror. Works in most cases.
#563737#15
Date:
2010-01-07 05:05:33 UTC
From:
To:
also sprach Julian Andres Klode <jak@debian.org> [2010.01.07.0443 +1300]:

But there are *_Release files, and I think the heuristic could be:

try:
  for each file in …/lists that would be deleted:
    set a=string before '_dists_'
    set b=string after and including '_dists_'
    set c=extracted suite from $b (e.g. 'lenny')

    if $a_dists_$c_Release matches /^Origin: Debian/:
      if there is a matching sources.list entry:
        set d=extracted base URI up to /dists/ from sources.list
        rename s/$a/$d/
      else
        throw
    else
      throw

catch:
  do regular delete.

That would not prevent apt-cacher-ng from redownloading or me having
to manually prevent that.

There are mirrors that have the repo in other paths than /debian/
#563737#20
Date:
2010-01-20 16:59:21 UTC
From:
To:
Julian Andres Klode <jak@debian.org> writes:

Maybe that is the problem.

I don't change my sources.list all the time but I have multiple debian
mirrors in the list, my local one, a near one and ftp.debian.org. That
way I get the newest files from the best source and out-of-sync
mirrors don't matter (they just make things slower). But, assuming the
mirrors are in sync, every Packages file gets downloaded 3 times now.

So maybe index files should be referenced by their checksum instead of
host,port,path. That way apt would download Packages.gz from my local
mirror and then reuse that file for the near mirror and
ftp.debian.org. It would only download the Release[.gpg] file from the
remote mirrors.

So
/var/lib/apt/lists/ftp.debian.org_debian_dists_sid_main_source_Sources
could become
/var/lib/apt/lists/537f0e9373ce078e6af6cd9a000218c4d4c477fc.


The same goes for downloading source files. Say I have
foo_1.2.orig.tar.gz and foo_1.2-3.dsc on my local mirror and Debian
has foo_1.2-4.dsc. I want apt-get source foo to download
foo_1.2.orig.tar.gz from my local mirror and foo_1.2-4.dsc from
debian. There is no need to download all files of a source from the
same host if the checksum matches.

MfG
        Goswin
#563737#25
Date:
2010-01-20 17:50:41 UTC
From:
To:
As far as I know, this would require patches for
	apt-pkg/deb/debindexfile.{cc,h}
	apt-pkg/deb/debmetaindex.{cc,h}
and probably some others. debIndexFile classes would need a field for the
checksum and e.g. debPackagesIndex::IndexFile needs to return the filename
with checksum.
I haven't looked at it, but we don't have a mapping of hashsums
to URIs as far as I know.
#563737#30
Date:
2010-01-20 18:06:27 UTC
From:
To:
Julian Andres Klode <jak@debian.org> writes:

I just thought of something.

If there is a Release file apt has a simple way to map from the
sources.list entry to the hash of the index file. But what if there is
no Release file? That is still allowed, right?

MfG
        Goswin