#779592 [apt] /var/lib/apt/lists/partial/ gets filled by Diff_index file

Package:
apt
Source:
apt
Description:
commandline package manager
Submitter:
Valerio Passini
Date:
2015-03-12 22:33:14 UTC
Severity:
normal
#779592#5
Date:
2015-03-02 19:34:33 UTC
From:
To:

deb http://ftp.it.debian.org/debian/ experimental main contrib non-free;

the directory /var/lib/apt/lists/partial/ is quickly filled by a Diff_index file
growing at a 30MB/s rate until the partition is full. Quite obviously the next
boot is going to fail for the lack of disk space. I can't understand if this
bug it's in the mirror or in apt, but it's quite annoying and should really be
fixed ASAP. Best regards

Valerio
Debian Release: 8.0
  990 unstable        repos.fds-team.de
  990 unstable        ftp.uk.debian.org
  500 testing         ftp.it.debian.org
  500 stable          mi.mirror.garr.it
  500 stable          download.mono-project.com
    1 experimental    ftp.uk.debian.org
--- Package information. ---
Depends                       (Version) | Installed
=======================================-+-=============
libapt-pkg4.12              (>= 0.9.16) | 1.0.9.7
libc6                         (>= 2.15) |
libgcc1                    (>= 1:4.1.1) |
libstdc++6                     (>= 4.9) |
debian-archive-keyring                  |
gnupg                                   |


Package's Recommends field is empty.

Suggests         (Version) | Installed
==========================-+-============
aptitude                   | 0.6.11-1+b1
 OR synaptic               | 0.81.3
 OR wajig                  |
dpkg-dev       (>= 1.17.2) | 1.17.24
apt-doc                    |
python-apt                 | 0.9.3.11
--- Output from package bug script ---
#779592#10
Date:
2015-03-08 11:40:43 UTC
From:
To:
Hi,

Was the file really just called "Diff_index"?

I presume it was:
ftp.it.debian.org_debian_dists_experimental_main_binary-amd64_Packages.IndexDiff
which is the filename for this file:
http://ftp.it.debian.org/debian/dists/experimental/main/binary-amd64/Packages.diff/Index

There isn't much we can do about it at the moment. apt/stretch (not
jessie!) will know (most) filesizes in advance and check that it isn't
getting feed too much, but that is just preventing bad sympthoms to
appear (= full disk), it isn't a solution for the (unknown) initial
problem.

Could be a misbehaving proxy (do you have one?), a misbehaving server,
your ISP (via a misbehaving proxy) or any classic man-in-the-middle
really. Hard to know without details. Can you even reproduce it?


I am not fully closing this bug as fixed in future version just yet
because I would like to understand what is going on here in case we can
do anything about it (further) to prevent this from happening, but I am
downgrading drastically as this isn't a new issue (= always possible in
all versions of apt), apt isn't made unusuable by it, we are not loosing
any data (well, with a fulldisk we potentially are, but that is the bug
of other tools not handling this case) and its not opening a security
hole. So neither of the reasons for 'grave' apply here and hence not
release critical.


Best regards

David Kalnischkies

#779592#19
Date:
2015-03-08 13:23:32 UTC
From:
To:
In data domenica 8 marzo 2015 12:40:43, David Kalnischkies ha scritto:

Hi David,

Thank you for taking care of the bug. You are right it's not always
reproducible, but it consistently happened to me and to understand the root of
this bug would be better for apt. After reporting this bug I removed that
mirror from my source.list and since then it disappeared. I'm trying to
reproduce it now (with no success).
The name I gave to you is correct. To be precise is:

ftp.it.debian.org_debian_dists_testing_contrib_binary-amd64_Packages.diff_Index

so I wrongly wrote the capital letters.
I fully understand why you are changing the bug priority, I supposed that in
the meantime someone else could confirm this bug, but it didn't happen so at
the moment it has affected just my system.
You are making good guesses about the root of this bug and apt could be not
the culprit, anyhow this open bug could be a good starting point for those who
meet similar problems. Best regards

Valerio Passini

#779592#24
Date:
2015-03-12 22:08:55 UTC
From:
To:
Dear Maintainer,

   * What led up to the situation?

apt-get update -o quiet=2 # run from cron-apt

   * What exactly did you do (or not do) that was effective (or
     ineffective)?

deleted /var/lib/apt/lists/partial/snapshot.debian.org_archive_debian_20130710T214032Z_dists_jessie_non-free_binary-i386_Packages.diff_Index was effective

i took a look at it before i deleted it.  the file started with md5sums as
expected, but instead of EOF at the end, there were a series of [presumably
infinite] null bytes (^@ in `less`)

i also find it odd that the filename is different to the non-partial version:
/var/lib/apt/lists/snapshot.debian.org_archive_debian_20130710T214032Z_dists_jessie_non-free_binary-i386_Packages.IndexDiff

some more things as i poke around lists/ and lists/partial/ (not sure if they
are related):

/var/lib/apt/lists/partial/snapshot.debian.org_archive_debian_20130710T214032Z_dists_jessie_non-free_binary-i386_Packages.bz2
is a text file (not a bz2)

there's a lot of .FAILED files in lists/partial/

there's a lot of
ftp.iinet.net.au_pub_debian_debian_dists_experimental_main_i18n_Translation-en.ed.$date-$time.gz
files in lists/ (going back two months)

   * What was the outcome of this action?

deleting the file worked as expected as apt was not running.

   * What outcome did you expect instead?

i expected apt not to write infinite null bytes.