Hi,

for the past few weeks I've been having weird errors in apt update,
related to symlinks in /var/lib/apt/lists/partial.  E.g. right now I
have:

lrwxrwxrwx 1 root root     81 Jan 14 10:41 deb.debian.org_debian_dists_stretch_main_binary-amd64_Packages.gz -> /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_main_binary-amd64_Packages
lrwxrwxrwx 1 root root     81 Jan 23 14:16 deb.debian.org_debian_dists_stretch_main_binary-amd64_Packages.xz -> /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_main_binary-amd64_Packages
lrwxrwxrwx 1 root root     89 Jan 14 10:41 deb.debian.org_debian_dists_stretch_main_dep11_Components-amd64.yml.gz -> /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_main_dep11_Components-amd64.yml.gz
lrwxrwxrwx 1 root root     89 Feb 11 11:02 deb.debian.org_debian_dists_stretch_main_dep11_Components-amd64.yml.xz -> /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_main_dep11_Components-amd64.yml.gz
lrwxrwxrwx 1 root root     84 Jan 14 10:41 deb.debian.org_debian_dists_stretch_main_dep11_icons-64x64.tar.gz -> /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_main_dep11_icons-64x64.tar.gz
lrwxrwxrwx 1 root root     79 Feb 11 11:02 deb.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2 -> /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_main_i18n_Translation-en
lrwxrwxrwx 1 root root     74 Jan 14 10:41 deb.debian.org_debian_dists_stretch_main_source_Sources.gz -> /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_main_source_Sources
lrwxrwxrwx 1 root root     74 Jan 23 14:16 deb.debian.org_debian_dists_stretch_main_source_Sources.xz -> /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_main_source_Sources
-rw-r--r-- 1 root root 117945 Jan 23 20:14 deb.debian.org_debian_dists_stretch_Release
-rw-r--r-- 1 root root   2434 Jan 23 20:26 deb.debian.org_debian_dists_stretch_Release.gpg

These things are unexpected (I'd expect the directory to be empty), and
at least the symlinks from compressed to uncompressed seem to confuse
apt.

apt update shows errors such as
  BZ2_bzread: /var/lib/apt/lists/partial/deb.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2 Read error (-5: DATA_ERROR_MAGIC)

  Could not open file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_stretch_main_source_Sources.xz - open (13: Permission denied) [IP: 2a04:4e42:1d::204 80]

which I guess are side effects of these bad links.

Cheers,
Julien

Hello,

I'm observing the same on Bullseye 11.6. This is an old i386
installation upgraded all the way from Etch 4.0 r3, but no complete
dist-upgrade.

Apt is version 1.4.10 from Stretch.

If I move those problematic symlinks out of the way, they are
re-created.

The bug breaks other things relying on "apt update" exiting without any
error (i.e. "pihole -up" here).

Christoph Franzen

Hello again,

I've looked at the names of those symlinks and other remaining files in
the "partial" directory, and found something hopefully leading to a
solution.

The file names were all related to Debian "Jessie".

The Pi-hole application I already mentioned, insisted (wrongfully) on
the PHP version provided there, this is the reason why I kept it in the
sources list.

In particular, the following three lines cause the problem on my machine:
---8<-------------------------------------
deb [ trusted=yes check-valid-until=no ] http://archive.debian.org/debian/ jessie main contrib non-free
deb [ trusted=yes check-valid-until=no ] http://archive.debian.org/debian/ jessie-backports main contrib non-free
deb [ trusted=yes check-valid-until=no ] http://archive.debian.org/debian/ jessie-backports-sloppy main contrib non-free
---8<-------------------------------------
Commenting them out solves the problem,
it occures again after re-activating those,
and calling "apt(-get) update" afterwards.

This does NOT happen with "Stretch" (tested here on the same machine),
which is also archived.

The aforementioned problems of pi-hole aren't the problems of Debian,
of course; however, there are legitimate reasons for keeping something
from such an old release around, and the bug will occur no matter
whether another application misbehaves, so I think that should be fixed.

"check-valid-until=no" "removes being an error" in Apt that "Jessie is
too old", "trusted=yes" eliminates the error that all signing keys have
expired.

But then, this bug creates the symlinks leading to the error message
when bzip2 tries to decompress actually uncompressed files. So
"apt(-get) update" exits with a non-zero value, which routinely breaks
scripts, despite of the problem being mostly harmless if you are
updating/installing manually.

The following are the "remainders" I've found in my "partial" directory
whenever the above 3 lines are present in sources.list:
---8<-------------------------------------
-rw-r--r-- 1 root root  148222 23. Jun 2018  archive.debian.org_debian_dists_jessie_Release
-rw-r--r-- 1 root root    2420 23. Jun 2018  archive.debian.org_debian_dists_jessie_Release.gpg
-rw-r--r-- 1 root root 9098100 16. Feb 2019  archive.debian.org_debian_dists_jessie_main_binary-amd64_Packages.gz
-rw-r--r-- 1 root root 9100355 16. Feb 2019  archive.debian.org_debian_dists_jessie_main_binary-i386_Packages.gz
lrwxrwxrwx 1 root root      83  6. Nov 03:59 archive.debian.org_debian_dists_jessie_main_binary-i386_Packages.xz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_main_binary-i386_Packages
lrwxrwxrwx 1 root root      82  6. Nov 03:59 archive.debian.org_debian_dists_jessie_main_i18n_Translation-en.bz2 -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_main_i18n_Translation-en
lrwxrwxrwx 1 root root      87  6. Nov 03:59 archive.debian.org_debian_dists_jessie_main_i18n_Translation-de%5fDE.bz2 -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_main_i18n_Translation-de%5fDE
lrwxrwxrwx 1 root root      84  6. Nov 03:59 archive.debian.org_debian_dists_jessie_main_binary-amd64_Packages.xz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_main_binary-amd64_Packages
lrwxrwxrwx 1 root root      82  6. Nov 03:59 archive.debian.org_debian_dists_jessie_main_i18n_Translation-de.bz2 -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_main_i18n_Translation-de
lrwxrwxrwx 1 root root      80  6. Nov 03:59 archive.debian.org_debian_dists_jessie_main_Contents-i386.gz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_main_Contents-i386.lz4
lrwxrwxrwx 1 root root      81  6. Nov 03:59 archive.debian.org_debian_dists_jessie_main_Contents-amd64.gz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_main_Contents-amd64.lz4
lrwxrwxrwx 1 root root      86  6. Nov 03:59 archive.debian.org_debian_dists_jessie_contrib_binary-i386_Packages.xz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_contrib_binary-i386_Packages
lrwxrwxrwx 1 root root      85  6. Nov 03:59 archive.debian.org_debian_dists_jessie_contrib_i18n_Translation-en.bz2 -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_contrib_i18n_Translation-en
lrwxrwxrwx 1 root root      84  6. Nov 03:59 archive.debian.org_debian_dists_jessie_contrib_Contents-amd64.gz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_contrib_Contents-amd64.lz4
lrwxrwxrwx 1 root root      87  6. Nov 03:59 archive.debian.org_debian_dists_jessie_contrib_binary-amd64_Packages.xz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_contrib_binary-amd64_Packages
lrwxrwxrwx 1 root root      83  6. Nov 03:59 archive.debian.org_debian_dists_jessie_contrib_Contents-i386.gz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_contrib_Contents-i386.lz4
lrwxrwxrwx 1 root root      87  6. Nov 03:59 archive.debian.org_debian_dists_jessie_non-free_binary-i386_Packages.xz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_non-free_binary-i386_Packages
lrwxrwxrwx 1 root root      88  6. Nov 03:59 archive.debian.org_debian_dists_jessie_non-free_binary-amd64_Packages.xz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_non-free_binary-amd64_Packages
lrwxrwxrwx 1 root root      86  6. Nov 03:59 archive.debian.org_debian_dists_jessie_non-free_i18n_Translation-en.bz2 -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_non-free_i18n_Translation-en
lrwxrwxrwx 1 root root      84  6. Nov 03:59 archive.debian.org_debian_dists_jessie_non-free_Contents-i386.gz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_non-free_Contents-i386.lz4
lrwxrwxrwx 1 root root      85  6. Nov 03:59 archive.debian.org_debian_dists_jessie_non-free_Contents-amd64.gz -> /var/lib/apt/lists/archive.debian.org_debian_dists_jessie_non-free_Contents-amd64.lz4
---8<-------------------------------------

As you can see, ".xz" and ".bz2" extensions link to uncompressed files,
while ".gz" extensions link to ".lz4" files.

Because of "ls -lArt" the files above from Nov. 6th should be in creation order.

The locale settings might also be important
(look at the "… i18n … %5fDE …" line above):
---8<-------------------------------------
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=
---8<-------------------------------------

Christoph Franzen