$ sudo aptitude update
...
W: Failed to fetch tor+http://security.debian.org/dists/stable/updates/contrib/i18n/Translation-en: BZ2_bzread: /var/lib/apt/lists/partial/security.debian.org_dists_stable_updates_contrib_i18n_Translation-en.bz2 Read error (-5: DATA_ERROR_MAGIC)
E: Some index files failed to download. They have been ignored, or old ones used instead.

Removing the file in question doesn't help, an error persists.

Hi Mykola,

thanks for the bug report.

Mykola Nikishov wrote:
[…]

In the past, I've seen such deadlocks occassionally. But haven't had
such a case for one or two years now. Back then it usually sufficed to
also remove the according file in /var/lib/apt/lists/ or, in the worst
case, to empty /var/lib/apt/lists/ completely.

Since things like package list downloads are not done by aptitude
itself but by calling functions from libapt-*, I'm reassigning this to
apt.

		Regards, Axel

<150935636765.12708.6154635335479747081.reportbug@think>

Hi,

Download issues in the update stage tend to not be the fault of
aptitude, as it shares the code with apt (via libapt). Whose fault
it is on the other hand…

Assuming you aren't doing something dangerous like disabling security
checks such an error means that the file passed hashsum checks while
downloading, but while trying to extract the file contents a bad file
was encountered where a good file should have been.

That tends to be hardware issues like a slowly failing disk – but there
is of course also the option that the archive had a bad file (less
likely through).


Would be nice if you could attach the failing file (and the Release file
which should sit there as well):
/var/lib/apt/lists/partial/security.debian.org_dists_stable_updates_Release
/var/lib/apt/lists/partial/security.debian.org_dists_stable_updates_contrib_i18n_Translation-en.bz2

So we can have a look as me trying to download what is available now
produces no such error (with apt, but same difference).


Also, if you include the output of a command, please include the entire
output: It can help figuring out what the tool did before reaching the
error as there tend to be multiple ways of arriving at the same error.
Talking back to you a triager will usually "cut down" the output to the
error messages, but that doesn't mean he hasn't read the cut lines.
[…]
[…]

mhhhh. That is the version from testing. Such "Frankendebian" systems
are not very advisable. I guess we have to add the possibility of
a hidden library incompatibility in the list above as a potential
contender.

btw: Yeah for using tor! You know that there are onion services for all
Debian archives available? See onion.debian.org [5nca3wxl33tzlzj5.onion]


Best regards

David Kalnischkies, apt(-transport-tor) developer

I can also confirm this issue with apt. It appears the upstream file does not exist and that may be causing the error.

apt: 1.6~alpha6

E: Failed to fetch https://deb.debian.org/debian-security/dists/stable/updates/contrib/i18n/Translation-en

If you follow that URL you will get a 404. This returns an error to apt and may cause the hash to be null or miscalculated.

i have also experienced this just now, on Debian 10.

emptying /var/lib/apt/lists resolved the problem.

background info: my internet is/was very crappy nowadays, randomly
failing some requests. this may be relevant, because i have never seen
this issue before.

Hi,

I just also ran into this problem. It's on a stretch system, so with an
older apt, but maybe my observations help regardless.

I saw this error running `aptitude update`. It happened with the stretch
security updates translation file for contrib and non-free.

I read that emptying /var/lib/apt/lists might help, so I tried a light
version: I deleted only
security.debian.org_dists_stretch_updates_non-free_i18n_Translation-en
and security.debian.org_dists_stretch_updates_contrib_i18n_Translation-en
from that directory. I had expected these files to be somehow
problematic, so I saved them to restore them later to reproduce the
problem again.

Removing both files indeed made the `aptitude update` complete
succesfully. To my surprise these two files had been recreated, with
exactly the same contents as before. Another `aptitude update` succeeds,
so apparently something else changed (but I did not make a full backup
of the lists directory, unfortunately). So I could not reproduce the
problem by restoring these files manually, as I had planned.

Looking on the server, I see that a
http://security.debian.org/dists/stretch/updates/contrib/i18n/Translation-en.bz2
does exist, so the 404-theory stated previously might not be
accurate.

I also looked for the failing file in /var/lib/apt/lists/partial, but
there were no files in there when I looked.

Sorry I don't have anything more specific, but maybe some of this
helps...

Gr.

Matthijs

I can reproduce this on command, and have two workarounds, (one dirty, one
ugly).

I'm using FAI on Debian 9.13.

So, the problem is, that when apt writes a translations related file, it is
writing a symlink to the extracted .bz2, rather than writing a .bz2 file.

Problem reproduction:

I am using apt-cacher-ng, and am currently pointing it to archive.debian.org.
I am also using apt against a chroot, as part of using the Fully Automated
Installer. This is happening while I'm trying to set up a mirror of debian,
with only the packages I have selected.

as part of that, my apt options are as follows:
```
-o Aptitude::Log=/dev/null -o
Aptitude::CmdLine::Ignore-Trust-Violations=yes -o
APT::Get::AllowUnauthenticated=true -o DPkg::force-conflicts::=yes -o
Dir::State=/usr/fai/mirror/aptcache//var/lib/apt -o
Dir::Log=/usr/fai/mirror/aptcache//var/log/apt -o
Dir::State::extended_states=/usr/fai/mirror/aptcache//var/lib/apt/lists/extended_states
-o Dir::State::status=/usr/fai/mirror/aptcache//statefile -o
APT::Get::Force-Yes=true -o
Dir::Cache=/usr/fai/mirror/aptcache//var/cache/apt -o
Dir::State=/usr/fai/mirror/aptcache//var/cache/apt -o
Dir::Cache::Archives=/usr/fai/mirror/aptcache//var/cache/apt/archives -o
Dir::Etc=/usr/fai/mirror/aptcache//etc/apt/ -o
Dir::State::Lists=/usr/fai/mirror/aptcache//var/lib/apt/lists/
```

The error in question is easy to produce:
```
faiserver:/home/fai# apt-get $aptoptions update
Ign:1 http://127.0.0.1:3142/archive.debian.org/debian stretch InRelease
Get:2 http://127.0.0.1:3142/archive.debian.org/debian stretch Release [118
kB]
Get:3 http://127.0.0.1:3142/archive.debian.org/debian stretch Release.gpg
[3,177 B]
Ign:3 http://127.0.0.1:3142/archive.debian.org/debian stretch Release.gpg
Hit:5 http://127.0.0.1:3142/archive.debian.org/debian stretch/main
Translation-en
Ign:5 http://127.0.0.1:3142/archive.debian.org/debian stretch/main
Translation-en
Hit:4 http://127.0.0.1:3142/archive.debian.org/debian stretch/main i386
Packages
Ign:4 http://127.0.0.1:3142/archive.debian.org/debian stretch/main i386
Packages
Hit:5 http://127.0.0.1:3142/archive.debian.org/debian stretch/main
Translation-en
Ign:5 http://127.0.0.1:3142/archive.debian.org/debian stretch/main
Translation-en
Hit:4 http://127.0.0.1:3142/archive.debian.org/debian stretch/main i386
Packages
Ign:4 http://127.0.0.1:3142/archive.debian.org/debian stretch/main i386
Packages
Err:5 http://127.0.0.1:3142/archive.debian.org/debian stretch/main
Translation-en
  BZ2_bzread:
/usr/fai/mirror/aptcache/var/lib/apt/lists/partial/127.0.0.1:3142_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2
Read error (-5: DATA_ERROR_MAGIC)
Get:4 http://127.0.0.1:3142/archive.debian.org/debian stretch/main i386
Packages [9,587 kB]
Fetched 121 kB in 3s (40.2 kB/s)
Reading package lists... Done
W: GPG error: http://127.0.0.1:3142/archive.debian.org/debian stretch
Release: The following signatures couldn't be verified because the public
key is not available: NO_PUBKEY 04EE7237B7D453EC NO_PUBKEY 648ACFD622F3D138
NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY EF0F382A1A7B6500
W: The repository 'http://127.0.0.1:3142/archive.debian.org/debian stretch
Release' is not signed.
N: Data from such a repository can't be authenticated and is therefore
potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration
details.
E: Failed to fetch
http://127.0.0.1:3142/archive.debian.org/debian/dists/stretch/main/i18n/Translation-en
 BZ2_bzread:
/usr/fai/mirror/aptcache/var/lib/apt/lists/partial/127.0.0.1:3142_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2
Read error (-5: DATA_ERROR_MAGIC)
E: Some index files failed to download. They have been ignored, or old ones
used instead.
faiserver:/home/fai#
```

that's the error in question. A quick inspection of what the file it is
complaining looks like is as follows:
```
faiserver:/home/fai# ls -la
/usr/fai/mirror/aptcache/var/lib/apt/lists/partial/127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2
lrwxrwxrwx 1 root root 122 Jun  8 12:41
/usr/fai/mirror/aptcache/var/lib/apt/lists/partial/127.0.0.1:3142_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2
-> /usr/fai/mirror/aptcache/var/lib/apt/lists/127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en
faiserver:/home/fai# ls -la
/usr/fai/mirror/aptcache/var/lib/apt/lists//127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en
-rw-r--r-- 1 root root 26417876 Jul 18  2020
/usr/fai/mirror/aptcache/var/lib/apt/lists//127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en
faiserver:/home/fai# file
/usr/fai/mirror/aptcache/var/lib/apt/lists//127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en
/usr/fai/mirror/aptcache/var/lib/apt/lists//127.0.0.1:3142_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en:
UTF-8 Unicode text
```

so apt-get has created a symlink to the extracted form of this file, and is
trying to extract it. no good.

There are two workarounds:

Workaround #1(DO NOT DO THIS): nuke the symlink, and download it yourself:
```
faiserver:/home/fai# rm
/usr/fai/mirror/aptcache/var/lib/apt/lists/partial/127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2
faiserver:/home/fai# wget -q
http://127.0.0.1:3142/archive.debian.org/debian/dists/stretch/main/i18n/Translation-en.bz2
-O /usr/fai/mirror/aptcache/var/lib/apt/lists/partial/127.0.0.1:3142
_archive.debian.org_debian_dists_stretch_main_i18n_Translation-en.bz2
```
after this, apt-get update works as expected.

Workaround #2(much more sane): do not try to download the translations at
all!
in your sources.list, after the 'deb' part of an entry, add '[ lang=none
]', to prevent apt-get from performing this step.

on my system, I go from:
```
deb http://127.0.0.1:3142/archive.debian.org/debian stretch main
```
to:
```
deb [ lang = none ] http://127.0.0.1:3142/archive.debian.org/debian stretch
main
```

This prevents apt-get from downloading the translations, and hence prevents
it from storing them incorrectly.



Let me know if any more information would help.