#997004 Flaky apt install behavior when using apt-transport-mirror config in a container

Package:
apt
Source:
apt
Description:
commandline package manager
Submitter:
丰
Date:
2021-10-22 09:30:03 UTC
Severity:
normal
#997004#5
Date:
2021-10-22 09:26:54 UTC
From:
To:
Dear Maintainer, I observed some flaky apt install behavior when
combining it with apt-transport-mirror in a docker container.

Problem X
----------

I would like an option passed to ``apt-*`` and delay the choice of an
apt network mirror as late as possible. So that when a mirror (which
typically is not under my control) doesn't work, I can change the
option and switch to different mirrors (in CI).

I had a look at ``man apt-transport-mirror``, and think I can combine
the config in it with ``--option Dir::Etc::SourceList`` to get dynamic
mirror choosing during ``apt-*``. It works fine on my Debian 10 host.
But when I tried it in my CI pipeline, the behavious in a (docker)
container was flaky. I tried to reproduced it with simpler steps. The
following was what I found.

Facts
------
The pkg.foo.net below is a static web serving in my local network.

Content of the two files under http://pkg.foo.net/config/debian/
http://mirrors.bfsu.edu.cn/debian-security/
http://mirrors.bfsu.edu.cn/debian/

And the ``00aptCache`` file points ``apt-get`` to my local squid
proxy. It didn't affect the test. I tried with and without it, the
result was the same. It's there just to reduce requests to the mirror
site. ``10.21.0.155`` is my squid HTTP proxy in this example.

The Debian docker images used is the ones released to Dockerhub on 2021-10-12.

Also the ``ca-certificates curl less`` packages are the minimal I need
to reproduce the error. Can't reproduce if I remove any one of them.
But trying to install ``reportbug`` and ``strace`` together also could
result in similar error like the Debian 10 example below. So it may be
related to multiple simultaneous downlown tasks.

Debian 11 container
====================

On my Debian 10 host
deb mirror://pkg.foo.net/config/debian/cn.security.mirror.txt
bullseye-security main contrib non-free
deb mirror://pkg.foo.net/config/debian/cn.mirror.txt bullseye main
contrib non-free
deb mirror://pkg.foo.net/config/debian/cn.mirror.txt
bullseye-backports main contrib non-free
deb mirror://pkg.foo.net/config/debian/cn.mirror.txt bullseye-updates
main contrib non-free

In the docker container:
...
Get:12 mirror://pkg.foo.net/config/debian/cn.security.mirror.txt
bullseye-security/main amd64 openssl amd64 1.1.1k-1+deb11u1 [851 kB]
Get:12 mirror://pkg.foo.net/config/debian/cn.security.mirror.txt
bullseye-security/main amd64 openssl amd64 1.1.1k-1+deb11u1 [851 kB]
Get:12 mirror://pkg.foo.net/config/debian/cn.security.mirror.txt
bullseye-security/main amd64 openssl amd64 1.1.1k-1+deb11u1 [851 kB]
Get:12 mirror://pkg.foo.net/config/debian/cn.security.mirror.txt
bullseye-security/main amd64 openssl amd64 1.1.1k-1+deb11u1 [851 kB]
Ign:12 mirror://pkg.foo.net/config/debian/cn.security.mirror.txt
bullseye-security/main amd64 openssl amd64 1.1.1k-1+deb11u1
72% [Working]

It freezed in the last line. If I repeat the last (following) command
after one failure,

the command worked.

Debian 10 container
====================

On my Debian 10 host
deb mirror://pkg.foo.net/config/debian/cn.security.mirror.txt
buster/updates main contrib non-free
deb mirror://pkg.foo.net/config/debian/cn.mirror.txt buster main
contrib non-free
deb mirror://pkg.foo.net/config/debian/cn.mirror.txt buster-backports
main contrib non-free
deb mirror://pkg.foo.net/config/debian/cn.mirror.txt buster-updates
main contrib non-free

In the docker container
...
Err:17 mirror://pkg.foo.net/config/debian/cn.mirror.txt buster/main
amd64 libpsl5 amd64 0.20.2-2
  File has unexpected size (60472 != 53716). Mirror sync in progress?
[IP: 10.21.0.155 3128]
  Hashes of expected file:
   - SHA256:290fc88e99d21586164d51f8562c3b4c6a3bfabdbb626d91b6541896d76a582b
   - MD5Sum:f7911e3d27efe3c5f81355aa83e9fb07 [weak]
   - Filesize:53716 [weak]
Get:15 mirror://pkg.foo.net/config/debian/cn.mirror.txt buster/main
amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u3 [158 kB]
Err:15 mirror://pkg.foo.net/config/debian/cn.mirror.txt buster/main
amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u3
  File has unexpected size (85016 != 158324). Mirror sync in progress?
[IP: 10.21.0.155 3128]
  Hashes of expected file:
   - SHA256:77eea03f0159284459d2c3be4a0baf40cdb4495f097ae56ca99318bfbd505fc9
   - MD5Sum:8b160539dd1febe0bc62523ae53b5ee5 [weak]
   - Filesize:158324 [weak]
...
54% [Working]

It freezed in the last line. If I repeat the last (following) command
after one failure,

the command worked.

The trick seems to be running the following two line together:

I could reproduce the error if I removed the packages and installed them again.

I installed strace and then reproduced the error. I attached the
relatively large log file in an attatchment.

You can copy above text and paste it to
http://rst.ninjs.org/?theme=nature for nicer rendering.

   * What outcome did you expect instead?

I should work like in a normal Debian environment.