Dear Maintainer,

Apologies for taking so long to track this down... but here we go.

In Kali Linux, between 11 November 2023, and 30 November 2023, the installation time of an emulated arm64 build on amd64 host jumped exponentially.

On 11 November, the time that it took to run this build was 1 hour 31 minutes.
On 30 November, the time that it took to run this build was 5 hours 47 minutes.

Initially, I blamed it on texlive, and even submitted bug 1070150 to report the increase in time that it took to install, however, as someone else pointed out in the bug report, using a minimal testcase of

$ podman run --arch arm64 --rm -it --pull always debian:stable /bin/bash
$ apt update &&
  apt install -y moreutils &&
  apt install -y --download-only python3-minimal
$ time apt install -y python3-minimal | ts

You end up with an install time around 52s (on my machine here, its actually around 1 minute 30 seconds).

Doing the same, but pointing at sid, the time is around 3 minutes 52 seconds.

Since I knew that Kali was affected by this from some time in November, I decided to ty to track it down in Debian, since it was something coming to us indirectly.

I used debootstrap to create chroots from 14 October to 30 November, as our britney is allowed to hold packages for 20 days, after my coworker pointed out that our britney is allowed to hold packages for up to 20 days.  Initially I had only been looking between 08 November and 30 November.  All of them showed roughtly the same time here - 3 minutes 42 seconds, on average.

After a lot of chroots (roughly 100), I finally tracked the issue down to being on the 14th of October, and it looks like the change being python 3.11.5 moving to 3.11.6 is what is causing the issue.  For completeness, I also tested python3.12-minimal to see if it has the same slow speed, and unfortunately it does.

The fastest way to reproduce this is to use the following commands on an amd64 host:

$ debootstrap --arch arm64 --components=main,contrib,non-free,non-free-firmware testing 20231114 https://snapshot.debian.org/archive/debian/20231114T1502172Z/
systemd-nspawn -D 20231114 /bin/bash
# echo "Acquire::Check-Valid-Until false;" > /etc/apt/apt.conf.d/99no-check-valid-until
# apt update
# apt-get install python3-minimal eatmydata moreutils -y --download-only
# apt-get install eatmydata moreutils -y
# time eatmydata apt-get install python3-minimal -y | ts

Then repeat it again, but using 20231113T085432Z instead of the 20231114 url.

$ debootstrap --arch arm64 --components=main,contrib,non-free,non-free-firmware testing 20231113 https://snapshot.debian.org/archive/debian/20231113T085432Z/
# echo "Acquire::Check-Valid-Until false;" > /etc/apt/apt.conf.d/99no-check-valid-until
# apt update
# apt-get install python3-minimal eatmydata moreutils -y --download-only
# apt-get install eatmydata moreutils -y
# time eatmydata apt-get install python3-minimal -y | ts

This slowdown only occurs during emulating arm64 on an amd64 host, and does not occur when emulating armhf, or armel, nor on an arm64 host itself.

I have not tested emulating any other 64bit architectures.

To be clear, this is a full kali linux installation in an arm64 chroot
emulated via qemu on an amd64 host, it was simply meant to explain the
catalyst that caused me to start looking into what happened in Debian
testing around these times.

I realize that I do not include the logs, which show the timing, I had
naively assumed that it would be good enough to give the commands to
see the difference.

As an example, I created the above chroots again, and installed the
package "sublime-music" as an example of a python application which
ends up showing the time difference.
The 20231013 installation took 13m59s, and the 20231014 installation
took 43m35s.

Attached are the log files from both of the chroots running the
installation and piping it to ts e.g. `time eatmydata apt-get install
sublime-music -y | ts` - this is after running `apt-get install
sublime-music -y --download-only` on both, to rule out potential
network issue.

I hope this makes the issue a little clearer?

Dear Maintainer,

I can see similar problems after upgrading from Bookworm to Trixie. I'm bootstrapping an arm64 system on an amd64 host using qemu.

Thanks @Steev for the nice evaluation, this helped to create a small reproducer.

I can see such slowdowns when setting up virtual environment. I don't know whether the mentioned package is the correct target, but I wanted to add my findings to the existing ones, as they are probably related.

Bookworm:
$ sudo debootstrap --arch arm64 --components=main,contrib,non-free,non-free-firmware bookworm bookworm
$ sudo chroot bookworm/
$ apt update
$ apt install python3-venv
$ time python3 -m venv /opt/abc
real    0m10.899s
user    0m10.532s
sys     0m0.337s

Trixie:
$ sudo debootstrap --arch arm64 --components=main,contrib,non-free,non-free-firmware trixie trixie
$ sudo chroot trixie/
$ apt update
$ apt install python3-venv
$ time python3 -m venv /opt/abc
real    1m17.892s
user    1m17.627s
sys     0m0.250s