For the past few weeks, I haven’t been able to update my Debian system. I
finally took some time to look into what’s going wrong.

The symptom is that “apt update” just hung forever after fetching all files from
my configured mirrors.

A closer look revealed 100% CPU usage of 1 core in the
/usr/lib/apt/methods/store process, which was writing to files in
/var/lib/apt/lists/partial.

Attaching gdb revealed that store was spending its time in a zstd function:

#0  0x00007f376cc0bf78 in ?? () from /usr/lib/x86_64-linux-gnu/libzstd.so.1
#1  0x00007f376cbcc275 in ?? () from /usr/lib/x86_64-linux-gnu/libzstd.so.1
#2  0x00007f376cbcdedf in ?? () from /usr/lib/x86_64-linux-gnu/libzstd.so.1
#3  0x00007f376cbce461 in ZSTD_compressContinue () from /usr/lib/x86_64-linux-gnu/libzstd.so.1
#4  0x00007f376cbcfe9e in ?? () from /usr/lib/x86_64-linux-gnu/libzstd.so.1
#5  0x00007f376d9874af in ?? () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0
#6  0x00007f376d97ffd1 in FileFd::Write(void const*, unsigned long long) () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0
#7  0x000055afaedf1073 in StoreMethod::Fetch(pkgAcqMethod::FetchItem*) () at include/apt-pkg/fileutl.h:168
#8  0x00007f376d939f83 in pkgAcqMethod::Run(bool) () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0
#9  0x000055afaedf0759 in main () at /usr/include/c++/8/bits/basic_string.h:252
#10 0x00007f376d32e09b in __libc_start_main (main=0x55afaedf06e0 <main>, argc=1, argv=0x7ffe0782f238, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe0782f228) at ../csu/libc-start.c:308
#11 0x000055afaedf083a in _start () at ../methods/store.cc:147

Sure enough, raising the cost of the zstd compression in my apt config fixed the
issue:

% cat /etc/apt/apt.conf.d/99disable-zstd
APT::Compressor::zstd::Cost "999";

I have a few remarks:

1. Why are the lists re-compressed at all?
   (Also note that I set Acquire::GzipIndexes "yes", so I have already signaled
    that I would prefer to accept the lists as they are.)

2. Why is store only using 1 of 16 CPU cores for compression?

3. Should we disable zstd for the time being, given that it turns a 1-minute apt
   update into a many-minute apt update, to the point where I had assumed it was
   hanging?

Thanks,

We'd have to figure out when this started, I guess.

No. You have indicated that you want them stored compressed. Which picks the
cheapest compressor, that's usually lz4.

Yes, it used to mean to not uncompress files and take them directly, but that
changed a few years ago, in 1.2 or so, as lz4 basically has not much overhead,
so it's much faster than keeping xz indexes around.

store is using 1 process per file (up to a limit that's a multiple of your
cores), parallel compression of single files is meh.

It's not used by default. It's unclear to me why this happens to you. Do you have
another config file which sets a lower zstd cost perhaps?

Thanks for the quick reply.

Sorry, didn’t keep track of that.

Ugh. Is there a way to tell apt to not bother re-compressing files ever?

That’s not what I’m observing. For me, precisely 1 store process is
running, and it seems to be handling all files sequentially.
% grep -ri zst /etc/apt
/etc/apt/apt.conf.d/99disable-zstd:APT::Compressor::zstd::Cost "999";
%

For the record,
https://bugs.debian.org/1108032
says in 2025 this feature does not exist, and asks for it.