Hi!

What do you think about support for zstd compression of Packages files?

The performance increase for compression with zstd instead of xz is
significant.  Speed comparison for doing 5 compressions of the trixie
Packages file of ~54MB on my laptop:

jas@frallan:~$ du -h Packages; time for i in $(seq 1 5); do cat Packages | /usr/bin/time xz > Packages.xz; done; du -h Packages.xz
54M	Packages
13.35user 0.06system 0:06.19elapsed 216%CPU (0avgtext+0avgdata 336528maxresident)k
0inputs+19240outputs (0major+17574minor)pagefaults 0swaps
13.20user 0.05system 0:06.12elapsed 216%CPU (0avgtext+0avgdata 336660maxresident)k
0inputs+19240outputs (0major+17572minor)pagefaults 0swaps
13.22user 0.07system 0:06.21elapsed 213%CPU (0avgtext+0avgdata 335796maxresident)k
0inputs+19240outputs (0major+17379minor)pagefaults 0swaps
13.27user 0.09system 0:06.16elapsed 216%CPU (0avgtext+0avgdata 334456maxresident)k
0inputs+19240outputs (0major+17509minor)pagefaults 0swaps
13.38user 0.06system 0:06.25elapsed 214%CPU (0avgtext+0avgdata 335836maxresident)k
0inputs+19240outputs (0major+17811minor)pagefaults 0swaps

real	0m31,027s
user	1m6,461s
sys	0m0,444s
9,4M	Packages.xz
jas@frallan:~$ du -h Packages; time for i in $(seq 1 5); do cat Packages | /usr/bin/time zstd > Packages.zst; done; du -h Packages.zst
54M	Packages
0.17user 0.03system 0:00.06elapsed 304%CPU (0avgtext+0avgdata 77344maxresident)k
0inputs+24704outputs (0major+2565minor)pagefaults 0swaps
0.13user 0.02system 0:00.04elapsed 340%CPU (0avgtext+0avgdata 75008maxresident)k
0inputs+24704outputs (0major+1920minor)pagefaults 0swaps
0.17user 0.02system 0:00.05elapsed 332%CPU (0avgtext+0avgdata 76868maxresident)k
0inputs+24704outputs (0major+1919minor)pagefaults 0swaps
0.19user 0.02system 0:00.06elapsed 324%CPU (0avgtext+0avgdata 79672maxresident)k
0inputs+24704outputs (0major+2520minor)pagefaults 0swaps
0.14user 0.02system 0:00.05elapsed 322%CPU (0avgtext+0avgdata 75044maxresident)k
0inputs+24704outputs (0major+1911minor)pagefaults 0swaps

real	0m0,323s
user	0m0,826s
sys	0m0,221s
13M	Packages.zst
jas@frallan:~$

That is going from over a minute to below a second!

What matters for users is transfer time and decompression time though.

Transfer time depends on network speed, and sometimes the size
difference of 13M vs 9.4M can be significant.

But let's look at decompression speed:

jas@frallan:~$ du -h Packages.zst; time for i in $(seq 1 10); do cat Packages.zst | /usr/bin/time unzstd > Packages; done; du -h Packages
13M	Packages.zst
0.04user 0.00system 0:00.03elapsed 148%CPU (0avgtext+0avgdata 6700maxresident)k
0inputs+110384outputs (0major+1235minor)pagefaults 0swaps
0.03user 0.01system 0:00.03elapsed 143%CPU (0avgtext+0avgdata 6260maxresident)k
0inputs+110384outputs (0major+1204minor)pagefaults 0swaps
0.02user 0.01system 0:00.03elapsed 143%CPU (0avgtext+0avgdata 6188maxresident)k
0inputs+110384outputs (0major+1203minor)pagefaults 0swaps
0.02user 0.01system 0:00.03elapsed 143%CPU (0avgtext+0avgdata 6312maxresident)k
0inputs+110384outputs (0major+1173minor)pagefaults 0swaps
0.04user 0.00system 0:00.03elapsed 146%CPU (0avgtext+0avgdata 6372maxresident)k
0inputs+110384outputs (0major+1237minor)pagefaults 0swaps
0.03user 0.01system 0:00.03elapsed 142%CPU (0avgtext+0avgdata 6352maxresident)k
0inputs+110384outputs (0major+1207minor)pagefaults 0swaps
0.02user 0.02system 0:00.03elapsed 150%CPU (0avgtext+0avgdata 6692maxresident)k
0inputs+110384outputs (0major+759minor)pagefaults 0swaps
0.02user 0.01system 0:00.03elapsed 146%CPU (0avgtext+0avgdata 6496maxresident)k
0inputs+110384outputs (0major+1207minor)pagefaults 0swaps
0.03user 0.02system 0:00.03elapsed 157%CPU (0avgtext+0avgdata 6664maxresident)k
0inputs+110384outputs (0major+1270minor)pagefaults 0swaps
0.03user 0.01system 0:00.03elapsed 154%CPU (0avgtext+0avgdata 6368maxresident)k
0inputs+110384outputs (0major+1206minor)pagefaults 0swaps

real	0m0,484s
user	0m0,341s
sys	0m0,312s
54M	Packages
jas@frallan:~$ du -h Packages.xz; time for i in $(seq 1 10); do cat Packages.xz | /usr/bin/time unxz > Packages; done; du -h Packages
9,4M	Packages.xz
0.25user 0.03system 0:00.13elapsed 223%CPU (0avgtext+0avgdata 87876maxresident)k
0inputs+110384outputs (0major+3654minor)pagefaults 0swaps
0.25user 0.03system 0:00.12elapsed 226%CPU (0avgtext+0avgdata 87468maxresident)k
0inputs+110384outputs (0major+3653minor)pagefaults 0swaps
0.24user 0.04system 0:00.12elapsed 223%CPU (0avgtext+0avgdata 87396maxresident)k
0inputs+110384outputs (0major+3652minor)pagefaults 0swaps
0.27user 0.01system 0:00.13elapsed 222%CPU (0avgtext+0avgdata 88012maxresident)k
0inputs+110384outputs (0major+4205minor)pagefaults 0swaps
0.28user 0.03system 0:00.14elapsed 217%CPU (0avgtext+0avgdata 88840maxresident)k
0inputs+110384outputs (0major+3655minor)pagefaults 0swaps
0.27user 0.01system 0:00.13elapsed 225%CPU (0avgtext+0avgdata 84768maxresident)k
0inputs+110384outputs (0major+3658minor)pagefaults 0swaps
0.26user 0.03system 0:00.13elapsed 235%CPU (0avgtext+0avgdata 85596maxresident)k
0inputs+110384outputs (0major+4257minor)pagefaults 0swaps
0.26user 0.03system 0:00.12elapsed 230%CPU (0avgtext+0avgdata 88008maxresident)k
0inputs+110384outputs (0major+4257minor)pagefaults 0swaps
0.26user 0.02system 0:00.13elapsed 223%CPU (0avgtext+0avgdata 88828maxresident)k
0inputs+110384outputs (0major+4293minor)pagefaults 0swaps
0.27user 0.03system 0:00.13elapsed 227%CPU (0avgtext+0avgdata 87400maxresident)k
0inputs+110384outputs (0major+4253minor)pagefaults 0swaps

real	0m1,565s
user	0m2,677s
sys	0m0,461s
54M	Packages
jas@frallan:~$

That is three times as fast.

Regardless of performance, having support for this in debootstrapp allows for further real-world testing.

Attached patch has also been submitted as a debootstrap merge request,
for the purpose of having an informed discussion:

https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/148

/Simon

control: affects -1 cdebootstrap
control: affects -1 mmdebstrap
thanks

if debootstrap gains support for this, cdebootstrap and mmdebstrap should get
it as well...

Holger Levsen <holger@layer-acht.org> writes:

That makes sense, although I don't fully understand the relationship
between these three implementations.

It seems mmdebstrap already supports zst.  There are references to zst
throughout the mmdebstrap code, and it claims to rely on apt which
supports Package.zst in archives fine.  Lack of zst support in
debootstrap was the reason that I (reluctantly -- the speed/size
tradeoffs are real) switched from zst to gz (which has properties closer
to zst than xz) for one of my archives.  I could see if replacing
debootstrap with mmdebstrap would solve this for me.  Using both zst and
gz was worse than just supporting gz for me.

As for cdebootstrap, it seems to lag behind on implementing new
features, which is fine, but I had a look at the code now.  Implementing
zst support should be straight forward and can/should be done eventually
if debootstrap adds this, and I hope to suggest a patch eventually.

/Simon

contro: affects -1 - mmdebstrap
thanks

there is none, they just do the same thing.

ok, great, adjusting metadata accordingly.

thanks!