* Package name    : zlib-ng
  Version         : 2.0.5
  Upstream Author : zlib-ng Team
* URL             : http://github.com/zlib-ng/zlib-ng
* License         : Zlib, Zlib-RFC, CC-BY-3.0, CC-BY-4.0, Public-Domain
  Programming Lang: C
  Description     : optimized zlib compression library

 zlib-ng is a fork of the zlib library implementing the deflate compression
 method found in gzip and PKZIP.
 .
 It includes and consolidates many optimizations found in alternative forks,
 that have not been merged in the official zlib library.


I just discovered this project, and started packaging [P] it to be able
to play with a dpkg branch switching its zlib support to zlib-ng. The
speed up is quite significant, for example when packing the 0ad-data,
on my system it takes 5m50s~ with zlib and 4m30s~ with zlib-ng.

  [P] https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/

The project has a compat mode which generates API "compatible" zlib
replacement libraries, but unfortunately it is stated not to guarantee
to be ABI compatible, so no compat packages or similar could be
produced right now as that could potentially break other packages. I've
filed a report [S] upstream to request a more usable shim instead.

  [S] <https://github.com/zlib-ng/zlib-ng/issues/1081>


I'm still pondering whether to upload this, although the packing is
already done, but w/o the above mentioned shim its utility seems
restricted as most upstream projects use it via its compat mode,
instead of with its native API. But if that happens, I think it would
make sense to upload, as it's currently being embedded in several
upstream projects and even if dpkg would not switch to it, it would
still help with removing embedded code copies, and speeding up other
packages. Or make other RPFs such as #901490 (an alternative fork)
unnecessary.

Thanks,
Guillem

not that i expect you to have run extensive benchmarks or
anything, but how do you feel this compares to libdeflate? the
few comparisons i've seen suggest that they are (or at least
were) pretty much a wash, performance-wise.

not that i expect you to have run extensive benchmarks or
anything, but how do you feel this compares to libdeflate? the
few comparisons i've seen suggest that they are (or at least
were) pretty much a wash, performance-wise.

I don't think it matters, because libdeflate does not support a
streaming API anyway, so it's of no use in many situations where
large files or streaming content needs to be dealt with. It also
does not provide a compatible zlib API, which means it cannot be
easily migrated to.

I think when I first noticed libdeflate being uploaded to Debian, I
took a look to also play with it for dpkg, but the above problems
meant that never got anywhere. From the benchmarks I've seen
libdeflate might be a bit faster, but then it would require tons more
memory to handle equivalent inputs and outputs.

The reasons I found the zlib-ng alternative interesting were because:

 - the Intel fork is not going to be adding non-Intel optimizations,
 - the Cloudflare fork does not look very lively,
 - the zlib upstream does not look very active, and apparently outright
   refuses contributions to the main code, and only accepts them into
   its contrib/ directory,

OTOH the zlib-ng looks very lively, accepts contributions, has had its
code modernized and cleaned up, and seems to be performing rather well
in comparison on multiple architectures.

Thanks,
Guillem

The alternative fork is the big issue here - there's several different
zlib projects active which don't seem to be working towards each other
and there's none of them that is clearly going to be the one that will
become the default in future if it's not the original zlib.  We should
just pick a new zlib if we want a new zlib rather than making users go
and figure out which they want.

The alternative fork is the big issue here - there's several different
zlib projects active which don't seem to be working towards each other
and there's none of them that is clearly going to be the one that will
become the default in future if it's not the original zlib.  We should
just pick a new zlib if we want a new zlib rather than making users go
and figure out which they want.

Hi!

Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:

https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=19382bf
    Initial packaging

diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..e4069a9
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.0.5-1) UNRELEASED; urgency=medium
+
+  * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <guillem@debian.org>  Mon, 20 Dec 2021 23:33:25 +0100

Hello,

I see you recently pushed some code into git, do you plan to push the
code also into Debian itself?

Thank you

David

Hi!

Given that it conflicts with the zlib package, and I'm not sure it
makes sense to upload just the zlib-ng specific library without the
compatibility shims, I'm currently not planning on doing so, no. But
I should probably discuss this further with Mark Brown.

Thanks,
Guillem

JFTR, Fedora is planning to transition to Zlib-ng

https://fedoraproject.org/wiki/Changes/ZlibNGTransition

Best regards,
Nelson

Hi!

Ah, thanks! I had in my mind getting back to this ITP, given that the
zlib-ng project has continued to gain traction and seems to have
consolidated most of the other forks around it.

So I'll draft another mail to Mark and probably to debian-devel to
discuss this.

Thanks,
Guillem

Hi

Some rust code I'm trying to package seem to depend on zlib-ng, this
can maybe be patched out, but I thought I should ask about status on
this packaging effort before I undertake that work.

best regards
Alexander Kjäll

Hi,

Do you want me to join your efforts? This looks interrestig. I may have
time ;)

Sebastian

Hello,

I think it already makes sense to push zlib-ng and let it co-exist with
zlib since you can port your software directly to the zlib-ng, which I'm
currently doing for Mesa3D.

I dropped the zlib-ng sources into https://salsa.debian.org/dh/zlib-ng
feel free to force push there any Debian relevant changes.

After introducing the zlib-ng, we could continue to the second phase
migrating software still relying on zlib to zlib-ng compat layer.

What do you think?

David

On Mon, 6 Nov 2023 21:44:05 +0100 Sebastian Andrzej Siewior
<sebastian@breakpoint.cc> wrote:

 > On 2023-10-25 23:17:06 [+0200], Guillem Jover wrote:
 > > Hi!
 > Hi,
 >
 > > Ah, thanks! I had in my mind getting back to this ITP, given that the
 > > zlib-ng project has continued to gain traction and seems to have
 > > consolidated most of the other forks around it.
 > >
 > > So I'll draft another mail to Mark and probably to debian-devel to
 > > discuss this.
 >
 > Do you want me to join your efforts? This looks interrestig. I may have
 > time ;)
 >
 > > Thanks,
 > > Guillem
 >
 > Sebastian
 >
 >

Hi!

Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:

https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=7a95e36
    Initial packaging

diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..aa56259
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.1.6-1) UNRELEASED; urgency=medium
+
+  * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <guillem@debian.org>  Thu, 02 May 2024 18:48:28 +0200

Hi!

Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:

https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=e0eed81
    Initial packaging

diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..aa56259
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.1.6-1) UNRELEASED; urgency=medium
+
+  * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <guillem@debian.org>  Thu, 02 May 2024 18:48:28 +0200

A recurrning question with the zlib package in Debian is interest in the
various alternative zlib implementations that are out there.  There was
a long period where upstream zlib development seemed very stalled,
during that period people who wanted improvements started forking their
own projects.  The main ones I'm aware of are:

   zlib-ng:  https://github.com/zlib-ng/zlib-ng
   chromium: https://chromium.googlesource.com/chromium/src/third_party/zlib

zlib-ng seems pretty healthy, the chromium fork is less generally active
but is used by Chrome/ChromeOS which is a big userbase.

The main thing people seem excited about is performance work for modern
platforms though both projects have been doing other work on the code.
Unfortunately it looks like there is little interest in bringing these
forks together in spite of zlib's upstream development having picked up
a bit again.

Fedora did a transition to zlib-ng relatively recently, in version 40:

https://fedoraproject.org/wiki/Changes/ZlibNGTransition
https://packages.fedoraproject.org/pkgs/zlib/zlib/
https://packages.fedoraproject.org/pkgs/zlib-ng/zlib-ng/

In the past I've pushed back on doing anything here since zlib is
essential and it seemed better to be consistent over the ecosystem than
to use a more niche implementation, and some of the early optimisation
efforts had not worked well on CPUs other than their immediate targets.
However given the user feedback and looking at the Fedora experience I
think it might be time to reevaluate that.

Obviously it's far too late to do anything with the default for trixie,
we might want to evaluate doing something after the release but for now
it's too late.

There's been some ongoing discussion (which sadly I wasn't looped into
most of) of zlib-ng in WNPP:

https://bugs.debian.org/1002056

with some packaging done, but not AIUI building the zlib compatible ABI
for zlib-ng yet which would allow it to be used as a replacement.
Adding support for the compatible ABI allowing it to be an alternative
for standard zlib seems to me like an obvious step we could take, it
would need a lot of care given that zlib is essentially but would let
people get zlib-ng if they wanted, and if there are problems it can be
held in unstable (or experimental) to avoid impact on trixie.  This
would allow people to kick the tires.

Does anyone have thoughts on this?

Hi!

Great! I'm happy to hear that.

Personally I don't think it's too late, there should be several months
until the freeze, and I think if we wanted to switch we could perhaps
do a staged transition and see how it goes and only do the final
replacement if everything seems fine.

Sorry, I've been meaning to bring this up again to your attention,
given that as you mention zlib-ng has seen steady development and
buy in from the community at large. But at the same time, I've been
both a bit reluctant to upload anything to avoid the impression of
some kind of attempt to a hostile takeover, and to bring this up to
you as from your earlier push back I thought that would require some
(perhaps) exceeding changed circumstances. But given your mail, I'm
happy to work on this again and start with say uploading some initial
stuff into experimental for example, after this thread settles a bit?

(I'll start by refreshing the packaging first though.)

Personally, I think fully migrating from zlib to zlib-ng would sound
great (even for trixie), but I guess we can take it slow if you do not
feel confident or have concerns over this.

Also if you'd prefer to take over the zlib-ng ITP, as a continuation
of zlib, that'd seem fine with me too.

Thanks,
Guillem

Hi!

Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:

https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=c8d9119
    Initial packaging

diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..1c25ecd
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.2.2-1) UNRELEASED; urgency=medium
+
+  * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <guillem@debian.org>  Tue, 24 Sep 2024 18:18:29 +0200

Hi!

Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:

https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=9decf65
    Initial packaging

diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..1c25ecd
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.2.2-1) UNRELEASED; urgency=medium
+
+  * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <guillem@debian.org>  Tue, 24 Sep 2024 18:18:29 +0200

* Guillem Jover <guillem@debian.org> [2024-09-24 17:45]:
[...]

As using an alternative zlib implementation could impact Reproducible Builds
[1], I would recommend taking that into consideration before deciding on this
kind of change.

- Fay

[1] https://lists.reproducible-builds.org/pipermail/rb-general/2024-September/003526.html

Hi!

Ah, this is related to something I wanted to mention too and forgot.

I don't think the specific case you mention is in itself a concern for
Debian, because we only guarantee reproducibility given the same inputs,
which includes the set of packages and their versions that were used
when building the binaries. So if there was a switch, those would end up
being recorded as well, and used when reproducing the outputs. And this
could also happen with a newer version of zlib itself.

The problem though is, that because the compressed stream is going to
change, that can make certain test suites fail if we perform this
switch, which I think would be the main fallout that we'd see from
this and would need manual fixing, although I assume Fedora has probably
handled most of these already. For example when I added explicit
zlib-ng support to dpkg, I had to fix its test suite to parametrize
sizes for test artifacts.

I think it would be pretty easy to at least see the extent of this
fallout by performing a mass rebuild for packages build-depending
on zlib1g-dev with a zlib-ng version.

Thanks,
Guillem

As someone who recently tested a rust port of zlib-ng, another factor to
take into account is that while zlib-ng is faster, it also looks like it
compresses less at the same compression level.
Using higher compression levels with the same compression rate is also
usually faster too, but without touching the compression levels, you
end up with something that compresses in less time, but also with a
bigger output.

Mike

We do OTOH package more software than most distros on more architectures
so we got a lot more exposure for testing coverage, and the revert would
involve switching the entire implementation which complicates things a
bit compared to a risky patch within a package.  I'm not totally
opposed, and if everything goes smoothly we could definitely implement
it within the timeframe, but it feels like an impactful change to
introduce now not having considered it sooner.

Sure.

I'm fine with you carrying on with it (actually there is some slight
non-technical complication for me with doing it myself), or we could
also consider a packaging team.  I think there was some other interest
in helping out but ICBW.  If you're packaging it I'm also more confident
in letting you worry about how risky it is to transition and deal with
any fallout!  :P

I guess this is also a risk for zlib upgrades, seems a bit fragile.

* Guillem Jover <guillem@debian.org> [2024-09-25 01:55]:

Whilst it indeed may not affect the reproducibility guarantees for Debian
packages themselves, it does affect being able to use a Debian system for
Reproducible Builds of other software for which the reference artefacts were
built with regular zlib and thus can no longer be reproduced on Debian if that
uses a different zlib implementation (so far I've only encountered the reverse,
which seems relatively rare -- for now).

For example, ZIP files or Android APKs built on a Debian system will have a
different compressed stream, like the test files you mention.  Which will likely
break Reproducible Builds tooling like apksigcopier [1] and
reproducible-apk-tools [2].

AFAIK all rebuilders (including my own [3]) for Android APKs use Debian base
systems, so this could cause quite a bit of breakage for Reproducible Builds
within that ecosystem, which is something I would like to avoid (or at least
have a decent workaround for -- e.g. being able to easily choose between
multiple zlib implementations during runtime in my Python tooling would be
great).

As you point out, we've been lucky that zlib has remained backwards-compatible
for a long time (even though it doesn't provide any guarantees of that AFAIK).
Which also makes me wonder how much more likely zlib-ng might be to produce
different compressed streams between different versions or using different
hardware (configurations).

There might also be issues with reproducibility of Debian packages themselves if
e.g. zlib-ng output can differ on different hardware (e.g. number of cores) even
with an otherwise identical build environment.  At the very least I think it
would be good to know how all this could be affected (and how likely things are
to remain as stable as zlib has been so far) before making a decision to switch.

- Fay

[1] https://tracker.debian.org/pkg/apksigcopier
[2] https://github.com/obfusk/reproducible-apk-tools
[3] https://github.com/obfusk/rbtlog

wouldn't it work to compare the decompressed stream? Is an identical ZIP
file a requirement?

I don't know at this time. Maybe we could throw it into exp first and
evaluate the situtation.

Sebastian

* Sebastian Andrzej Siewior <sebastian@breakpoint.cc> [2024-10-03 22:03]:

By definition a Reproducible Build means a bit-by-bit identical APK, including
the signature (which is why I built a tool to extract an existing signature and
use it as a build input instead of the private key).  Which means you need
identical compressed data for Reproducible Builds.

Having identical uncompressed data gets you pretty close to the goals of RB, but
unpacking and/or skipping over signatures is very very hard to get right and
simply cannot provide the same guarantees as having two bitwise identical files.

And it's impossible to create an APK you can actually install if it's not
bit-by-bit identical as the signature would not be valid otherwise.  So yes,
unfortunately an identical ZIP file is a requirement and comparing the
decompressed stream not an option, which is why this kind of change is not
something we can just consider an implementation detail or work around.

I wrote more about the very messy situation Fedora's switch to zlib-ng already
created for Android Reproducible Builds [1].  Which likely would have broken a
lot more reproducible Android apps already if Fedora's OpenJDK packages linked
against the system zlib like Debian's OpenJDK packages do (instead of using an
embedded copy of regular zlib).

- Fay

[1] https://lists.reproducible-builds.org/pipermail/rb-general/2024-September/003547.html

compression [1] if built against zlib-ng.
It's highly likely that nginx is affected too.

[1] https://t.me/angie_support/4205

пт, 4 окт. 2024 г. в 03:13, Fay Stegerman <flx@obfusk.net>:

Hi!

Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:

https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=2815593
    Initial packaging

diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..1c25ecd
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.2.2-1) UNRELEASED; urgency=medium
+
+  * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <guillem@debian.org>  Tue, 24 Sep 2024 18:18:29 +0200

Hi!

[ I'll try to summarize the current discussion and status, what might
  be blockers, and a potential incremental way forward. ]

True, also two months have passed since (that's on me!). At this time,
I'm now not sure whether it is feasible to consider such a switch, even
if there was agreement to do it. As it is, I think there are too many
unknowns!

I did that, and the current WIP zlib-ng packaging provides now two
builds, one with the new native zng_* API and another (tentatively)
with the compat API/ABI one in libz-dev and libz1 binary packages.

I've tentatively chosen those package names for the compat libraries
to avoid having to go through NEW multiple times (with the assumption
that we'd either go ahead with the switch or the packages could then
simply be dropped). I think this should initially only be uploaded to
experimental, to avoid getting packages built with either zlib or
zlib-ng. But depending on the outcome of this discussion, I think other
(probably better) options would be to perhaps name the compat packages
something like libz-ng-compat*, or drop them completely?

WIP package at <https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/>.

Ok, so after the feedback on this thread, and Sebastian asking how we
can proceed, here are the concerns brought on this thread, along my
own and things I think we need to check or consider:

  * There were concerns (from Fay) about whether given same input the
    output changes per arch or hw setup, we'd need to check this; I'd
    expect this not to be the case for different arches, but it might
    be an issue with number of cores for example, but if either is true
    this would be a serious blocker.
  * I've had concerns both about providing the zlib compat API and the
    native zlib-ng API in sid, and then getting a mess of packages
    linking against (true) zlib and against (native) zlib-ng, or
    packages relying on specific behaviors from either and breaking
    when switching from (true) zlib to zlib-ng-compat or vice versa,
    for example.
  * There were concerns (from Fay) about the output stream changing due
    to a potential implementation switch and that affecting external
    reproducibility. Personally I think while I can see how this is
    annoying for the involved parties, it's part of the "you need
    the same tools to generate the same output" premise that we also
    assume in Debian. I guess keeping both implementations around
    indefinitely, I think, would make this less of an issue, with the
    potential drawbacks mentioned in the previous point.
  * There was a concern (from Konstantin) about at least one known
    upstream (Angie) misbehaving with zlib-ng generated streams.
  * There were concerns (from Mark) that even though projects like
    Fedora have done such switch, we have way more packages and
    architectures, so we might see more fallout that has not already
    been handled.
  * There was a concern (from Mike) about whether the performance gain
    at the cost of stream size makes sense, given that the compression
    level could be reduced instead to similar effect (?). I'm not sure
    how these compare, so it would be interesting to analyze this,
    because perhaps that's a less traumatic way to look at it (but that
    might require redefining compression level semantics globally in
    zlib, or patching users, with neither look very enticing options).
    My perception from when I tested it is that the speed up was
    significant enough and the size increase not so much, but… In any
    case switching to zlib-ng upstream would also imply other benefits,
    like (supposedly) a more responsive upstream with more frequent
    releases, the new native API, and an implementation other
    distributions are switching to.
  * Some upstreams have started to use the zlib-ng native API, so
    regardless of whether we plan a switch or not, I guess packaging
    zlib-ng (w/ or w/ the compat API) might still make sense.
  * To consider a switch we'd need to do a mass rebuild of the
    archive. Ideally running autopkgtests and similar to exercise the
    packages?


After having written the above, and if Mark agrees, I think I'd opt for
uploading zlib-ng to experimental, with the compat packages renamed to
libz-ng-compat* or similar (even if that implies later on another trip
through NEW if we want to perform a full switch), because that might
make it easier to move them to sid as a way less disruptive change,
even if we decide not to switch the default zlib implementation.

OTOH and unfortunately I don't think I'm currently prepared to drive any
of what I think might be required mass archive rebuilds and testing or
the analysis mentioned above.

Thanks,
Guillem

You might find the history of the sdl12-compat source package interesting.
It's a reimplementation of the SDL 1.2 API/ABI in terms of SDL 2, and it
went through several stages:

- only SDL 2 available (as of bullseye);
- sdl12-compat available as non-default and outside the normal library
  search path, but can take over from "classic" SDL 1.2 if explicitly
  requested via LD_LIBRARY_PATH or by installing a extra "shim" package
  (this state is what we shipped in bookworm);
- sdl12-compat takes over binary packages from "classic" SDL 1.2,
  "classic" SDL 1.2 disappears, sdl12-compat's old binary packages become
  transitional (this is the current state in trixie);
- in forky, we'll presumably drop the transitional packages

Having the second step made it a lot easier to check for regressions.

Switching from glibc's libcrypt to the external libxcrypt, and from IJG
libjpeg to libjpeg-turbo, both went through similar processes.

sdl12-compat is a bit different because it's *only* the
backwards-compatible API/ABI, and not the "native" API/ABI (but you could
think of libsdl2, in its own source package, as being the corresponding
"native" API/ABI).

libxcrypt has a "native" legacy-free API/ABI available, but in Debian we don't
seem to be compiling it at the moment, and we only have the glibc- and
musl-compatible backwards-compatible ABIs (but I think some other distros
like Arch have moved to the legacy-free ABI).

libjpeg-turbo has a "native" legacy-free API/ABI libturbojpeg* in parallel
with a backwards-compatible API/ABI resembling IJG libjpeg 6b, so in that
sense it seems the closest to what you're proposing for zlib-ng.

Is there any problem with that, other than duplicate functionality?
From the other things you've said about the native API, it sound as
though the symbols all have different names, so this is no worse than
the duplication you would get if you have indirect dependencies on both
zlib and libdeflate, or on both OpenSSL and GNUTLS; it isn't like trying
to link to both libreadline and libedit (which export essentially the
same symbols and so will collide badly), or more than one of GTK 2/3/4
(likewise).

    smcv

Hi,

…

just built that.
…

So you upload it to experimental then autopkgtests should pick it up and
run against unstable. You should the results in
https://release.debian.org/britney/pseudo-excuses-experimental.html

I just kicked a rebuild locally with libz1 in the chroot. The proper
-dev gets installed if asked so that should work. Dose computed 38659
packages so this could a while.

Sebastian

* Guillem Jover <guillem@debian.org> [2024-11-22 12:29]:
[...]
[...]

As I reported to the Reproducible Builds mailing list in September, Fedora
switching to zlib-ng has created a very messy situation [1] for reproducibility
of Android APKs, which would have been far worse if not for the fact that
Fedora's own OpenJDK packages use a bundled zlib instead of linking against the
system zlib-ng.

Unfortunately, Reproducible Builds for Android APKs involve reproducing APKs
built by individual upstreams, not Debian buildds.  Most upstreams don't use
Debian, many even use Windows.  It's simply not feasible to use the exact same
build environment as upstream in a lot of cases, nor can we expect them to
switch to building on Debian to match the rebuilders.

So far, we have still been able get identical results 99% of the time because
the Android toolchain generates identical results on different platforms -- with
some exceptions, like newlines, that we have created workarounds for, though
those workarounds rely on being able to recreate a bitwise identical zlib output
stream.

I agree that it *should not* be Debian's responsibility to ensure compatibility
with Fedora/Windows/etc., but the reality is that if "you need the same tools to
generate the same output" -- which right now means using the same JDK and
Android toolchain but in 99% of cases doesn't require using the same OS since
everyone, including Google [2], standardised on zlib -- becomes "you cannot
reproduce APKs built on a OS other than Debian on Debian", that's not just
"annoying for the involved parties": it will effectively break the ability to
verify reproducibility of many Android apps.

Having both zlib implementations available and being able to choose between them
at runtime (perhaps using LD_LIBRARY_PATH in the tooling when needed) would help
a lot to at least allow creating workarounds, but that would still require
significant changes that someone needs to implement in order to unbreak
something that currently works fine.

- Fay

[1] https://lists.reproducible-builds.org/pipermail/rb-general/2024-September/003547.html
[2] https://android-developers.googleblog.com/2016/12/saving-data-reducing-the-size-of-app-updates-by-65-percent.html

* Fay Stegerman <flx@obfusk.net> [241123 19:58]:
bound by the decision of a BigTech Corporate and by thousands of
individual Android developers, neither of which might be interested
in Debian?

To maybe make the argument the other way 'round: if Google switches
to zlib-ng tomorrow, should Debian be required to switch to zlib-ng?

I just don't see that's how it works.

Chris

* Chris Hofstaedtler <zeha@debian.org> [2024-11-23 20:56]:

What I'm saying is that this change will have consequences for downstreams using
Debian for Reproducible Builds.  That includes e.g. hundreds of F-Droid apps,
which would no longer be able to get updates if Reproducible Builds break.

That's clearly not Debian's fault.  And of course Debian isn't *required* to do
anything.  I agree Debian should not be bound by the decisions of a "BigTech
Corporate" and thousands of individual Android developers.  And I very much
dislike the fact that this matters.

But Debian's choices here do have consequences for downstreams, and I think
that's something we should take into account when reasonably possible.  Similar
to how we didn't switch i386 to t64 to avoid breaking running existing legacy
binaries.

For example, if it can be made easy to install both and choose between zlib and
zlib-ng at runtime, so it's easy to build APKs using either zlib or zlib-ng as
needed, downstream breakage can be avoided.  Considering whether that can
reasonably be done doesn't seem like an unreasonable request to me.

What I would like is to be able to continue to use Debian for Reproducible
Builds regardless of what Google does or doesn't do.  Right now that means being
able to choose to keep using the original zlib for backwards compatibility.  If
Google switched to zlib-ng I would be asking if Debian could provide a way to
opt in to using that instead.

- Fay

Hi,
…

One thing I didn't debug and it might be expected: If I install
libz1_2.2.2-1_amd64.deb and then try to install build dependencies of
0ad via
	apt build-dep 0ad

then it removes libz1 while doing so. sbuild aborts complains and
aborts. I worked around it by installing the lib + header files in the
chroot before building.

ongoing… 59% left. In the meantime I uploaded what looks like possible
fallout to
https://people.debian.org/~bigeasy/zlib-ng/todo/

I moved cpp-httplib to done after filling #1088193 and
https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=zlib-ng@tracker.debian.org&tag=zlib-ng-ftbfs

lists it, too.

Sebastian

On 2024-11-24 21:36:25 [+0100], To debian-devel@lists.debian.org wrote:
…

I've been looking at cdebootstrap. It is one of the failing. This
happens in the rules files:
|  ( echo -n "misc:Built-Using="; dpkg-query -f='${source:Package} (=
| ${source:Version}), ' -W libc6-dev libdebian-installer4-dev
| libdebconfclient0-dev libbz2-dev liblzma-dev zlib1g-dev; echo )  >>
| debian/cdebootstrap-static.substvars

and then
| dpkg-gencontrol: warning: Depends field of package cdebootstrap-static: substitution variable ${shlibs:Depends} used, but is not defined
| dpkg-gencontrol: warning: can't parse dependency zlib1g-dev (= )
| dpkg-gencontrol: error: parsing package 'cdebootstrap-static' Built-Using field: bzip2 (= 1.0.8-6), glibc (= 2.40-4), cdebconf (= 0.273), libdebian-installer (= 0.125), xz-utils (= 5.6.3-1),  zlib1g-dev (= ),

Since we don't have zlib1g-dev anymore, this goes boom. If we exclude
zlib1g-dev then it is not recorded. Can't we smart about this?

Sebastian

Total              : 38655
Passed             : 37188
Failed with zlib-ng: 64
Failed always      : 1401

The total amount of build logs is ~20GiB. I uploaded compressed logs of
the which failed with zlib-ng and overview to
https://people.debian.org/~bigeasy/zlib-ng/

Three packages failed due to the way Build-Using is generated. I moved
those to another folder (cdebootstrap, dar, sash).

Sebastian

You CCed some people but not me so I only saw this today...

It does seem more safe to offer zlib-ng as an alternative at this
point...

The packages could have a different name and support diversion or
replacement of the zlib library packages?  That'd let people use them
if they wanted to.

What are those concerns?  Like Simon says the zlib and zlib-ng APIs are
just two separate APIs, they happen to have overlapping functionality
but while that might be a bit wasteful it's not obvious that there's any
actual problem.

Yes, I don't think this should be a blocker - this seems like it's on a
similar level to needing to use the same compilers.

Yes, it seems clear that we should package zlib-ng and there's just a
question about handling of the compat interface it provides.

Yes, that sounds sensible to me.

* Guillem Jover <guillem@debian.org> [2024-11-22 12:29]:
[...]
[...]

I did some more testing with zlib-ng.  With the original zlib, you will always
get an identical output stream given the same input stream and compressor
parameters (compression level being the only one that's commonly varied in ZIP
files).  I expected that zlib-ng would often produce a different output steam
than the original, but what I found was a lot more non-deterministic than just
that.

With zlib-ng, feeding the data into the compressor in e.g. 1024-byte chunks
always gave me a different output stream than using 4096-byte chunks (at
compression level 6).  In fact, every chunk size I tried gave a different
output.  And that's with fixed size chunks, which is not a given if you're
handling e.g. a stream of input.

Even using the same buffer size, I cannot get an identical compressed output
stream with Python and Java any more, presumably because of subtle
implementation differences (in the stdlib code that ends up calling zlib to do
the compression) that do not affect zlib but clearly do affect zlib-ng.

Which makes zlib-ng unsuitable for use cases where you need to be able to create
an identical output stream without knowing exactly how the bytes were fed into
the zlib compressor (or simply have no way to control this).  This fundamentally
breaks my tooling in ways I can't fix by using the same build environment.
Because programs that used to produce identical and deterministic output with
zlib no longer do with zlib-ng, despite using the exact same zlib-ng .so.

- Fay