Hello,

while working on a bookworm -> trixie upgrade failure, I noticed a strange line
showing up.

| Preparing to unpack .../openssh-server_1%3a10.0p1-5_amd64.deb ...
| systemctl: error while loading shared libraries: libcrypto.so.3: cannot open shared object file: No such file or directory

This is openssh-server.preinst failing to systemctl stop
rescue-ssh.service. I talked to Colin and both of us agreed that this
instance is probably practically irrelevant. However, I still think
there is a problem. Due to the time64 transition, libssl3 was renamed to
libssl3t64 and for some reason apt ends up removing libssl3 before
unpacking libssl3t64. Given Breaks+Replaces, unpacking libssl3t64 after
having deconfigured libssl3 before removing libssl3 should be fine, but
apt does not like that solution. As a result, libcrypto.so.3 is
temporarily removed.

deb-systemd-invoke is part of init-system-helpers and therefore
essential. It calls out to systemctl, which is not essential but for all
practical matters we really should be treating it as if it was and
maintainer scripts expect it to work at all times. libssl3 or libssl3t64
are pseudo-essential. Some part (apt or openssl) violates policy during
the upgrade as being pseudo-essential requires it to work at all times
even when unpacked.

In practice, this means that systemctl cannot be expected to work in
maintainer scripts. This will mostly affect preinst scripts (not just
openssh-server) trying to stop services. For instance, it is conceivable
that we could fail to stop mariadb or postgresql due to this (but there
is no practical evidence of this ever having happened). Failure to stop
services violates assumptions placed by package maintainers and that may
have all sorts of consequences. I have several reports of systemctl
having failed during release upgrades without having failed the upgrade
transaction as a whole.

It really is unclear whether this has practical consequences and whether
there is a dataloss scenario something else that makes this problem
practically relevant. We typically reboot after a dist upgrade (at least
that's what release notes strongly recommend) and doing so tends to fix
any failure to stop or start services. I have no evidence of this
problem having caused a real issue (beyond that message).

If you have earlier upgraded from bookworm to trixie. You should be able
to search in your /var/log/apt/term.log* for the earlier message to see
whether you were affected.

In talking to Ivo and Paul, we agreed to report the problem to d-devel
via upgrade-reports. At this stage we want to gauge the impact and
better understand how serious this actually is, so following up on the
bug report with evidence (dropping all lists if that's all you add) is
highly appreciated.

The options for fixing this are dim. Reverting the t64 transition for
openssl and going dual-ABI seems highly unlikely even though it would
fix the problem at the root. Other options are dim, because we have no
scripts that are guarantueed to run before apt chooses to remove
bookworm's libssl3. We considered doing changes to bookworm to mitigate.
Conceivably, a bookworm update could add a libssl3.preinst that diverts
the library to keep it around until it is overwritten by libssl3t64.

I invite others to work on the problem as I have no capacity to do it.
I'm still yak shaving another release upgrade problem and would like to
enjoy DebConf. Thank you

Helmut

Hi Chris,

I believe this is independent and I merely caught it as I was carefully
reading test results for your other issue. This particular one is solely
caused by libssl3 having done a t64 transition when it was not
appropriate to do so. I do not expect my proposed change to glibc to
improve the situation regarding systemctl.

Helmut

Is this problem still happening?

If so, are there any ideas on what to do about it?

Chris

[looping in bluca as well]

On Thu, 24 Jul 2025 02:43:27 +0200 Chris Hofstaedtler <zeha@debian.org> wrote:
Not sure if it's relevant, but I post it here for completeness sake:

To make systemctl more robust during dist-upgrades, in the past we built
systemd with `-Dlink-systemctl-shared=false`, so systemctl has a minimal
set of library dependencies and then we added all those library
dependencies to Pre-Depends via
https://salsa.debian.org/systemd-team/systemd/-/blob/debian/master/debian/rules?ref_type=heads#L302

This was under the assumption that systemctl is quasi-essential and can
be triggered at arbitrary times during a dist-upgrades.


Luca changed that (recently) in
https://salsa.debian.org/systemd-team/systemd/-/commit/8ca2b266789bff0b1af348100e72da7245864174

I.e. systemctl now links against libsystemd-shared which in turn has a
couple of side-effects

a/ libsystemd-shared is not an so-versioned package, i.e. different
versions can not be co-installed.
So there is a time window where systemctl is in a non-functional state,
e.g. when you have an old systemd + new libsystemd-shared or a new
systemd + old libsystemd-shared unpacked

This could be addressed by making libsystemd-shared so-versioned, i.e.
naming it libsystemd-shared-$VER or going back to linking systemctl
statically against libsystemd-shared.

b/ If Luca want's to continue to use `-Dlink-systemctl-shared=true`,
maybe it would make sense to treat all those library dependencies of
libsystemd-shared as Pre-Depends.


We are rather late in the trixie release cycle, so maybe the safest
approach is to revert 8ca2b266789bff0b1af348100e72da7245864174 for
trixie and re-apply it early during the  forky release?


Michael

Am 24.07.25 um 23:18 schrieb Michael Biebl:

Especially, since trixie contains both the t64 and usrmove related
changes, which will no longer be an issue for forky.

Hi Michael,

I appreciate that you chime in here.

Could you help me understand why reverting this commit would improve the
symptoms of the bug report at hand?

I understand that having libsystemd-shared and systemctl can be upgraded
separately and that doing so results in yet another window where
systemctl is dysfunctional. The problem reported here is about systemctl
linking libcrypto.so.3 however. Both bookworm and trixie do that.
Therefore my understanding is that the aforementioned commit does not
cause the libcrypto.so.3 linkage that is the problem here. Conversely,
reverting it will not remove libcrypto.so.3 and therefore will not help
with the t64-induced library rename.

What I'm trying to argue here is that even if reverting it improves
robustness in some way, it does not improve robustness regarding libssl3
to libssl3t64 upgrades. Do you concur here?

Helmut

Hi Helmut

Am 25.07.25 um 00:18 schrieb Helmut Grohne:

My hope was, that the systemctl dependencies would be significantly
trimmed down but it seems we regressed in that regard already in
bookworm. In bullseye e.g. systemctl didn't link against libcrypto:



systemctl gained a libcap, libcrypto, libm and libmount dependency.


systemctl (also via libsystemd-shared) gained a libacl, libaudit,
libblkid, libcap-ng, libcrypt, libpam and libseccomp dependency.


Building systemd with -Dlink-systemctl-shared=false (in trixie) results in

The list of dependencies shrinks significantly, but you are right,
libcrypto does not go away. So it's not immediately obvious, if that
change would help or not. It might still have an influence as libcrypto
(libssl3t64) would be pulled into the Pre-Depends set:


Michael

reproducer for this?

Given there were unclear reasons for the deconfigure/unpack order of
libssl3(t64), I do wonder if other changes in the archive might
have improved the situation.

Best,
Chris

Hello,

please unsubscribe me from all lists.

best,

Daniel