Maintainers have seen bugs in various packages where an old file (one
that was shipped in an old .deb, but not under the same name in a new
.deb) appears to have been retained on-disk rather than deleted. At some
later time, perhaps months or years later, that file causes an upgrade
failure triggered by the superseding file moving from one directory in
a search path to another.

I was involved in investigating an instance of this in GLib, but
I've recently been shown examples of similar things happening in other
packages, so I think this might be more common than we'd thought (although
still rare). Please see below for summaries and links.

Because some maintainers believe the best approach to the "/usr merge"
is for each package that has historically been on the root filesystem to
carry out its own migration from / to /usr, this seems likely to occur
increasingly often as more packages do that transition.

My questions about this:

- Is/was there a known dpkg bug that could cause this?
- Is there anything that individual packages should be doing when they
  move files into different directories in a search path, to work around
  this?
- Is there anything that dpkg could/should be doing to help?

When I say /lib/MA or /usr/lib/MA below, I mean the multiarch library
directory.

GLib (#896019 and friends)
--------------------------

In GLib, we've had several reports of an obsolete file like
/lib/x86_64-linux-gnu/libglib-2.0.so.0.4200.0 somehow being kept when
it should have been superseded by a newer version like
/lib/x86_64-linux-gnu/libglib-2.0.so.0.5000.0. Much later, when the
intended path for this library changed from /lib/MA/libglib-2.0.so.0.* to
/usr/lib/MA/libglib-2.0.so.0.*, ldconfig(8) would create symlinks like:

    /lib/MA/libglib-2.0.so.0 -> libglib-2.0.so.0.4200.0
    /usr/lib/MA/libglib-2.0.so.0 -> libglib-2.0.so.0.5600.0

and because /lib/MA has higher precedence in the search path than
/usr/lib/MA, the older version in /lib/MA was preferred, and packages
with a versioned dependency on newer GLib releases started failing.

- Report: <https://bugs.debian.org/896019>
- Report: <https://bugs.debian.org/894763>
- Technical committee discussion: <https://bugs.debian.org/911225>
  (I haven't yet implemented the suggested workaround in GLib)

libcrypt (#948318)
------------------

Similar to the GLib bugs above, /lib/x86_64-linux-gnu/libcrypt-2.25.so
somehow wasn't deleted on upgrade from glibc 2.25 to 2.26+. Years later,
the canonical path of libcrypt.so.1 changed from /lib/MA to /usr/lib/MA
during the transition to libxcrypt. Similar to the GLib bugs, ldconfig(8)
created a symlink /lib/x86_64-linux-gnu/libcrypt.so.1 -> libcrypt-2.25.so
which was higher-precedence than the one in /usr/lib/MA, causing programs
that reference newer libcrypt symbols to fail.

- Report: <https://bugs.debian.org/948318>

systemd (#945582)
-----------------

This one is a bit different: an old copy of systemd-machine-id-setup in
/usr/bin takes precedence over a new copy in /bin, and has a lockstep
dependency on an equally old libsystemd-shared-241.so, but dpkg correctly
*did* remove that library.

I don't think this is necessarily the same thing, because the systemd
maintainers say systemd never installed systemd-machine-id-setup into
/usr/bin and has always put it in /bin. Nobody involved in that bug
knows how it got into /usr/bin.

However, if an executable legitimately migrated from /usr/bin to /bin,
a similar failure mode could happen if the old executable somehow isn't
deleted during the dpkg transaction.

- Report: <https://bugs.debian.org/945582>
--------

Any ideas?

    smcv

dpkg maintainers: This looks similar to #896019 in GLib and #948318
in libcrypt, except instead of their transition from /lib/MULTIARCH
to /usr/lib/MULTIARCH, it involves libusb's transition from /lib to
/lib/MULTIARCH. I'm worried that we might see this more often as more /lib
libraries, like libdbus in experimental, transition from /lib to /usr/lib.

If we need to be preemptively detecting and fixing up this sort of thing
in all packages that move libraries from one directory to another, then
the script in https://salsa.debian.org/gnome-team/glib/-/merge_requests/9
might be a useful basis for that - review and testing would be
appreciated.

Please move that file to another directory for further investigation,
for example:

mkdir /lib/removed-for-bug-972703
mv /lib/libusb-1.0.so.0* /lib/removed-for-bug-972703/

(This should fix the immediate failure.)

Then please send the results of:

ls -l /lib/removed-for-bug-972703/*
md5sum /lib/removed-for-bug-972703/*

It would also be interesting to know whether there are other libraries
in /lib and /usr/lib that dpkg doesn't know about:

    find /lib /usr/lib -name '*.so.*' | while read -r x; do dpkg -S "$x" >/dev/null; done

You'll get messages on stderr like:

    dpkg-query: no path found matching pattern /lib/removed-for-bug-972703/libusb-1.0.so.0

Some of the results of this will be files that are managed by
alternatives, like /usr/lib/x86_64-linux-gnu/libblas.so.3, which are
harmless - but others might not be.

This seems reminiscent of rare upgrade issues that we've seen before in
GLib, when libglib-2.0.so.0 moved from /lib/x86_64-linux-gnu to
/usr/lib/x86_64-linux-gnu - but you're maybe seeing it for a previous
transition, when libusb-1.0.so.0 moved from /lib to /lib/x86_64-linux-gnu.

The libusb-1.0-0 package installed /lib/libusb-1.0.so.0 before 2011.
The ls -l and md5sum output will help to determine whether this is an
outdated version from Debian, or a different version from outside Debian.

Are these old Debian installations that were installed before 2011 and
have been upgraded since then?

If yes, did they have unreliable disks, frequent crashes, or anything
like that in the 2011-2012 timeframe?
package to a new package, and the new package ships different files,
dpkg is meant to delete the old file, leaving only the new file. It is
possible that there is a rare bug that causes it to *not* delete the
old file. If that happens, then ldconfig prefers the new file, and you
don't immediately notice that anything is wrong.

Perhaps years later, the library moves from a lower-priority directory in
the search path to a higher-priority directory, and suddenly ldconfig
will be preferring the older version: but you *still* don't notice
anything is wrong until a different program or library (in your case
libusbredirhost.so.1 and qemu) requires a symbol that wasn't available
in the new version.

    smcv

Hi Simon

This is embarassing but I found now what was the issue.  There is one
package that I installed some months ago that was not from Debian
repos and for some reason it installs /lib/libusb.  It's not even a
deb file which is why dpkg-query does not know about it.

The package in question is xiAPI, from ximea, a company for microscope
cameras https://www.ximea.com/support/wiki/apis/xiAPI .  Without the
whole working from home thing, I would have never have installed on my
computer.

I'm very sorry for the noise.  Thank you for the help, it was the
process of answering your questions, and searching for other libs that
dpkg did not know about, that made me remember this.

David

Control: block 984884 by 949395
[...]

Hello,

I have got another instance in libgcrypt20
<https://bugs.debian.org/984884> but the actual error (leaving files
behind) happened ages ago; the leftovers are from pre-stretch versions
of libgcrypt.

These leftovers trigger an error now, because the library has moved from
/lib to /usr/lib in buster and ldconfig activates both the leftover and
the proper copy (by adding libgcrypt.so.20 symlinks). /lib has the
higher priority and wins.

cu Andreas

able to work out why. In the glib2.0 bugs, the leftover versions were
older: from wheezy (we think) in #954960, and from half way through the
wheezy -> jessie cycle in #894763, #896019 and #955331.

I asked the Technical Committee for advice about how to deal with this in
glib2.0, and we came to the conclusion that cleaning up the old library
in the maintainer scripts was the least-bad option.

There's a script in glib2.0 that can be called from a maintainer script
to work around this. I've only used it for GLib, but it should work
equally for any library that moved from /lib/MULTIARCH to /usr/lib/MULTIARCH,
since it takes a bug number, a multiarch tuple and a SONAME as command-line
options:
https://salsa.debian.org/gnome-team/glib/-/blob/debian/bullseye/debian/clean-up-unmanaged-libraries

Example use in a postinst (look for "cleanup"):
https://salsa.debian.org/gnome-team/glib/-/blob/debian/bullseye/debian/libglib2.0-0.postinst.in

GLib uses a generated postinst where #MULTIARCH# gets replaced
by the multiarch tuple of the package. As far as I know, this
is going to be necessary for any multiarch library that will use
my clean-up-unmanaged-libraries script, because dpkg tells us the
DPKG_MAINTSCRIPT_PACKAGE and DPKG_MAINTSCRIPT_ARCH, but there is no
DPKG_MAINTSCRIPT_MULTIARCH.

I'll open a separate wishlist bug for DPKG_MAINTSCRIPT_MULTIARCH,
because that would be useful to have - GLib needed it anyway, even before
the cleanup was implemented, to handle GIO plugins.

    smcv

On 2021-09-11 Simon McVittie <smcv@debian.org> wrote:
[...]

Hello,

dpkg-buildpackage sets DEB_BUILD_MULTIARCH but afaiui
one cannot /rely/ on it being set since it is not listed in policy 4.9.
I will probably make sure it is set in debian/rules and use
#ENV.DEB_BUILD_MULTIARCH#

cu Andreas

You want DEB_HOST_MULTIARCH, not DEB_BUILD_MULTIARCH, for this.  When
cross-compiling, DEB_BUILD_MULTIARCH is the machine on which the package
is being built, which is only rarely an important piece of information.

(For example if you cross-compile a package for mipsel on amd64, the host
architecture is mipsel and the build architecture is amd64. The multiarch
tuple my script would need from the postinst is the one for mipsel.)

If in doubt, the _HOST_ variables are usually the right ones to look at.

    smcv