Hi,

This is a follow-up / underlying issue for Bug#1121151 against
mariadb-server ("Salsa CI 'default-mysql-server and Bookworm upgrade'
job fails due to provider plugin ABI mismatch").

While debugging that CI failure, we found that apt's resolver behaviour
changed between 3.1.8 and 3.1.10/3.1.12 in a way that leaves already
installed Recommends at old versions in a real upgrade path, which then
breaks MariaDB.

Scenario:

 - Base system: Debian 12 (bookworm)
 - First install:

    apt-get update
    apt-get install -y default-mysql-server

   I think this installs mariadb-server 1:10.11.14-0+deb12u2 and the
   Recommends mariadb-plugin-provider-* (bzip2, lz4, lzma, lzo, snappy)

 - Then I enable sid and a local repo with MariaDB 11.8.3 and upgrade
apt:

    echo 'deb http://deb.debian.org/debian sid main' >
/etc/apt/sources.list
    apt-get update
    apt-get install -y apt      # on the current system this installs
apt 3.1.12

 - Finally I try to upgrade MariaDB:

    apt-get install -y default-mysql-server

Previously, when running the same steps with apt 3.1.8, this last
command upgraded both mariadb-server and all mariadb-plugin-provider-*
Recommends to 11.8.3 from the local repo and MariaDB started fine.

Now, with apt 3.1.12, the same command upgrades mariadb-server to 11.8.3
but leaves all mariadb-plugin-provider-* at the old 10.11.14 versions
from bookworm. The system ends up with:

  - server:    11.8.3
  - providers: 10.11.14

and MariaDB fails to start with DAEMON plugin API errors, for instance:

  ERROR: mariadbd: Can't open shared library 'provider_bzip2.so'
  (errno: 8, API version for DAEMON plugin provider_bzip2 not supported
by this version of the server)

This regression is visible in CI:

 - Old passing job (before apt change): upgrades providers too:
https://salsa.debian.org/mariadb-team/mariadb-server/-/jobs/8421127

 - New failing job with apt 3.1.12: only the server is upgraded:
https://salsa.debian.org/mariadb-team/mariadb-server/-/jobs/8590614

Summary of regression:

Previously (3.1.8), "apt-get install default-mysql-server" would also
upgrade the already-installed mariadb-plugin-provider-* Recommends when
newer versions were available, so server and plugins stayed in sync.

Now (3.1.10-3.1.12), the solver considers those Recommends already
satisfied and leaves them at the old version, which in this real upgrade
path produces a broken MariaDB installation.

This seems related to enabling the new solver3 by default in 3.1.10 and
its handling of already satisfied Recommends, but the practical issue is
that a stable apt update changed resolver behaviour and broke a real
bookworm -> sid upgrade scenario that used to work.

Thanks,

Aquila Macedo

Hi!

Thanks for spotting this. This actually means that due to changes in
how the apt resolver works, upgrades from Bookworm to Forky are broken
for anyone running MariaDB.

I wonder what the specific change in the resolver was that caused
this, and if it was intentional or not, and if this should be fixed by
adjusting the resolver or redoing the 'default-mysql-server' or
'mariadb-server' dependencies.

[...]

This is a bug in the packaging which does not express those dependencies
either as versioned dependencies or breaks in either direction (which
one to pick is tricky, plugins breaking the server is _technically_
wrong as the server upgrade is breaking it, but it probably has fewer
chances of removing the server as the server breaking a plugin).

Not really. What you'll find is that the old solver has a special case
to *try* to upgrade all binary packages in a source packages together.

This has been infeasible in the new solver until last month, but may
be feasible now that we have eager Recommends. It may ultimately fail
validation against Canonical's internal solver regression test suite
(which is internal because it runs on private data reported by real
world users).

At no point did we guarantee upgrading all packages, but it is useful to
paper over bugs like this so they hit fewer users.

On the flip side even with the old solver (and once this is reinstated),
CI likely should set APT::Get::Upgrade-By-Source-Package to false to
ensure that the packaging is actually correct.

This is a misconception, APT uses the odd/even version scheme,
3.1.10 is the 11th development release of the 3.2 series (and
3.9.0 is the first development release of the 4.0 series).

While we hold each (pre-)release to the same requirements in terms
of functional stability, odd releases are allowed to make breaking
changes relative to a previous release.

That is, changes are released to development series when they are
ready, and not queued up in say experimental for a big eventual drop.

The exception are the .9 series every 5 years which break the API
and ABI; A.9.C lands in experimental first until the A{P,B}I are
finalized.

Am Fri, Nov 21, 2025 at 05:17:10PM -0800, schrieb Otto Kekäläinen:
Let me correct the (repeated) misconception here:
An upgrade from bookworm -> sid/forky is performed by apt/bookworm,
not by apt/forky, in the general case. More importantly, a call like
"apt install foo" is NOT representative of an upgrade.

In an upgrade (think apt full-upgrade) all installed packages are marked
for installation (which boils down to an upgrade of each package).
In an explicit install request only that package is marked for install.
Everything else happens only as needed. And as Julian explained, given
missing breaks, nothing is needed.

Now some people make these explicit install requests to selectively
upgrade ~ individual packages ~, but its not the norm. I would go as far
as saying it usually is a user error (think: "Oh my god, a CVE for foo!
Lets upgrade foo!" but foo uses the not upgraded CVE-buggy libfoo).
And as Julian mentioned the old resolver had (not always, just since
2.5.2, see NEWS) errata to make that work more consistently by assuming
the user meant to upgrade selectively the entire source package.

(A bug for restoring this exists already, but the number eludes me ATM)


Best regards

David Kalnischkies

Hi!

We have extensive CI that tests this upgrade (see previous messages
and links to CI) which simulates users who have installed something in
Debian Bookworm that depends on default-mysql-server. These CI tests
have been passing for years and started failing now due to a new
apt/resolver.

If this is the result of a new version of apt/bookworm or apt/forky I
don't know for sure, I guess that is something to debug as part of
this bug report.

This is a simplified case to reproduce the issue. The full CI code
runs`apt-get full-upgrade` to exactly simulate the case a end-user
would have, see CI code at
https://salsa.debian.org/mariadb-team/mariadb-server/-/blob/debian/latest/debian/salsa-ci.yml,
or read the CI logs linked to earlier, they show each command followed
by its output.

If may of course be that the CI has been running `apt-get
full-upgrade` wrongly for years and years, or that the package
depends/breaks relationships have been wrong for years and only now
surfaced as the resolver was "fixed" to no longer satisfy "wrong"
relationships, but I wouldn't jump to that conclusion just yet.

Also, even in the case that the resolver is now "correct", if there
are a large number of packages that depend on the old behaviour, the
new correct might still not be the optimal solution.

Assuming the apt changes were recent, let's see if other package
maintainers report similar issues that updates with virtual/indirect
packages no longer pull in what they used to.

Am Sat, Nov 22, 2025 at 12:05:25PM -0800, schrieb Otto Kekäläinen:

So, have you read what you want others to look at or … ?

If you had, you would have noticed that the CI test that failed uses
'apt install', also apparent by the "fix"¹ that was deployed for that
test to make it green, because green is good.
See e.g. https://salsa.debian.org/mariadb-team/mariadb-server/-/jobs/8622649#L1881

Never mind that many of the tests are testing entirely unsupported
setups – as skipping releases is not supported in Debian. But as long
as they are green, never mind what they actually test, I suppose.

¹ I at least would not expect users to write this monstrosity of an
  install command, but if you are set on not fixing your dependencies
  as suggested by Julian… oh well.

As mentioned, the old resolver implemented that behaviour since *drumroll
please* bookworm. So your test worked "for years and  years" just because
it was added right after apts default resolver tried to help users by
papering over insufficient dependencies.

This behaviour can be disabled in the old resolver (some users do) and
the old is far from the only solver used in the wild, e.g. aspcud for
experimental builds and by adventures users, aptitude, … which all do
not have this behaviour (as it has the obvious downside of pulling too
much into the upgrade set if the maintainer wrote sufficient deps).

Julian already said that he wants to add this user-protection feature
to the new resolver as well. But even if he does, any number of packages
that rely on it remain RC-buggy for declaring insufficient dependencies
and users that not use apt (and its default config) will continue to run
into problems with these packages.

APT behaviour is not a popularity contest. Never mind that by the time
this popularity contest would find traction is as explained earlier in
the forky to duke upgrade – so way too late if you don't want to
displease the stable release managers of forky and the release team
trying to release duke at that time around duke hard freeze.

Your problem is also not about "virtual packages" (see Debian policy
what that is) nor "indirect packages" (whatever that may be). Your
packages do not pull in what they need, period. That didn't change.
What changed is that apt was graciously papering over your bug by
assuming you meant to pull those into the upgrade set, too.

It currently doesn't anymore, maybe it does again by the time forky is
released. Until then users of non-defaults might even consider that an
apt feature as it directly helps them by having users with the defaults
run into the same problem they encounter.

As Julian already said, it might make sense to disable this behaviour in
tests if it is reintroduced.


Best regards

David Kalnischkies