Hi,

while running the MariaDB upgrade tests in Salsa CI, I noticed that the
bookworm -> sid upgrade path can leave the system in a state where
mariadbd fails to start due to an ABI mismatch between the server and
the provider plugins:
https://salsa.debian.org/mariadb-team/mariadb-server/-/jobs/8590614

AFAICS, this only hits bookworm-based systems. A similar bullseye-based
upgrade job does not hit this specific mismatch, because bullseye does
not ship the mariadb-plugin-provider-* packages.

During a partial upgrade from bookworm to sid, mariadbd 11.x aborts at
startup with:

  ERROR: mariadbd: Can't open shared library 'provider_bzip2.so'
  (errno: 8, API version for DAEMON plugin provider_bzip2 not supported
by this
  version of the server)

The reason is that the provider plugins are still at 10.11.x while the
server is now 11.x. At the same time, the default Debian configuration
asks MariaDB to load at least provider_lz4:

  plugin_load_add=provider_lz4
  provider_lz4=force_plus_permanent

With an ABI mismatch, this combination is fatal and the server never
comes up.

On bookworm, MariaDB 10.11 installs several provider plugins as
Recommends of mariadb-server.

In the failing scenario, the upgrade is only partial:

* mariadbd is upgraded to 11.x
* the provider plugins stay at 10.11.x, because upgrading
default-mysql-server alone does not force a upgrade of Recommends
* the configuration still tells mariadbd to load the provider plugins
* mariadbd 11.x finds 10.11.x DAEMON plugins and aborts with an API/ABI
mismatch

So we end up with:

  server: 11.x
  providers: 10.11.x

and the server is unusable.

In the MariaDB CI job "default-mysql-server and Bookworm upgrade", the
script does roughly:

  apt-get install default-mysql-server  # installs 10.11 + provider
plugins
  enable sid
  enable artifacts
  apt-get install default-mysql-server  # upgrades only the server
package

Because the provider plugins were already present from the first
install, they are not upgraded when only default-mysql-server is
requested. This reproduces the situation above and the job fails with
the DAEMON plugin API error

To make the CI job pass again, one possible workaround on the CI side
would be to explicitly install the provider plugins together with
default-mysql-server during the bookworm -> sid upgrade, so that apt
upgrades them in one go:

  apt-get install -y
  default-mysql-server
  mariadb-plugin-provider-bzip2
  mariadb-plugin-provider-lz4
  mariadb-plugin-provider-lzma
  mariadb-plugin-provider-lzo
  mariadb-plugin-provider-snappy

This would keep the server and provider plugins in sync and make the
upgrade path pass in CI.

Before changing the job, I wanted to check with you if you consider this
an acceptable CI-only workaround for this upgrade path, or if you would
prefer that this be addressed on the packaging side instead (for
instance by tightening dependencies or otherwise ensuring that provider
plugins cannot be left behind when upgrading the server).

Thanks, great research!

Indeed, in latest
https://salsa.debian.org/mariadb-team/mariadb-server/-/jobs/8590614
server fails to restart on with error log showing:

2025-11-14 19:56:13 0 [ERROR] /usr/sbin/mariadbd: unknown variable
'provider_bzip2=force_plus_permanent'
2025-11-14 19:56:13 0 [ERROR] /usr/sbin/mariadbd: unknown variable
'provider_lz4=force_plus_permanent'
2025-11-14 19:56:13 0 [ERROR] /usr/sbin/mariadbd: unknown variable
'provider_lzma=force_plus_permanent'
2025-11-14 19:56:13 0 [ERROR] /usr/sbin/mariadbd: unknown variable
'provider_lzo=force_plus_permanent'
2025-11-14 19:56:13 0 [ERROR] /usr/sbin/mariadbd: unknown variable
'provider_snappy=force_plus_permanent'

The same job a month ago didn't have any issues in
https://salsa.debian.org/mariadb-team/mariadb-server/-/jobs/8421127
and I can see that the resolver is now behaving differently, and no
longer upgrading the provider plugins which it earlier did (screenshot
attached).

Most likely related to the new versions in Bookworm apt 3.1.8->3.1.12
and libapt-pkg7.0 3.1.8->3.1.12.

I can also see that the output of `apt-get install` is different.

Compare before:
$ apt-get install -q --yes sysvinit-utils && cp -av /usr/lib/lsb /lib/
|| true [0;m
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  gcc-12-base gpgv libapt-pkg6.0
Use 'apt autoremove' to remove them.
The following packages will be upgraded:
  sysvinit-utils
1 upgraded, 0 newly installed, 0 to remove and 265 not upgraded.

Now:
$ apt-get install -q --yes sysvinit-utils && cp -av /usr/lib/lsb /lib/
|| true [0;m
Reading package lists...
Building dependency tree...
Reading state information...
Solving dependencies...
The following packages were automatically installed and are no longer required:
  gcc-12-base gpgv libapt-pkg6.0
Use 'apt autoremove' to remove them.
The following packages will be upgraded:
  sysvinit-utils
1 upgraded, 0 newly installed, 0 to remove and 265 not upgraded.

(there is a new line `Solving dependencies...`)

If you have time, can you please file a bug report on apt 3.1.12 in
Bookworm with complaint that the resolver in a stable release update
changed behaviour and no longer properly updates MariaDB minor
releases, with reference to this bug?

According to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1121158#15
this is a bug in src:mariadb debian/control, but also that apt was
previously handling this "bug" automatically, and might start handling
it again. So perhaps let this be for a while and return back in 2-3
months to check if packaging changes are really required.