#994836 exim: dpkg fatal error due to Debian-exim in statoverride

#994836#5
Date:
2021-09-21 16:57:41 UTC
From:
To:
whilst trying to install build-deps for therion in an unstable chroot
i.e
apt source therion (6.0.2ds1-3 is downloaded)
cd therion-6.0.2ds1
sudo apt --no-install-recommends build-dep .

I got (after downloading 887MB of stuff, (304MB, 270 packages)):

debconf: delaying package configuration, since apt-utils is not installed
dpkg: unrecoverable fatal error, aborting:
 unknown system group 'Debian-exim' in statoverride file; the system group got removed
before the override, which is most probably a packaging bug, to recover you
can remove the override manually with dpkg-statoverride
E: Sub-process /usr/bin/dpkg returned an error code (2)
E: Failed to process build dependencies

That's quite bad. Debian-exim is indeed mentionned in the override file.
$ cat /var/lib/statoverride:
root crontab 2755 /usr/bin/crontab
root Debian-exim 640 /etc/exim4/passwd.client
root messagebus 4754 /usr/lib/dbus-1.0/dbus-daemon-launch-helper

these exim* packages are installed:
$ dpkg -l | grep exim
ii  exim4-base                           4.94-9+b1                      arm64        support files for all Exim MTA (v4) packages
ii  exim4-config                         4.94-9                         all          configuration for the Exim MTA (v4)
ii  exim4-daemon-light                   4.94-9+b1                      arm64        lightweight Exim MTA (v4) daemon

The set of pakcages being changed is:
The following packages were automatically installed and are no longer required:
  libdav1d4 libwavpack1
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  libavresample4
The following NEW packages will be installed:
  ca-certificates-java catch2 default-jdk default-jdk-headless default-jre default-jre-headless default-libmysqlclient-dev
  faketime fonts-urw-base35 gcc-11-base gdal-data gfortran-10 ghostscript ibverbs-providers imagemagick imagemagick-6-common
  imagemagick-6.q16 java-common libaom-dev libarmadillo-dev libarmadillo10 libarpack2 libarpack2-dev libavcodec-dev
  libavformat-dev libavutil-dev libblas-dev libboost-dev libboost1.74-dev libbrotli-dev libcfitsio-dev libcfitsio9 libdap-dev
  libdap27 libdapclient6v5 libdapserver7v5 libdav1d-dev libdav1d5 libde265-0 libde265-dev libdeflate-dev libdeflate0
  libegl-dev libeigen3-dev libevdev2 libevent-core-2.1-7 libevent-dev libevent-extra-2.1-7 libevent-openssl-2.1-7
  libevent-pthreads-2.1-7 libfabric1 libfaketime libfmt-dev libfmt7 libfontconfig-dev libfontconfig1-dev libfreetype-dev
  libfreetype6-dev libfreexl-dev libfreexl1 libfyba-dev libfyba0 libgdal-dev libgdal29 libgeos-3.9.1 libgeos-c1v5 libgeos-dev
  libgeotiff-dev libgeotiff5 libgfortran-10-dev libgif-dev libgif7 libgl-dev libgl1-mesa-dev libgl2ps-dev libgl2ps1.4
  libgles-dev libgles1 libgles2 libglew-dev libglew2.1 libglu1-mesa libglu1-mesa-dev libglvnd-core-dev libglvnd-dev
  libglx-dev libgs9 libgs9-common libgudev-1.0-0 libhdf4-0-alt libhdf4-alt-dev libhdf5-mpi-dev libhdf5-openmpi-103-1
  libhdf5-openmpi-cpp-103-1 libhdf5-openmpi-dev libhdf5-openmpi-fortran-102 libhdf5-openmpi-hl-100 libhdf5-openmpi-hl-cpp-100
  libhdf5-openmpi-hl-fortran-100 libheif-dev libheif1 libhwloc-dev libhwloc-plugins libhwloc15 libibverbs-dev libibverbs1
  libice-dev libidn12 libijs-0.35 libinput-bin libinput10 libjbig2dec0 libjs-jquery libjs-jquery-ui libjson-c-dev
  libjsoncpp-dev libjsoncpp24 libkml-dev libkmlbase1 libkmlconvenience1 libkmldom1 libkmlengine1 libkmlregionator1 libkmlxsd1
  libkpathsea6 liblapack-dev liblqr-1-0 liblz4-dev libmagickcore-6.q16-6 libmagickwand-6.q16-6 libmariadb-dev
  libmariadb-dev-compat libmd4c0 libminizip-dev libminizip1 libmtdev1 libnetcdf-c++4 libnetcdf-cxx-legacy-dev libnl-3-200
  libnl-3-dev libnl-route-3-200 libnl-route-3-dev libnotify4 libnspr4 libnss3 libnuma-dev libodbc1 libogdi-dev libogdi4.1
  libogg-dev libopengl-dev libopengl0 libopenjp2-7-dev libopenmpi-dev libopenmpi3 libpaper-utils libpaper1 libpciaccess0
  libpcre16-3 libpcre2-16-0 libpcre3-dev libpcre32-3 libpcrecpp0v5 libpcsclite1 libpmix-dev libpmix2 libpoppler-dev
  libpoppler-private-dev libpoppler102 libpq-dev libpq5 libproj-dev libproj19 libptexenc1 libpthread-stubs0-dev libqhull-dev
  libqhull-r8.0 libqhull8.0 libqhullcpp8.0 libqt5core5a libqt5dbus5 libqt5gui5 libqt5network5 libqt5widgets5 librdmacm1
  librttopo-dev librttopo1 libshp-dev libshp2 libsm-dev libspatialite-dev libspatialite7 libsqlite3-dev libsqlite3-tcl
  libsrt1.4-gnutls libsuperlu-dev libsuperlu5 libswresample-dev libswscale-dev libsynctex2 libtbb-dev libtbb2 libtcl8.6
  libteckit0 libtexlua53 libtheora-dev libtk8.6 libucx0 liburiparser-dev liburiparser1 libutfcpp-dev libvtk9 libvtk9-dev
  libvtk9-java libvtk9-qt libwacom-common libwacom2 libwebp-dev libwebpdemux2 libwxbase3.0-0v5 libwxbase3.0-dev
  libwxgtk3.0-gtk3-0v5 libwxgtk3.0-gtk3-dev libx11-dev libx265-dev libxau-dev libxcb-icccm4 libxcb-image0 libxcb-keysyms1
  libxcb-randr0 libxcb-render-util0 libxcb-util1 libxcb-xinerama0 libxcb-xinput0 libxcb-xkb1 libxcb1-dev libxdmcp-dev
  libxerces-c-dev libxerces-c3.2 libxext-dev libxft-dev libxkbcommon-x11-0 libxnvctrl0 libxrender-dev libxss-dev libxt-dev
  libzimg2 libzstd-dev libzzip-0-13 mpi-default-bin mpi-default-dev ninja-build odbcinst odbcinst1debian2 openjdk-11-jdk
  openjdk-11-jdk-headless openjdk-11-jre openjdk-11-jre-headless openmpi-bin openmpi-common pkg-config poppler-data proj-data
  python3-mpi4py python3-vtk9 rpcsvc-proto survex tcl tcl-dev tcl8.6 tcl8.6-dev tex-common texlive-base texlive-binaries
  texlive-metapost tk tk-dev tk8.6 tk8.6-dev unixodbc-dev vtk9 wx-common wx3.0-headers x11proto-dev xorg-sgml-doctools
  xtrans-dev
The following packages will be upgraded:
  binutils binutils-aarch64-linux-gnu binutils-common cpp-10 ffmpeg g++-10 gcc-10 gcc-10-base libaec-dev libaec0 libasan6
  libatomic1 libavcodec58 libavdevice58 libavfilter7 libavformat58 libavutil56 libbinutils libblas3 libc-bin libc-dev-bin
  libc6 libc6-dev libcc1-0 libctf-nobfd0 libctf0 libegl1 libgcc-10-dev libgcc-s1 libgfortran5 libgl1 libglib2.0-0 libglvnd0
  libglx0 libgnutls-dane0 libgnutls30 libgomp1 libitm1 libjson-c5 liblapack3 liblsan0 liblz4-1 libmariadb3 libnettle8
  libnuma1 libogg0 libopenjp2-7 libpostproc55 librubberband2 libsqlite3-0 libstdc++-10-dev libstdc++6 libswresample3
  libswscale5 libsz2 libtsan0 libubsan1 libvpx6 libwebp6 libwebpmux3 libx11-6 libx11-xcb1 libxau6 libxcb1 libxext6 libzstd1
66 upgraded, 286 newly installed, 1 to remove and 499 not upgraded.
Need to get 512 MB of archives.
After this operation, 887 MB of additional disk space will be used.

exim is not mentioned in that list.

Turns out that the install list doesn't matter. any use of dpkg will hit this statoverride issue.

/etc/exim4/passwd.client does exist but is owned by an unknown group.
$ll /etc/exim4/passwd.client
-rw-r----- 1 root 132 204 Nov  4  2020 /etc/exim4/passwd.client

And groupno 132 is not used in /etc/group:
...
ssl-cert:x:129:
avahi-autoipd:x:130:
nm-openvpn:x:131:
apt-cacher-ng:x:145:
sbuild:x:149:wookey01,wookey
...

So it does indeed look like the Debian-exim group was removed, but at least one file owned by it, and the statoverride remain.

Is that expected? - I presume not.

/var/lib/dpkg/info/exim4-base.postinst looks like it creates a Debian-exim user still (although not obviously the group?)

$ sudo ls -ld /var/spool/exim4:
drwxr-x--- 5 130 132 4096 Dec  9  2020 /var/spool/exim4
so the Debian-exim user is gone too.

$ cat /etc/passwd
...
nm-openvpn:x:128:131:NetworkManager OpenVPN,,,:/var/lib/openvpn/chroot:/usr/sbin/nologin
hplip:x:129:7:HPLIP system user,,,:/var/run/hplip:/bin/false
apt-cacher-ng:x:132:145::/var/cache/apt-cacher-ng:/usr/sbin/nologin
sbuild:x:133:149:Debian source builder,,,:/var/lib/sbuild:/bin/bash
...

I grepped for mentions of Debian-exim in /var/dpkg/info and only found it in exim scripts, so not clear what might have removed it:
$ grep Debian-exim /var/lib/dpkg/info/*
/var/lib/dpkg/info/exim4-base.postinst
/var/lib/dpkg/info/exim4-config.postinst
/var/lib/dpkg/info/exim4-config.postrm

Anything else I should check?

#994836#8
Date:
2021-10-19 09:14:55 UTC
From:
To:
Control: reassign -1 exim4

Reassigning to correct package.

Kind regards,
Andrei

#994836#15
Date:
2021-10-19 16:56:15 UTC
From:
To:
[...]
[...]
[...]

Hello Wookey,

no, it is not expected. Both -base and -config will add a Debian-exim
system user with its own private group in postinst if it does not exist.
But none of the exim packages ever remove the user.

So it looks like random system breakage to me. - Isn't user removal
logged usually?

cu Andreas

#994836#20
Date:
2024-06-19 11:45:29 UTC
From:
To:
...

I think this is a problem specific to persistent schroots (those
that carry over on-disk state from one run to another), such as
type=directory. I saw a similar symptom in a schroot running a Debian
trixie derivative, on a Debian 12 host.

By default, as per /etc/schroot/default/nssdatabases and
/etc/schroot/sbuild/nssdatabases, schroot copies the host system's
/etc/passwd into the chroot during chroot setup. This means it should
be possible to reproduce the issue that Wookey reported like this:

- have a host system (virtual machine will be easiest) that has never
  had Exim installed, for example a fresh bookworm installation

- have a persistent schroot, like this one (it's a trixie derivative):

    [steamrt_steamrt5_amd64]
    description=Steam Runtime 5 development
    directory=/var/chroots/steamrt_steamrt5_amd64
    personality=linux
    groups=sudo
    root-groups=sudo
    preserve-environment=true
    type=directory

- in one schroot session, install Exim: it will add Debian-exim to the
  /etc/passwd in the chroot

- completely end the schroot session (reboot the machine, etc.)
  but keep the on-disk state in (in my case)
  /var/chroots/steamrt_steamrt5_amd64

- open a new schroot: schroot will overwrite
  /var/chroots/steamrt_steamrt5_amd64/etc/passwd with the /etc/passwd
  from the host, which does not contain Debian-exim

- try to run dpkg (for example I ran `apt autoremove`):
  dpkg sees that its statoverride database contains Debian-exim but
  /etc/passwd has no record of that user, and panics

Using transient schroots (those that are documented as implementing
"source chroot" options, such as type=file, type=btrfs-snapshot,
type=zfs-snapshot, type=lvm-snapshot) would likely avoid this problem,
by only copying /etc/passwd during creation of a new chroot session, and
not overwriting it during the lifetime of the session. The cost of this is
that if you *want* state to persist, that's harder to achieve.

I do not see any straightforward way that this could be addressed from
Exim's side, and an equivalent issue would affect any package that creates
a user or group dynamically and uses it for dpkg-statoverride; so maybe this
should be a schroot bug marked as affecting exim4, rather than an exim4 bug.
https://bugs.debian.org/499014 seems to be essentially the same bug, but
for dbus/messagebus rather than exim4/Debian-exim.

    smcv

#994836#25
Date:
2024-06-25 12:33:39 UTC
From:
To:
Control: reassign -1 schroot
Control: retitle -1 schroot: type=directory can make user IDs disappear, causing dpkg-statoverride fatal errors
Control: affects -1 + exim4-base src:exim4
...

Reassigning to schroot based on that reasoning.

    smcv

#994836#36
Date:
2024-07-17 10:54:41 UTC
From:
To:
I just ran into this issue.  Here's how I worked around it:

# dpkg-statoverride --list
root crontab 2755 /usr/bin/crontab
root Debian-exim 640 /etc/exim4/passwd.client

# dpkg-statoverride --remove /etc/exim4/passwd.client

# dpkg-statoverride --list
root crontab 2755 /usr/bin/crontab