Hi,

many piuparts bullseye2bookworm tests are curently failing with

0m50.4s ERROR: FAIL: After purging files have disappeared:
  /etc/systemd/system/multi-user.target.wants/	 not owned
  /etc/systemd/system/multi-user.target.wants/e2scrub_reap.service -> /lib/systemd/system/e2scrub_reap.service	 not owned

This is a new systemd unit in package e2fsprogs. If this failure is
actually e2fsprogs's fault by incorrectly using the helpers, please
reassign the bug there (with instructions how to do it correctly).

The following happens before the bug manifests.

Piupart generally uses --no-install-recommends, but that does not seem
to have an influence here.

The minimal bullseye chroot used by piuparts contains e2fsprogs but not
systemd. If I upgrade that chroot to bookworm,
  /etc/systemd/system/multi-user.target.wants/e2scrub_reap.service -> /lib/systemd/system/e2scrub_reap.service
gets created. e2fsprogs is still installed, systemd is still not
installed.

If I install systemd into the bullseye chroot and upgrade that to
bookworm, both systemd and e2fsprogs are still installed, but
  /etc/systemd/system/multi-user.target.wants/e2scrub_reap.service
does *NOT* get created.

piuparts then removes the tested packages and their deps from the chroot
and compares the result against the upgraded reference chroot.

I added some debug code to e2fsprogs.postinst (set -x,
_DEB_SYSTEMD_HELPER_DEBUG=1, listing multi-user.target.wants), and used
that version during the dist-upgrade, but there was no significant
difference in output between the two upgrade paths:

===== upgrade without systemd =====
  Setting up e2fsprogs (1.47.0-2+deb12anbe1) ...
  Installing new version of config file /etc/mke2fs.conf ...
  + set -e
  + [ -x /usr/sbin/update-initramfs -a -e /etc/initramfs-tools/initramfs.conf ]
  + export _DEB_SYSTEMD_HELPER_DEBUG=1
  + ls -la /etc/systemd/system/multi-user.target.wants
  ls: cannot access '/etc/systemd/system/multi-user.target.wants': No such file or directory
  + true
  + [ configure = configure ]
  + deb-systemd-helper unmask e2scrub_all.timer
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = unmask, scriptname = e2scrub_all.timer, service_path = /lib/systemd/system/e2scrub_all.timer
  (deb-systemd-helper DEBUG) rmdir_if_empty /var/lib/systemd/deb-systemd-helper-masked
  (deb-systemd-helper DEBUG) rmdir(/var/lib/systemd/deb-systemd-helper-masked) failed (No such file or directory)
  (deb-systemd-helper DEBUG) rmdir_if_empty /var/lib/systemd/deb-systemd-user-helper-masked
  (deb-systemd-helper DEBUG) rmdir(/var/lib/systemd/deb-systemd-user-helper-masked) failed (No such file or directory)
  + deb-systemd-helper --quiet was-enabled e2scrub_all.timer
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = was-enabled, scriptname = e2scrub_all.timer, service_path = /lib/systemd/system/e2scrub_all.timer
  (deb-systemd-helper DEBUG) Reading state file /var/lib/systemd/deb-systemd-helper-enabled/e2scrub_all.timer.dsh-also
  (deb-systemd-helper DEBUG) Contents: $VAR1 = [
            '/etc/systemd/system/timers.target.wants/e2scrub_all.timer'
          ];

  (deb-systemd-helper DEBUG) All links present, considering e2scrub_all.timer was-enabled.
  + deb-systemd-helper enable e2scrub_all.timer
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = enable, scriptname = e2scrub_all.timer, service_path = /lib/systemd/system/e2scrub_all.timer
  (deb-systemd-helper DEBUG) Renaming temp file /var/lib/systemd/deb-systemd-helper-enabled/.statedbM8J.tmp to state file /var/lib/systemd/deb-systemd-helper-enabled/e2scrub_all.timer.dsh-also
  + [ configure = configure ]
  + deb-systemd-helper unmask e2scrub_reap.service
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = unmask, scriptname = e2scrub_reap.service, service_path = /lib/systemd/system/e2scrub_reap.service
  (deb-systemd-helper DEBUG) rmdir_if_empty /var/lib/systemd/deb-systemd-helper-masked
  (deb-systemd-helper DEBUG) rmdir(/var/lib/systemd/deb-systemd-helper-masked) failed (No such file or directory)
  (deb-systemd-helper DEBUG) rmdir_if_empty /var/lib/systemd/deb-systemd-user-helper-masked
  (deb-systemd-helper DEBUG) rmdir(/var/lib/systemd/deb-systemd-user-helper-masked) failed (No such file or directory)
  + deb-systemd-helper --quiet was-enabled e2scrub_reap.service
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = was-enabled, scriptname = e2scrub_reap.service, service_path = /lib/systemd/system/e2scrub_reap.service
  (deb-systemd-helper DEBUG) Reading state file /var/lib/systemd/deb-systemd-helper-enabled/e2scrub_reap.service.dsh-also
  (deb-systemd-helper DEBUG) Contents: $VAR1 = [
            '/etc/systemd/system/default.target.wants/e2scrub_reap.service'
          ];

  (deb-systemd-helper DEBUG) All links present, considering e2scrub_reap.service was-enabled.
  + deb-systemd-helper enable e2scrub_reap.service
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = enable, scriptname = e2scrub_reap.service, service_path = /lib/systemd/system/e2scrub_reap.service
  (deb-systemd-helper DEBUG) Renaming temp file /var/lib/systemd/deb-systemd-helper-enabled/.state7t5xW.tmp to state file /var/lib/systemd/deb-systemd-helper-enabled/e2scrub_reap.service.dsh-also
  + [ configure = configure ]
  + [ -d /run/systemd/system ]
  + ls -la /etc/systemd/system/multi-user.target.wants
  total 0
  drwxr-xr-x 2 root root  60 May  5 07:15 .
  drwxr-xr-x 5 root root 100 May  5 07:15 ..
  lrwxrwxrwx 1 root root  40 May  5 07:15 e2scrub_reap.service -> /lib/systemd/system/e2scrub_reap.service
  + exit 0
==========

===== upgrade with systemd =====
  Setting up e2fsprogs (1.47.0-2+deb12anbe1) ...
  Installing new version of config file /etc/mke2fs.conf ...
  + set -e
  + [ -x /usr/sbin/update-initramfs -a -e /etc/initramfs-tools/initramfs.conf ]
  + export _DEB_SYSTEMD_HELPER_DEBUG=1
  + ls -la /etc/systemd/system/multi-user.target.wants
  total 0
  drwxr-xr-x 2 root root  60 May  5 07:15 .
  drwxr-xr-x 7 root root 140 May  5 07:15 ..
  lrwxrwxrwx 1 root root  36 May  5 07:15 remote-fs.target -> /lib/systemd/system/remote-fs.target
  + [ configure = configure ]
  + deb-systemd-helper unmask e2scrub_all.timer
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = unmask, scriptname = e2scrub_all.timer, service_path = /lib/systemd/system/e2scrub_all.timer
  (deb-systemd-helper DEBUG) rmdir_if_empty /var/lib/systemd/deb-systemd-helper-masked
  (deb-systemd-helper DEBUG) rmdir(/var/lib/systemd/deb-systemd-helper-masked) failed (No such file or directory)
  (deb-systemd-helper DEBUG) rmdir_if_empty /var/lib/systemd/deb-systemd-user-helper-masked
  (deb-systemd-helper DEBUG) rmdir(/var/lib/systemd/deb-systemd-user-helper-masked) failed (No such file or directory)
  + deb-systemd-helper --quiet was-enabled e2scrub_all.timer
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = was-enabled, scriptname = e2scrub_all.timer, service_path = /lib/systemd/system/e2scrub_all.timer
  (deb-systemd-helper DEBUG) Reading state file /var/lib/systemd/deb-systemd-helper-enabled/e2scrub_all.timer.dsh-also
  (deb-systemd-helper DEBUG) Contents: $VAR1 = [
            '/etc/systemd/system/timers.target.wants/e2scrub_all.timer'
          ];

  (deb-systemd-helper DEBUG) All links present, considering e2scrub_all.timer was-enabled.
  + deb-systemd-helper enable e2scrub_all.timer
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = enable, scriptname = e2scrub_all.timer, service_path = /lib/systemd/system/e2scrub_all.timer
  (deb-systemd-helper DEBUG) Renaming temp file /var/lib/systemd/deb-systemd-helper-enabled/.staterx1xW.tmp to state file /var/lib/systemd/deb-systemd-helper-enabled/e2scrub_all.timer.dsh-also
  + [ configure = configure ]
  + deb-systemd-helper unmask e2scrub_reap.service
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = unmask, scriptname = e2scrub_reap.service, service_path = /lib/systemd/system/e2scrub_reap.service
  (deb-systemd-helper DEBUG) rmdir_if_empty /var/lib/systemd/deb-systemd-helper-masked
  (deb-systemd-helper DEBUG) rmdir(/var/lib/systemd/deb-systemd-helper-masked) failed (No such file or directory)
  (deb-systemd-helper DEBUG) rmdir_if_empty /var/lib/systemd/deb-systemd-user-helper-masked
  (deb-systemd-helper DEBUG) rmdir(/var/lib/systemd/deb-systemd-user-helper-masked) failed (No such file or directory)
  + deb-systemd-helper --quiet was-enabled e2scrub_reap.service
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = was-enabled, scriptname = e2scrub_reap.service, service_path = /lib/systemd/system/e2scrub_reap.service
  (deb-systemd-helper DEBUG) Reading state file /var/lib/systemd/deb-systemd-helper-enabled/e2scrub_reap.service.dsh-also
  (deb-systemd-helper DEBUG) Contents: $VAR1 = [
            '/etc/systemd/system/default.target.wants/e2scrub_reap.service'
          ];

  (deb-systemd-helper DEBUG) All links present, considering e2scrub_reap.service was-enabled.
  + deb-systemd-helper enable e2scrub_reap.service
  (deb-systemd-helper DEBUG) is purge = no
  (deb-systemd-helper DEBUG) action = enable, scriptname = e2scrub_reap.service, service_path = /lib/systemd/system/e2scrub_reap.service
  (deb-systemd-helper DEBUG) Renaming temp file /var/lib/systemd/deb-systemd-helper-enabled/.stateHcfa1.tmp to state file /var/lib/systemd/deb-systemd-helper-enabled/e2scrub_reap.service.dsh-also
  + [ configure = configure ]
  + [ -d /run/systemd/system ]
  + ls -la /etc/systemd/system/multi-user.target.wants
  total 0
  drwxr-xr-x 2 root root  60 May  5 07:15 .
  drwxr-xr-x 7 root root 140 May  5 07:15 ..
  lrwxrwxrwx 1 root root  36 May  5 07:15 remote-fs.target -> /lib/systemd/system/remote-fs.target
  + exit 0
==========

I tried varying the upgrade order (in the normal upgrade path e2fsprogs
got configured while the new systemd was unpacked but not yet
configured), but that didn't change the outcome.
I tried installing the new systemd first, then performing the
dist-upgrade. And also installing the new e2fsprogs first, then
upgrading the rest of the chroot. In both cases
/etc/systemd/system/multi-user.target.wants/e2scrub_reap.service was
missing.


I'm filing this bug as serious as it looks like systemd units my not get
enabled on upgrades from bullseye to bookworm while they should be.

I'm attaching the full piuparts log in case you are interested in which
packages were involved in the upgrade.
(First part if the upgrade of the reference chroot, then the upgrade
after installing systemd).


Andreas

/lib/systemd/system/e2scrub_reap.service         not owned

Isn't this a variation of
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1031695 ?

Am 05.05.23 um 11:14 schrieb Luca Boccassi:

It's not, both bullseye and bookworm ship this file in e2fsprogs. The
difference is this

bullseye:
cat /lib/systemd/system/e2scrub_reap.service | tail -n 2
[Install]
WantedBy=default.target

bookworm
# cat /lib/systemd/system/e2scrub_reap.service | tail -n 2
[Install]
WantedBy=multi-user.target


I.e. it changed the target from default.target to multi-user.target.
i-s-h does not have support for removing obsolete enablement symlinks in
such a case.


  If this failure is

Doesn't look like it.

Andreas, you filed this with RC severity with the justification that
systemd units are not enabled. I don't see that, so could you please
clarify.

What I could find out so far is the change of WantedBy target not being
properly cleaned up on upgrades, but that doesn't strike me as RC (and
would need to be done in e2fsprogs)


Michael

Yes, looks like this is in the e2fsprogs realm.

Please reassign it there together with instructions how to fix it, i.e.
what should be done in the maintainer scripts.

IMO the expected outcome from e2fsprogs perspective should be identical
system configuration (w.r.t. enabled systemd units) after both fresh
install and upgrade from bullseye, regardless of the installation status
of systemd.
If e2fsprogs doesn't care about this, it can downgrade the severity.


Andreas

Am 14.05.23 um 11:30 schrieb Andreas Beckmann:

Can someone send the instructions on how to fix this?

I'm always amused by people who claim systemd is "simpler" to
understand than init.d scripts.  :-)

I have no clue what's going on; is default.target obsolete?  If we
change the Wanted-by in a systemd unit file, what magic incantation is
required?  Can someone point me at documentation that describes all of
this?

Thanks,

					- Ted

I'm having trouble reconciling these two log lines during the upgrade without
systemd:

  ls: cannot access '/etc/systemd/system/multi-user.target.wants': No such file or directory
  ...
  (deb-systemd-helper DEBUG) All links present, considering e2scrub_reap.service was-enabled.

Could those indicate some kind of inconsistency/bug?

I think we want to remove the old default.target.wants directory link
and replace it with a multi-user.target.wants link at some point
during the upgrade process.

Would calling the 'reenable' action implemented by
deb-systemd-helper[1] (an equivalent to the corresponding action in
systemctl[2]) during the e2fsprogs postinst script solve the problem?

(the contents of the deb-systemd-helper service state file seem very
relevant here.  for this to work correctly, I think it needs to
contain the old link during the 'disable' step, and then should use
the new link during 'enable'.  I could be mistaken, however.  I have
read #717603 while trying to figure out a solution here)

[1] - https://manpages.debian.org/bullseye/init-system-helpers/deb-systemd-helper.1p.en.html

[2] - https://manpages.debian.org/bullseye/systemd/systemctl.1.en.html

Hi,

I sat down with Jochen in Hamburg to try and fix this.

We wish we could give you. Instead, we document our findings, so maybe the
next one looking into this bug has a better idea, but for now we give up
as it is too late for bookworm anyway.

As Michael pointed out, i-s-h does not support removing a WantedBy from
a unit. Arguably, this is a bug in i-s-h, but we wouldn't be changing
i-s-h at this time of the freeze.

In the upgrade scenario, debhelper inserts code that becomes d-s-h
unmask; d-s-h was-enabled; d-s-h enable. None of that code is able to
clean up the old link (in default.target). When working in a chroot, it
is sufficient to do:

    d-s-h unmkas
    if d-s-h was-enabled
        d-s-h disable        # added
        d-s-h update-state   # added
        d-s-h enable
    else
        d-s-h update-state

The disable would remove the old link from /etc. The update-state would
clean it from /var and then things would just work. However, when
running this in a VM with systemd running, the unit would then end up as
disabled, which is not what we want. Comparing the debug output from
d-s-h, we could not figure out why it ends up disabling the unit.

So all we can do now, is document this and attach the dysfunctional
patch (please don't apply).

Helmut and Jochen

Helmut, Jochem, thanks so much for trying to look into this.  Here's
some additional context from my research.

First of all, the change to use WantedBy=default.target to
Wantedby-multi-user.target, as described in Message #19 of this bug,
was in response to a bug report from Ansgar, bug report #991349:

At the time, I thought to myself, sure, makes sense, and made the
change in commit b42c9788c75d ("e2scrub: use
WantedBy=multi-user.target in e2scrub_reap.service"), and in the
commit I noted "Addresses-Debian-Bug: #991349"

As near as I can tell, on a system that started with the Bullseye
version of e2fsprogs, and which has then updated to the Bookform
version e2fsprogs, via periodic updates to testing (Bookworm), the
default.target link still exists:

% ls -l /etc/systemd/system/default.target.wants/e2scrub_reap.service
0 lrwxrwxrwx 1 root root 40 Dec 19  2020 /etc/systemd/system/default.target.wants/e2scrub_reap.service -> /lib/systemd/system/e2scrub_reap.service

... and this is enough for systemctl status to seem to think that
e2scrub_reap is still enabled:

% systemctl status e2scrub_reap
○ e2scrub_reap.service - Remove Stale Online ext4 Metadata Check Snapshots
     Loaded: loaded (/lib/systemd/system/e2scrub_reap.service; enabled; preset: enable>
     Active: inactive (dead) since Sat 2023-05-27 17:53:22 EDT; 1h 34min ago
       Docs: man:e2scrub_all(8)
    Process: 1309 ExecStart=/sbin/e2scrub_all -A -r (code=exited, status=0/SUCCESS)
   Main PID: 1309 (code=exited, status=0/SUCCESS)
        CPU: 12ms
 ...

So sure, /etc/systemd.d/system/multi-user.target.wants/e2scrub_reap.service
doesn't exist.  *But* it still exists in .../default.target.wants/...
which seems to be enough to keep the e2scrub_reap service enabled.  Right?

What am I missing?

In any case, I am still unclear (a) what is actually broken in this
particular setup, since according to systemctl status the systemd unit
is apparently still appropriate enabled, even if it isn't via the
expected Wanted-b: multi-user.target.

And secondly, (b) what is e2fsprogs's control scripts supposed to have
done differently?  That is, if this is indeed this is a bug in
e2fsprogs --- what did I do wrong, and how do I fix it?

And if the answer is you should never, ever, try to change a Wanted-by
line in a systemd script, because debian's systemd unit file
infrastructure is too fragile to handle this correctly, given that
bookworm is about to ship with "Wanted-by: multi-user.target", what's
the best path forward at this point?

I'll note that e2scrub_reap.service is just a helper unit file which
is only needed to clean up after a system crash while e2scrub is
running --- and that will only happen if the user has edited and
appropriately configured e2scrub in /etc/e2scrub.conf.

So from my maintainer's perspective, what I am going for is that
e2scrub_reap.service and e2scrub_all.timer should *always* be enabled,
since the real control point (as far as I am concerned) is
/etc/e2scrub.conf.  I really don't actually *care* whether it is
enabled via default.target.wanted or multi-user.target.wanted.

If I need to be sent to some systemd re-educational camp to understand
the finer points about default.target vs multi-user.target, and
whether it acctually makes any difference whether the systemd unit
file says "Wanted-by: multi-user.target", but in the upgraded
bullseye->bookworm installation, the symlink is still in
*/default.target.wanted/* --- please point me at the documentation.

Otherwise, I'm beginning to think that nothing is actually broken, and
the bullseye2bookworm piuparts tests is just being overly picky, but
nothing is actually broken in actual practice.  And perhaps I should
just close this bug as "Working as Intended".

Again, what am I missing?

					- Ted

P.S.  I really *am* trying to get with the systemd program, but this
all of this complexity is just hopelessly confusing.  :-( :-( :-(

P.P.S.  And there is actually a case where this will actually break a
real user, can someone give me clear reproduction instructions, which
starts with "install bookworm in a VM", then do X, then do Y, and then
observe breakage.  Please don't just point me at piuparts output,
because unfortunately, it's too complicated for my tiny brain to
understand.

Hi Ted,

* Theodore Ts'o <tytso@mit.edu> [2023-05-27 19:45]:

Yes, that's fine.

The point of piuparts is that an upgraded system is different to a newly
installed system, i.e. that the e2scrub_reap.service symlink lies in a
different directory.

Arguably nothing and init-system-helpers/dh_installsystemd should detect
the change and move the symlink.
In general there are two things. One is to fix the immediate problem
this issue is about and we can still do that in a point release.
The other one is to have general support for changing Wanted-by: (or
state that it is not supported). I would propose to ask the
init-system-helpers/dh_installsystemd maintainers for a general solution
and then apply that for a bookworm point release.

Cheers Jochen

Actually the difference is between the minimal bullseye chroot upgraded
to bookworm and the bullseye chroot with some packages to be tested
installed (here: systemd) and upgraded to bookworm. Ideally, after
removing the packages to be tested and their dependencies, the two
bookworm chroots should be identical ...


Andreas

I've been 'approximately' testing this locally on bookworm by:

  * Editing the Install.WantedBy in /lib/systemd/system/e2scrub_reap.service

  * Reconfiguring the package using 'dpkg-reconfigure e2fsprogs'

(I know, it's not a comprehensive workflow - but I think that it calls the
relevant deb-systemd-helper and e2fsprogs postinst script sections)

Also: Marcin's patch[1] from #985787 is also intended to fix a very similar
problem (perhaps exactly the same issue).

Some puzzles:

  * Why does the 'deb-systemd-helper disable' invocation not work (as found
    by Helmut and Jochen)?  It seems like it should read the symlink path to
    remove from the dsh-also state file, so the Install.WantedBy change should
    not affect it.

  * Is the /var/lib/systemd/deb-systemd-helper-enabled/ path relevant?  This
    seems to contain a shadow copy of much of the /etc/systemd/system service
    state.

   * Is the 'create links unless no links installed' logic correct?  (that
     sounds like it could be incorrect, but I'm not sure)


I did manage to get something kinda-working locally with a combination of an
'update-state' call and Marcin's patch.  However, I'd like to understand more
about the 'deb-systemd-helper disable' call failure before recommending that.


And, quoting Andreas:

I agree on the goals there.  Being unhappy about systemd and maintaining a
package that has divergent on-filesystem results depending on how users
upgraded seems distinctly worse than being unhappy about systemd while
maintaining a consistently-deployed package.

[1] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=985787#25

[2] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035543#62

Would a 'move /etc/systemd/system/default.target.wants/e2scrub_reap.service
to /etc/systemd/system/multi-user.target.wants/e2scrub_reap.service if the
source file exists' during the postinst be enough to restore consistency here,
and satisfy piuparts?

If waiting to the point-release is better, then I'm OK with that.  I have some
slight frustration about the path divergence - because I think consistent
results are important for robust packaging - but if there's higher risk of
breakage or a sense that it's too late to fix for bookworm, I can accept that.

The title of this bug report is highly misleading.

Let's see what's actually happening here:


bullseye chroot:
# find /etc/systemd/system/ -name e2scrub_reap.service
/etc/systemd/system/default.target.wants/e2scrub_reap.service

bookworm chroot:
# find /etc/systemd/system/ -name e2scrub_reap.service
/etc/systemd/system/multi-user.target.wants/e2scrub_reap.service

bullseye chroot upgraded to bookworm:
# find /etc/systemd/system/ -name e2scrub_reap.service
/etc/systemd/system/multi-user.target.wants/e2scrub_reap.service
/etc/systemd/system/default.target.wants/e2scrub_reap.service


So, you have the service enabled both in default.target and
multi-user.target. Does this pose any actual problem? Not really.
It is properly started during boot and if you purge the package, both
symlinks are removed. It is thus imho not a serious issue, but something
you might want to fix at it is a bit confusing.

That said, I find the piuparts output not very readable, so there might
be another issue I don't see. In this case, Andreas needs to clarify the
RC status of this bug.


So what I would suggest is a
if dpkg --compare-versions "$2" lt "$ver of your next upload"; then
   rm -f /etc/systemd/system/default.target.wants/e2scrub_reap.service
fi
in your postinst to clean up the stray enablement symlink.

Michael

Hi Michael,

* Michael Biebl <biebl@debian.org> [2023-05-30 13:23]:

But when you use a VM:

$ debvm-create -r bullseye
$ debvm-run
# apt install e2fsprogs

# find / -name e2scrub_reap.service
/var/lib/systemd/deb-systemd-helper-enabled/default.target.wants/e2scrub_reap.service
/lib/systemd/system/e2scrub_reap.service
/etc/systemd/system/default.target.wants/e2scrub_reap.service

# cat /var/lib/systemd/deb-systemd-helper-enabled/e2scrub_reap.service.dsh-also
/etc/systemd/system/default.target.wants/e2scrub_reap.service

# sed -i 's/bullseye/bookworm/' /etc/apt/sources.list
# apt update
# apt dist-upgrade

# find / -name e2scrub_reap.service
/var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants/e2scrub_reap.service
/var/lib/systemd/deb-systemd-helper-enabled/default.target.wants/e2scrub_reap.service
/usr/lib/systemd/system/e2scrub_reap.service
/etc/systemd/system/default.target.wants/e2scrub_reap.service

# cat /var/lib/systemd/deb-systemd-helper-enabled/e2scrub_reap.service.dsh-also
/etc/systemd/system/default.target.wants/e2scrub_reap.service
/etc/systemd/system/multi-user.target.wants/e2scrub_reap.service

I think this would disable the service for users that upgraded from
bullseye as shown above.

Cheers Jochen

[trimmed the list of CCs]

Am 30.05.23 um 13:36 schrieb Jochen Sprickerhof:

[...]


Interesting. Will check.

Followup-For: Bug #1035543

Is there a way to pause and step through the postinst script for a package when
it runs? (similar to an interactive debugging session)

Followup-For: Bug #1035543

There is/was-and-may-be-again bashdb[1]; it was removed[2] from Debian in Y2017.

[1] - https://bashdb.sourceforge.net

[2] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=870992

[ re-introducing the larger cc list audience, plus debian-release ]

Would reverting the Install.WantedBy modification[1][2], restoring e2scrub_reap
enablement using 'default.target' on relevant systems, be a sensible approach
for bookworm until we can figure out the debhelper-system behaviour when that
setting changes?

[1] - https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/commit/?id=b42c9788c75de10ee3b6d1a7f9fbc2f529b64889

[2] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991349

* James Addison <jay@jp-hosting.net> [2023-06-01 12:44]:

No, bookworm is frozen and done:

"In the last week prior to the freeze, testing will be completely
frozen and only emergency bug fixes will be considered in this period."

https://lists.debian.org/debian-devel-announce/2023/04/msg00007.html

Cheers Jochen

Package: e2fsprogs
Followup-For: Bug #1035543
Control: tags -1 patch

Package: e2fsprogs
Followup-For: Bug #1035543
X-Debbugs-Cc: jspricke@debian.org

Drats - but also, thank you for confirming that.  I'd read the announcement, and it is
clear, but then somehow afterwards forgot, and thought that the relevant
closing date was one week before the release itself.  I'll return to this after
bookworm is out and until then will try to relax and perhaps may even be able to
de-serious myself temporarily to allow 'celebration'.  Anyway, a revert patch is
attached, although naturally it'd be better to determine the cause.

In addition to Bookworm being hard frozen, I question the importance
of this patch, the bug priority, and whether the title is correct.
After all, at least with respect to e2fsprogs systemd unit *will*
still be enabled.  It will just be enabled using
../multi-user.target/wanted instead of ../default.target/wanted.

Ok, piuparts whines about it, and I agree that it's ideal if things
are the stame regardless of whether the distro is freshly installed or
uprgaded --- but e2scrub-repeat.service will *still* be enabled,
right?  And so the bug title is misleaing, right?

So it's not a big deal; is that correct so this patch is not worth
trying to shoehorn in beform Bookworm ships, and this particular patch
can be safely downgraded from importanted, right?

Or else, what am I missing?

Thanks,

						- Ted

P.S.  And repeat after me, "systemd is *so* much more simpler than
init.d scripts because it's declarative...."

Also, arguing against my own revert-patch: I think it could be said
that multi-user is the "better" target to use here, because the
default could be "graphical" or some later-reached system state
whereas this is a relatively low-level (if small) system cleanup
service.

It's taken me too long to figure all this out, but retrospectively I
suppose my preference order here was:

1. Fix comprehensively in deb-systemd-helper.
2. Revert the original change (but I thought of / suggested that too late).
3. Apply a package-specific workaround (but I didn't find one that I
was comfortable with suggesting).

I'd agree with downgrading the severity to below RC (and it's your
package, so feel free, I think?  maybe even close it?).  If anyone
arrives here/reports other bugs as a result of experiencing it, I
think we can let them know that it's safe to remove the legacy
'default.target' symlink (does that sound correct too?).

Without getting too much into opinions on systemd itself: I think
declarative systems are good, but that managing stateful transitions
between multiple declared states can be challenging.  Not impossible,
and with sufficient bugreporting and fixing, it can be made solid - so
that's something to continue on after the release.

(and yep, I claimed I wouldn't look at this bug again for a while..
and I was tempted not to.. but I think clearing it from the RC queue
is probably worthwhile)

severity 1035543 normal
retitle 1035543 e2fsprogs: on an upgrade from bullseye e2scrub-reap.service may be wanted by default.target instead of multi-user.target
thanks

OK, I've reset this to normal.

Right, that's I believe the point of bug #991349; it's possible that
the system adminsitrator might manually set default.target to point to
graphical.target, per [1].  And since multi-user.target is a subset of
graphical.target, it makes sense to make the Wanted-by to be
multi-user.target.

[1] https://www.baeldung.com/linux/systemd-target-multi-user

And so it's fair to keep this bug open, but I think it makes it a
normal bug, with the resumption that hopefully it can be fixed via
deb-systemd-helper or init-system-helpers.

In this particular case, since we *always* want it to be
default.target, since the whole *point* is to clean up after a failed
e2scrub, it seems really unlikely to me that the system administrator
would change this.  So this is one where it's probably fair for the
postinstall script to just fix the wanted-by link **always** if the
the systemd unit file says Wanted-by: default.target, and the symlink
is inconsistent with it.

Maybe this will mess with some kind of hidden systemd rules of the
road, but quite frankly, the fact that this is causing so much pain
and confusion, I'm not sure I care.

I think so.  The symlink should *never* be considered normative,
right?  The unit file should be considered the single source of truth.
If the system administrator wants to do something weird, they are
supposed to copy /lib/systemd/system/e2scrub_reap.service to
/etc/systemd/system/e2scrub_reap.service, and if the symlink is
inconsistent with what's in the unit file (with the one in
/etc/systemd/...  if it exists taking precedence over the one in
/lib/systemd/..., we should just fix the symlink.

So this should be fixable in deb-systemd-helper, or I can just
implement a Big Fat Hammer in e2fsprogs's postinst script.

Does that sound right?

						- Ted

systemd.special[1] does indicate that the target it will load at boot
is default.target -- whatever it's configured to (usually multi-user
or graphical).

In other words: default.target should be a subset of all of
multi-user.target, graphical.target, and whatever else a system
administrator might configure as the boot target state.

What do you mean by "fix the link" in this context?

[1] - https://manpages.debian.org/bullseye/systemd/systemd.special.7.en.html

targets, but for any given systemd-enabled boot environment,
default.target should be guaranteed (as far as systemd lives up to the
guarantees it aims to provide) to be on the service startup path
between systemd loading as init and systemd reaching the desired boot
state.

After further thought here: if the change from #991349 to use
multi-user.target means slightly earlier invocation of the service when
working towards graphical.target, but _also_ means that the service won't
be invoked _at all_ on some systemd-enabled systems (whenever
default.target is configured to a state that doesn't visit
multi-user.target)... then I think rolling back to default.target would be
better.

Am 01.06.23 um 13:44 schrieb James Addison:

Nah, I think that would cause more churn, going back and forth (using
multi-user.target is the correct thing to do).
I have a pretty good idea how to address this but want to double check
it, so please just be patient.
There is no reason to rush this. As already mentioned, this is more a
cosmetic issue then a functional issue (at least in this specific
instance). Keeping piuparts happy is certainly important but not the
only deciding factor.

Regards,
Michael