Hi,

when using ssh as a socket activated service (systemctl stop/disable
ssh.service, systemctl enable/start ssh.socket), after a library update
needrestart will offer to restart ssh.service. This fails since port 22
is occupied by the instance services and causes the machine to be
without listening process after logging out.

A possible workaround is masking ssh.service, see #1001320.

Restarting services...
 systemctl restart console-log.service cron.service exim4.service haveged.service ippl.service ntp.service rsyslog.service serial-getty@ttyS0.service ssh.service systemd-journald.service systemd-networkd.service systemd-resolved.service systemd-udevd.service
Job for ssh.service failed because the control process exited with error code.
See "systemctl status ssh.service" and "journalctl -xeu ssh.service" for details.
Service restarts being deferred:
 /etc/needrestart/restart.d/dbus.service
 systemctl restart getty@tty1.service
 systemctl restart systemd-logind.service
 systemctl restart user@1001.service

and the following log entries:
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopping LSB: Puts a logfile pager on virtual consoles...
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopping Regular background program processing daemon...
Dec  8 12:58:26 emptybookworm82 systemd[1]: cron.service: Deactivated successfully.
Dec  8 12:58:26 emptybookworm82 cron[429258]: (CRON) INFO (pidfile fd = 3)
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopped Regular background program processing daemon.
Dec  8 12:58:26 emptybookworm82 systemd[1]: cron.service: Consumed 15min 4.856s CPU time.
Dec  8 12:58:26 emptybookworm82 systemd[1]: Started Regular background program processing daemon.
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopping LSB: exim Mail Transport Agent...
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopping Entropy Daemon based on the HAVEGE algorithm...
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopping LSB: IP protocols logger...
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopping Network Time Service...
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopping System Logging Service...
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopping Serial Getty on ttyS0...
Dec  8 12:58:26 emptybookworm82 systemd[1]: serial-getty@ttyS0.service: Deactivated successfully.
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopped Serial Getty on ttyS0.
Dec  8 12:58:26 emptybookworm82 systemd[1]: Started Serial Getty on ttyS0.
Dec  8 12:58:26 emptybookworm82 systemd[1]: ssh.socket: Deactivated successfully.
Dec  8 12:58:26 emptybookworm82 systemd[1]: Closed OpenBSD Secure Shell server socket.
Dec  8 12:58:26 emptybookworm82 systemd[1]: ssh.socket: Consumed 10.571s CPU time.
Dec  8 12:58:26 emptybookworm82 systemd[1]: Starting OpenBSD Secure Shell server...
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopping Flush Journal to Persistent Storage...
Dec  8 12:58:26 emptybookworm82 systemd[1]: systemd-networkd-wait-online.service: Deactivated successfully.
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopped Wait for Network to be Configured.
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopping Wait for Network to be Configured...
Dec  8 12:58:26 emptybookworm82 systemd[1]: Stopping Network Name Resolution...
Dec  8 12:58:26 emptybookworm82 systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION
Dec  8 12:58:26 emptybookworm82 systemd[1]: ssh.service: Failed with result 'exit-code'.
Dec  8 12:58:26 emptybookworm82 systemd[1]: Failed to start OpenBSD Secure Shell server.
Dec  8 12:58:26 emptybookworm82 ntpd[298]: ntpd exiting on signal 15 (Terminated)
Dec  8 12:58:26 emptybookworm82 ntpd[298]: 2a01:4f8:140:246a::2 local addr 2a01:4f8:140:246a::52:100 -> <null>
Dec  8 12:58:26 emptybookworm82 haveged[220]: haveged: Stopping due to signal 15
Dec  8 12:58:27 emptybookworm82 cron[429258]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
Dec  8 12:58:27 emptybookworm82 systemd[1]: systemd-journal-flush.service: Deactivated successfully.
Dec  8 12:58:27 emptybookworm82 systemd[1]: Stopped Flush Journal to Persistent Storage.
Dec  8 12:58:27 emptybookworm82 exim4[429259]:  exim4_listener.

Here is what Timo Weingärtner found out in relation to my bug report against sshd:

| To me it looks like a problem in needrestart. The (forked off) sshd process
| handling your client connection belongs to cgroup session-NN.scope, no matter
| if it was started by systemd socket activation or regular sshd.
|
| needrestart (invoked with "-vlp" here) detects a process with outdated libs:
|
| [main] #2111961 uses deleted /lib/x86_64-linux-gnu/libnss_files-2.32.so
| [main] #2111961 is a child of #2111904
|
| Then it figures out the binary and the cgroup:
|
| [main] #2111961 exe => /usr/sbin/sshd
| [main] trying systemctl status
|
| cgroup detection didn't work, so:
|
| [main] #2111961 running /etc/needrestart/hook.d/10-dpkg
| [main] #2111961 package: openssh-server
| [main] #2111961 running /etc/needrestart/hook.d/20-rpm
| [main] #2111961 running /etc/needrestart/hook.d/90-none
|
| /etc/needrestart/hook.d/10-dpkg also finds /etc/init.d/ssh and we end up with:
|
| Services:
| […]
| - spamassassin.service
| - ssh
| - systemd-journald.service
| […]
|
| Note the missing ".service". Then you have it invoke "systemctl restart
| ssh.service" and voilà.
|
| A workaround might be masking ssh.service.

A per-client process should not get restarted by needrestart. This might
happen even if ssh.service is running, but it is usually harmless.

You might find more information in #1001320 regarding ssh.

Greetings
Marc, thanking Timo for his insights and help

Hi Marc,


could you please provide the content of /proc/$PID/cgroup for an socket
activated sshd instance?

As a workaround you might blacklist sshd in needrestart but I think a
generic approach handling socket activation services in needrestart
would be better. Therefore needrestart need a way to detect if the
process belongs to a socket activated service.


TIA & HTH,
Thomas

Hi Marc,


could you please provide the content of /proc/$PID/cgroup for an socket
activated sshd instance?

As a workaround you might blacklist sshd in needrestart but I think a
generic approach handling socket activation services in needrestart
would be better. Therefore needrestart need a way to detect if the
process belongs to a socket activated service.


TIA & HTH,
Thomas

Hi Thomas,
1 [1/4996]mh@torres:~ $ pgrep ssh
315675
315738
[2/4997]mh@torres:~ $ sudo cat /proc/315675/cgroup
[sudo] password for mh on torres:
0::/user.slice/user-1001.slice/session-296.scope
[3/4998]mh@torres:~ $ sudo cat /proc/315738/cgroup
0::/user.slice/user-1001.slice/session-296.scope
[4/4999]mh@torres:~ $

It is also possible to mask ssh.service entirely in systemd. But of
couse having the heuristic fixed would be better.

Greetings
Marc

Hi Marc,

thanks! Needrestart should ignore those ssh instances since there is a
user slice cgroup. It does not work due to this check[1] in
needrestart.

[1] https://github.com/liske/needrestart/blob/v3.5/needrestart#L637

Looks like a systemd/cgroup related change in bullseye, buster seems
not to be affected.


Regards,
Thomas

That sounds correct, I had attributed this behavior to some ssh updates
in the past.

Greetings
Marc

Es gibt eine Familienspende in Höhe von 1.850.000,00 USD von Cheng Charlie
Saephan. Bitte antworten Sie für weitere Informationen. Denken Sie daran,
Ihrer Familie und den Bedürftigen in Ihrer Umgebung Gutes zu tun.

Dies ist bereits der zweite Versuch, Sie zu erreichen. Bitte antworten Sie
für weitere Details.