Hi,
I've seen this  in package libvirt in Ubuntu as well as Debian, but it
might be more widespread without me being aware of it. If we install
the package all is fine, but on a --reinstall or upgrade it will show
warnings about services like:

Setting up libvirt-daemon-system (6.0.0-0ubuntu6) ...
Job failed. See "journalctl -xe" for details.

It comes down to:
  systemd[1]: libvirtd-ro.socket: Socket service libvirtd.service
already active, refusing.
but I was tracking that down for [1] and it is more complex than just that.
I found the ordering of sockets mentioned as "Also" to be the reason,
let me outline the details ...

libvirtd.service has a few related sockets.
$ grep Also /lib/systemd/system/libvirtd.service
Also=virtlockd.socket
Also=virtlogd.socket
Also=libvirtd.socket
Also=libvirtd-ro.socket

Due to that the following entry in d/rules:
  dh_installsystemd -p libvirt-daemon-system --restart-after-upgrade
libvirtd.service

Gets generated into postinst automatically pulling in all the "Also" statements.
But also ordering them alphabetically.

if [ -n "$2" ]; then
    _dh_action=restart
else
    _dh_action=start
fi
deb-systemd-invoke $_dh_action 'libvirtd-ro.socket' 'libvirtd.service'
'libvirtd.socket' 'virtlockd.socket' 'virtlogd.socket' >/dev/null ||
true

Due to that the "libvirtd-ro.socket" is restarted before the service
and fails as the service is already up. The sockets listed later work
fine.

The following order would work fine (service first, then sockets):
 $ deb-systemd-invoke restart 'libvirtd.service' 'libvirtd-ro.socket'
'libvirtd.socket' 'virtlockd.socket' 'virtlogd.socket
(but IMHO that might be racy still).

I was wondering if dh_installsystemd should maybe:
A) always order the service it was called with as first entry?
B) not pull in "ALSO" statements if they are sockets?
C) add an option to influence that behavior as needed per-package?
D) ... ?

I'm a bit lost on what the best option might be, therefore I wanted to
open this bug for a discussion.

[1]: https://bugs.launchpad.net/ubuntu/+source/debhelper/+bug/1869796

Hi Christian,
found the problem is not the ordering but rather the relationship
between libvird.socket, libvirtd-ro.socket and libvirtd.service.

If you e.g. restart libvirtd.socket (which is what happens in the
postinst) yet get:

$ sudo systemctl restart libvirtd.socket
Job failed. See "journalctl -xe" for details.

and in the journal:
Apr 01 14:12:07 foo systemd[1]: libvirtd.socket: Socket service libvirtd.service already active, refusing.
Apr 01 14:12:07 foo systemd[1]: Failed to listen on Libvirt local socket.

But if you do:

$ sudo systemctl stop libvirtd.service libvirtd-ro.socket
sudo systemctl restart libvirtd.socket
[no error printed]

So the problem to me looks like the socket service trying to start an
already running daemon backed by .service unit. So (if what we do here
is valid) it looks like a systemd issue (otherwise it would be within
libvirt itself).

I assume what we do *is* valid on the libvirt side since this also
reproduces the problem:

$ sudo systemctl stop libvirtd.service
$ sudo systemctl start libvirtd.socket libvirtd-ro.socket
$ ps awux | grep -qs [l]ibvirtd || echo "Libvirtd not running"
Libvirtd not running
# trigger libvirtd start via socket:
$ virsh list
$ sudo systemctl restart libvirtd.socket
Job failed. See "journalctl -xe" for details.

So the problem seems to be that systemd prints that warning whenever you
try to restart a socket activated service that is already active and also
backed by a .service unit - and it doesn't matter whether it's activated
by socket or via the service file).

Does that make sense?

Cheers,
 -- Guido

Hi Guido,

I like to summarize this from another POV:
- services can start and restart no matter what state the .socket files are in
- sockets can start if the service isn't up yet
- sockets can start if the service is and the socket is already up (is
just a no-op)
- restart = stop+start

IMHO that just means we might want to "start" .socket files but never
ever "restart" them.

I'm in a bad position to decide if that means to better fix it in::
A) debhelper: generate it differently into maintainer scripts
B) systemd: make systemctl "restart" *.socket behave like "start"

I think if there is a corner case left that makes "systemctl restart
socket" needed (and I can think of a few) then the fix might be better
in debhelper.
But that discussion is exactly why I filed this bug - so thanks for chiming in.

Hi,

I initially thought the same but then we wouldn't make changes to
.socket file effective until reboot so it looks to me as if systemd
should handle this gracefully.
 -- Guido

Thanks, from that POV it makes sense.
I'm not good at re-assigning debian bugs could one of you please do so
to affect system as well?

control: reassign -1 systemd
control: retitle -1 systemd fails to restart socket unit of already running service

Dear systemd maintainers

The tl;dr; version is that

    systemctl restart libvirtd.socket

fails like

    Job failed. See "journalctl -xe" for details.

    Apr 02 09:29:57 foo systemd[1]: libvirtd.socket: Socket service libvirtd.service already active, refusing.
    Apr 02 09:29:57 foo systemd[1]: Failed to listen on Libvirt local socket.

if libvirtd.service is already running (libvirtd.socket has):

    [Socket]
    ListenStream=/run/libvirt/libvirt-sock
    Service=libvirtd.service
    SocketMode=0666

and libvirtd.service is https://salsa.debian.org/libvirt-team/libvirt/-/blob/debian/sid/src/remote/libvirtd.service.in

The issue that this also happens in libvirt-daemon-system's postinst so
a restart of the socket unit does not seem to be possible. This could
be worked around by not restarting the socket units (changes in the
socket units seem to be picked up when restarting libvirtd.service
as well) but that would give problems when switching to socket only
activation.

Since this looks like a common problem i'm likely missing something.

Cheers,
 -- Guido

Am 02.04.20 um 09:52 schrieb Guido Günther:
root@pluto:~# systemctl status libvirtd.service libvirtd.socket
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/lib/systemd/system/libvirtd.service; disabled;
vendor prese>
     Active: inactive (dead)
TriggeredBy: ● libvirtd.socket
             ● libvirtd-ro.socket
             ● libvirtd-admin.socket
       Docs: man:libvirtd(8)
https://libvirt.org

● libvirtd.socket - Libvirt local socket
     Loaded: loaded (/lib/systemd/system/libvirtd.socket; enabled;
vendor preset:>
     Active: active (listening) since Thu 2020-04-02 10:17:40 CEST; 24s ago
   Triggers: ● libvirtd.service
     Listen: /run/libvirt/libvirt-sock (Stream)
     CGroup: /system.slice/libvirtd.socket

Apr 02 10:17:40 pluto systemd[1]: libvirtd.socket: Succeeded.
Apr 02 10:17:40 pluto systemd[1]: Closed Libvirt local socket.
Apr 02 10:17:40 pluto systemd[1]: Stopping Libvirt local socket.
Apr 02 10:17:40 pluto systemd[1]: Listening on Libvirt local socket.


root@pluto:~# systemctl restart libvirtd.socket
root@pluto:~# echo $?
0

Hi,

you have:

service not running but
then restarting socket).

# systemctl status libvirtd.service libvirtd.socket
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-04-02 09:44:39 CEST; 49min ago
TriggeredBy: ● libvirtd-admin.socket
             ● libvirtd.socket
             ● libvirtd-ro.socket
       Docs: man:libvirtd(8)
https://libvirt.org
   Main PID: 484267 (libvirtd)
      Tasks: 19 (limit: 32768)
     Memory: 31.6M
     CGroup: /system.slice/libvirtd.service
             ├─  1538 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
             ├─  1539 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
             └─484267 /usr/sbin/libvirtd

Apr 02 09:44:39 foo systemd[1]: Starting Virtualization daemon...
Apr 02 09:44:39 foo systemd[1]: Started Virtualization daemon.
Apr 02 09:44:39 foo dnsmasq[1538]: read /etc/hosts - 6 addresses
Apr 02 09:44:39 foo dnsmasq[1538]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Apr 02 09:44:39 foo dnsmasq-dhcp[1538]: read /var/lib/libvirt/dnsmasq/default.hostsfile

● libvirtd.socket - Libvirt local socket
     Loaded: loaded (/lib/systemd/system/libvirtd.socket; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-04-02 09:44:39 CEST; 49min ago
   Triggers: ● libvirtd.service
     Listen: /run/libvirt/libvirt-sock (Stream)
     CGroup: /system.slice/libvirtd.socket

Apr 02 09:44:39 foo systemd[1]: Listening on Libvirt local socket.
root@foo:/var/scratch/src/libvirt/libvirt# systemctl restart libvirtd.socket
Job failed. See "journalctl -xe" for details.

Cheers,
 -- Guido

I don't think this is going to work.

You'll need to restart both the service and socket in the correct order,
i.e.



systemctl restart libvirtd.service libvirtd.socket

Am 02.04.20 um 10:50 schrieb Michael Biebl:

Remember that a systemctl stop foo.socket will tear down the socket.
If that is in use by a service, that would not work.

@Michael:
I have tried "systemctl restart service socket socket" before and it works.
But I was wondering if it might work just by accident, due to the service being
not yet fully up when the sockets restart. And I was unsure if this
could be racy.
Are you saying this is "the right order" and this way around systemd
will take care
that service and sockets won't collide in a bad way?
at least the
ordering of dh_installsystemd would need to ensure that it does not order
everything alphabetically but instead start with the service followed by all th
rest it picks up.

Today:
$ grep Also /lib/systemd/system/libvirtd.service
Also=virtlockd.socket
Also=virtlogd.socket
Also=libvirtd.socket
Also=libvirtd-ro.socket

And due to that

   dh_installsystemd -p libvirt-daemon-system --restart-after-upgrade
libvirtd.service

Gets generated into postinst as:

deb-systemd-invoke $_dh_action 'libvirtd-ro.socket' 'libvirtd.service'
'libvirtd.socket' 'virtlockd.socket' 'virtlogd.socket'

Triggering exactly that issue that we discuss here.

If it would generate the following it would work (I tried)
deb-systemd-invoke $_dh_action 'libvirtd.service' 'libvirtd-ro.socket'
'libvirtd.socket' 'virtlockd.socket' 'virtlogd.socket'

@Michael:
- would you agree that due to the issues in discussion
dh_installsystem should change the order to always start with the
service?

I have always thought that every service should have Requires= all the
sockets it uses. Does it fix the problem?

Stooping only the socket should either crash the daemon or stop it
gracefully first. I would think the latter is much better.

Hi,

See 955542 , we can't get the order right with the current Also= parsing
it seems. I'm also not sure if it's correct that Also= units are being
restarted as well.
Cheers,
 -- Guido

Am 02.04.20 um 12:45 schrieb Christian Ehrhardt:

No, I don't know for sure.

Am 02.04.20 um 13:09 schrieb Guido Günther:

Also= should only have an effect when enabling a service, not when
(re)starting a service, see #841095

Hi,

IMO the restart of sockets worked by accident because it was OK to
restart then when the service is not running.
When the service is running systemd can't restart the socket safely
and thus IMO systemd is right denying the operation.

Systemd could implement systemctl restart -r <set of units> where it
would find what and in which order should be restarted to have a
restart of the set of units and it would do stops then starts when the
ordering required that. For example it would do stop foo.service ;
restart foo.socket; start foo.service, etc.

I think it would be very complicated or to implement that reliably in
dh_installsystemd or in deb-systemd-invoke partly because local
configuration could also impact the order and set of units to restart.
In the meantime deb-systemd-invoke could fallback to do systemctl stop
...; systemctl start when restarts fail.

Cheers,
Balint