#1004687 systemd: When NFS filesystems are mounted, systemctl actions (ex: daemon-reload) are exceedingly slow

Package:
systemd
Source:
systemd
Description:
system and service manager
Submitter:
Brandon Applegate
Date:
2022-07-05 20:33:03 UTC
Severity:
normal
Tags:
#1004687#5
Date:
2022-01-31 19:15:23 UTC
From:
To:
Dear Maintainer,

*** Reporter, please consider answering these questions, where appropriate ***

   * What led up to the situation?

	Installed fresh Bullseye (amd64).  At some point experienced extreme delay with systemctl operations.

   * What exactly did you do (or not do) that was effective (or
     ineffective)?

	Increased systemd log level to debug.  Through trial and error determined unmounting all NFS alleviates this condition.

   * What was the outcome of this action?

	"systemctl daemon-reload" can now be ran normally with no delay.

   * What outcome did you expect instead?

	N/A

*** End of the template - remove these template lines ***
#1004687#10
Date:
2022-01-31 20:20:12 UTC
From:
To:
FYI - I noticed that while one of these hangs was happening (~ 90 seconds by the way…) - this was running for some amount of time:

root      180486  0.0  0.0 167376  5956 ?        S    09:26   0:00 (sd-executor)
root      180496  0.0  0.0  13120  5552 ?        D    09:26   0:00  \_ /usr/lib/systemd/system-generators/systemd-fstab-generator /run/systemd/generator /run/systemd/generator.early /run/systemd/generator.late

This is what led me to try unmounting my NFS.
#1004687#15
Date:
2022-01-31 21:59:12 UTC
From:
To:
Am 31.01.22 um 20:15 schrieb Brandon Applegate:

This sounds like an issue with your NFS setup.
Calling systemctl will cause NFS mounts to be stated, but that shouldn't
take too long.
In a test bed here I couldn't reproduce the problem.
Thus marking accordingly.

Please provide more information how this issue can be reproduced.


Michael
#1004687#22
Date:
2022-01-31 22:29:00 UTC
From:
To:
Hello,

It could be.  I will tell you though that for what it’s worth, up until about a week ago this machine ran Debian 9, and the NFS server setup was exactly the same.  Never had this issue at all.

I don’t have any errors in logs on the NFS server side (TueNAS (nee: FreeNAS) by the way).  The only thing close to an NFS related error I have on the Debian client side are a few nfsidmap nss_getpwnam errors when initially mounting.

One other thing I think I will try is to mount these rw.  I’ve had them mounted ro forever (by design).  Just wondering if something is unhappy about them being ro.
#1004687#27
Date:
2022-02-01 00:03:49 UTC
From:
To:
Mounting as rw didn’t seem to help.  Shortly after that - a random try of “systemctl reload-daemon” resulted in the same hang/delay.  I noticed again that the systed fstab generator was running.  I managed to strace it (missed the beginning of course) - and it was hanging.  Eventually I got this:

strace: Process 56818 attached
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=56808, si_uid=0} ---
+++ killed by SIGTERM +++

I’m guessing it wasn’t responding so systemd killed it ?

Is there anything else I can do to try to troubleshoot this ?
#1004687#32
Date:
2022-07-05 17:19:46 UTC
From:
To:
Brandon,

this doesn't appear to be a Debian specific issue.
Could you please install systemd v250 from bullseye-backports and test
if the problem is still reproducible.

If so, I would kindly ask you to file the issue upstream at
https://github.com/systemd/systemd/issues/new

Michael
#1004687#37
Date:
2022-07-05 19:28:47 UTC
From:
To:
Well now I can’t even reproduce it on the original system.  Grrr.

You can close this if you need to.  If I hit this again, I will try pulling in newer systemd from backports as you suggest and file an upstream bug if need be.

Thanks.
#1004687#42
Date:
2022-07-05 20:29:20 UTC
From:
To:
Am 05.07.22 um 21:28 schrieb Brandon Applegate:

Ok, sounds like a plan.

Regards,
Michael