Dear Maintainer,
On my hypervisor with many disks, the xfs_scrub_all systemd service hangs indefinitely during its scheduled run.
The process state showed:
PID PPID STAT WCHAN
4025445 1 SNs do_wait
4025770 4025445 SN anon_pipe_write
The parent process was:
/usr/bin/python3 /usr/sbin/xfs_scrub_all --auto-media-scan-interval 1mo
and the child was:
lsblk -o NAME,KNAME,TYPE,FSTYPE,MOUNTPOINT -J
/proc/4025770/stack showed:
anon_pipe_write
vfs_write
ksys_write
__x64_sys_write
strace of the parent showed:
wait4(4025770, ...
A standalone invocation of the exact lsblk command completed normally:
$ time lsblk -o NAME,KNAME,TYPE,FSTYPE,MOUNTPOINT -J > /tmp/lsblk.out
real 0m0.042s
user 0m0.027s
sys 0m0.015s
The cause appears to be find_mounts() in /usr/sbin/xfs_scrub_all:
result = subprocess.Popen(cmd, stdout=subprocess.PIPE)
result.wait()
if result.returncode != 0:
return fs
sarray = [x.decode(sys.stdout.encoding) for x in result.stdout.readlines()]
The parent waits for lsblk to terminate before consuming stdout. If lsblk produces enough JSON output to fill the pipe, lsblk blocks in anon_pipe_write waiting for the parent to consume stdout, while the parent blocks in wait4 waiting for lsblk to exit.
I locally changed this to use subprocess.run(), so that stdout is consumed without this deadlock:
result = subprocess.run(cmd, stdout=subprocess.PIPE)
if result.returncode != 0:
return fs
output = result.stdout.decode(sys.stdout.encoding)
After this change:
$ time /usr/sbin/xfs_scrub_all
real 0m0.447s
The systemd service also completed successfully:
Process: 4056542 ExecStart=/usr/sbin/xfs_scrub_all --auto-media-scan-interval 1mo (code=exited, status=0/SUCCESS)
Main PID: 4056542 (code=exited, status=0/SUCCESS)
xfs_scrub_all.service: Deactivated successfully.
Finished xfs_scrub_all.service.
This appears to correspond to the upstream fix for the xfs_scrub_all deadlock when lsblk produces a large amount of output.
Could this fix please be backported to the xfsprogs package in Debian 13/trixie?
Upstream fix:
https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/commit/?id=1c52b36e3234aedaaa4b467f64c01636e96c1946
Regards,