Hi,
I have hit a strange lockup when rsyncing to an nfs storage mounted
localy. At first I though it would be a network related problem but
further investigation indicates otherwise.

rsync -avz SOURCE_DIR TARGET_DIR spawns 2 children processes
31043 pts/2    S+     0:25           rsync -azv --progress SOURCE_DIR TARGET_DIR
31044 pts/2    S+     0:03             rsync -azv --progress SOURCE_DIR TARGET_DIR
31045 pts/2    S+     0:03               rsync -azv --progress SOURCE_DIR TARGET_DIR

I have tried to strace all three (they run under root) as root.
# strace -tt -T -p 31043
strace: attach: ptrace(PTRACE_ATTACH, ...): Operation not permitted
# strace -tt -T -p 31044
Process 31044 attached
11:48:30.309529 select(2, [], [1], [], {29, 878783}) = 0 (Timeout) <29.908793>
11:49:00.218621 select(2, [], [1], [], {60, 0}) = 0 (Timeout) <60.039005>
11:50:00.257837 select(2, [], [1], [], {60, 0}) = 0 (Timeout) <60.043860>

So it is waiting to write to fd 1 which is
# ls -l /proc/31044/fd/1
lrwx------ 1 root root 64 Apr 13 11:48 /proc/31044/fd/1 -> socket:[151908]
# lsof -p 31044 | grep -w 151908
rsync   31044 root    1u  unix 0xffff8801bea58b80      0t0  151908 socket
# ss -nxp | grep 151908
u_str  ESTAB      0      68951                  * 151908                 * 151907 users:(("rsync",pid=31044,fd=1))
u_str  ESTAB      64049  0                      * 151907                 * 151908 users:(("rsync",pid=31043,fd=5))

So it is a unix socket (most probably socketpair) to the first rsync
task.  I have only briefly checked the code and it seems to be
fd_pair(from_child_pipe) because it matches stdout fd.

The other task is in a similar state
# strace -tt -T -p 31045
Process 31045 attached
17:50:33.740635 select(8, [], [7], [], {52, 42706}) = 0 (Timeout) <52.094874>
17:51:25.835829 select(8, [], [7], [], {60, 0}) = 0 (Timeout) <60.060079>
17:52:25.895988 select(8, [], [7], [], {60, 0}^CProcess 31045 detached
 <detached ...>
# ls -l /proc/31045/fd/7
lrwx------ 1 root root 64 Apr 13 12:00 /proc/31045/fd/7 -> socket:[151917]
# lsof -p 31045 | grep 151917
rsync   31045 root    7u  unix 0xffff88041587fbc0      0t0  151917 socket
# ss -nxp | grep 151917
u_str  ESTAB      13052  0                      * 151916                 * 151917 users:(("rsync",pid=31044,fd=6))
u_str  ESTAB      0      54272                  * 151917                 * 151916 users:(("rsync",pid=31045,fd=7))

except 31044 is on the other end which cannot read from the socket
buffer because it is stuck trying to write to its parent (safe_write
seems to be retrying endlessly). So it all looks like the first tasks
is blocking the remaining two. I wish I could find out more about the
primary process but all I can see is

# cat /proc/31043/stack
[<ffffffff8108aaf0>] hrtimer_wakeup+0x0/0x30
[<ffffffff811bb2d5>] poll_schedule_timeout+0x45/0x60
[<ffffffff811bbc9e>] do_select+0x5ae/0x770
[<ffffffff811880c8>] alloc_pages_vma+0x98/0x160
[<ffffffff811bb460>] pollwake+0x0/0x70
[<ffffffff8108c216>] atomic_notifier_call_chain+0x16/0x20
[<ffffffff81094579>] set_task_cpu+0x99/0x1a0
[<ffffffff8109f744>] check_preempt_wakeup+0xe4/0x1d0
[<ffffffff8109f766>] check_preempt_wakeup+0x106/0x1d0
[<ffffffff81094325>] check_preempt_curr+0x85/0xa0
[<ffffffff8109f744>] check_preempt_wakeup+0xe4/0x1d0
[<ffffffff81094325>] check_preempt_curr+0x85/0xa0
[<ffffffff81094354>] ttwu_do_wakeup+0x14/0xd0
[<ffffffff81096b93>] try_to_wake_up+0xd3/0x2d0
[<ffffffff810a7547>] __wake_up_common+0x57/0x90
[<ffffffff8109be86>] set_next_entity+0x56/0x70
[<ffffffff811bbfe8>] core_sys_select+0x188/0x280
[<ffffffff81076fe1>] ptrace_stop+0x1a1/0x260
[<ffffffff81077112>] ptrace_do_notify+0x72/0x80
[<ffffffff8101b3d5>] read_tsc+0x5/0x20
[<ffffffff810c7bd2>] ktime_get_ts+0x42/0xe0
[<ffffffff811bc18a>] SyS_select+0xaa/0xf0
[<ffffffff81513f1f>] system_call_trace_compare_end+0x10/0x15
[<ffffffffffffffff>] 0xffffffffffffffff

So it seems to be stuck in select() as well.
# cat /proc/31043/syscall
23 0x5 0x7ffe4ccfb270 0x7ffe4ccfb370 0x7ffe4ccfb2f0 0x7ffe4ccfb240 0xffffffffffba62de 0x7ffe4ccfb1f8 0x7f7bf4d35ac3

but this didn't really tell me much withtout accessing the memory.  I am
not sure how reproducible this is so I have left the rsync running in
case you need a further information.

Thanks!

OK, found the reason I cannot strace the first task (another admin has
left his strace running...). Anyway this looks like a deadlock.
# strace -p 31043
Process 31043 attached
select(5, [], [4], [], {31, 179565})    = 0 (Timeout)
select(5, [], [4], [], {60, 0})         = 0 (Timeout)
# ls -l /proc/31043/fd/4
lrwx------ 1 root root 64 Apr 14 09:35 /proc/31043/fd/4 -> socket:[151906]
# lsof -p 31043 | grep 151906
rsync   31043 root    4u  unix 0xffff8801a41a9b80      0t0   151906 socket
# ss -nxp | grep 151906
u_str  ESTAB      18465  0                      * 151905                 * 151906 users:(("rsync",pid=31045,fd=0))
u_str  ESTAB      0      53760                  * 151906                 * 151905 users:(("rsync",pid=31043,fd=4))

To summarize:
31043 wants to write to a socket with 31045 on the other end but the socket
buffer is most probably full, because select would tell us otherwise.
31045 wants to write to a socket with 31044 on the other end and cannot
proceed as well.
31044 wants to write to a socket with 31043 on the other end and we are
in the waiting loop. So this looks like a deadlock unless I have screwed
something in the above.

I ran into what appeared to be an identical deadlock today when rsyncing a
directory hierarchy between two local directories. I too had all three
rsync processes all stuck inside a select system call with a sixty second
timeout.

I was executing:

 rsync --delete-before --exclude 'pattern.*' -vat /source/ /destination/

with rsync 3.1.1-3 from Jessie.

I noticed that the files and directories in destination hierarchy were
owned by the wrong user and the permissions did not permit the user running
rsync to write to them.

Initially, during the delete-before phase, rsync complains about failing to
unlink with Permission denied, but continues.

Then it complains with:

 rsync: failed to set times on "/destination/.": Operation not permitted (1)

for each directory as it comes to it.

But it does not complain about individual files. It gets through about 580
files (with relatively long filenames) and then gets stuck seemingly
indefinitely as described above.

Fixing the destination ownership resolved the problem.

This may not have been the same cause for the original bug reporter,
however, it's possible that a more subtle problem could cause the same
failure mode.

Mike.

I spent some time tracing and reproducing what might have been this same bug 820916, and documented my travels on some long forgotten thread over at Serverfault (as user "ThePythonicCow"):

https://serverfault.com/questions/743412/rsync-hangs-expand-file-list-pointer-array-to-n-bytes-did-move

I ended up building a nice little test case - a shell script of a dozen or so lines, that reproduced the hang on the rsync version 3.2.3 in my Ubuntu 22.04 system.  I posted that test case at the above serverfault page.

Then I was looking around for where to file this as a bug, came across https://rsync.samba.org, where newer rsync versions, up to 3.2.7 were available.

My hanging test case, which depended on using rsync --verbose option twice, no longer hangs with 3.2.7

Given that there might not be enough detail in this old 820916 bug to know whether or not whatever failed back then has been fixed or not, perhaps it's time to close 820916 as "maybe fixed - reopen if you can still reproduce it."

Hello,
does anyone has fresh info on that issue. Can anyone reproduce it with
version 3.2.7 or newer?

Regards,
Christian Buhtz