When I call

    rsync z a b c dest/

rsync re-orders die filelist and transfers in alphabetical order.
This breaks some remote applications, in case they expect the files
appearing in some specific order. (debian reprepro + inoticoming as one
exampl).

And I believe, this behaviour breaks that what a unix user would expect.

severity 640492 normal
tags +wontfix
merge 640492 160982
thanks

Filesystems don't guarantee that files will be written into the
directory in order of writing. Additionally, rsync first transfers to a
temp file and renames when done, so anything that is watching a
directory will see multiple events happening anyway.

I'm a unix user since 1984, and I've learnt to test things to understand
how they work :)  Also rsync does work (more or less) on more than just
unix-like systems, so that's hardly an argument.

In this case rsync orders the files to be sent as that is appropriate
for the algorithm that is used to update the destination. In this
special case that wouldn't strictly be necessary, but catering for such
special cases would make rsync slower for the vast majority of use
cases.

A simple workaround is of course to transfer the out of order files in
separate rsync runs. In your example: first do z, then do the rest.
As a workaround is trivially made, I cannot see how this could be
classified as an important bug; especially since this behaviour has been
around since the beginnings of rsync. Hence I'm changing the severity to
"normal".  Additionally, it's basically the same issue as described in
bug report 160982 ("rsync alphabetizes files") , so I'm merging these
bug reports, and marking it wontfix: I cannot imagine that rsync will be
changed ever to not sort the file list.


Paul

Hello Paul,

thanks for the fast response …

Paul Slootman <paul@debian.org> (Mon Sep  5 12:30:09 2011):

I'd agree with you, as soon as rsync supports parallelity in sending the
files. But as long as the sender and the receiver transfer files in a
sequential order, I'd expect that these files appear in exactly that
(timely) order in the destination.

In our example, inoticoming waits for the advent(?) of a *.changes file,
ignoring all other events. In case rsync uses a temp file, the
appearance of the *.changes file is even atomical. (Rsync does not need
to use temp file, --inplace modifies the destination file itself.)

Ok, thus you have about 6 years more unix exerience ☺. For me testing
is not a good base for understanding, reading the docs should be the
first part. (comes into my mind: at least this behaviour of rsync should
be documented...)

And I insist on calling this "unexpected". rsync does a great job in
replacing {,s}cp, but both tools do *not* re-order the files.

I can't reply here, since I didn't check the source.
I'd agree for all cases, where rsync gets passed wildcards (not talking
about passing the wildcards to the invoking shell).

This workaround causes one more prompting for the passphrase (in case no
ssh-agend is used).

Sorry for not finding the similiar issues. And I'll contact the rsync
mailing list, probably they can convince me about that.

Should I file a bug report about a missing note in the manpage?

Greetings from Dresden,

I believe this bug is worth revisiting given the use case of LTFS with
rsync.

LTFS is a linear media with exceedingly slow seek times when compared with
raw read/write speed. To do efficient reads, each file on the tape must be
read off in the order it was written. Each file has an extended attribute
(ltfs.startblock) which returns its exact location (startblock) on the
tape. In this manner, you can easily obtain a list of files in the optimal
order for reads (sorted, ascending, in order of their startblock).

rsync's behavior of copying all files on and off in alphabetical order
works fine if all you ever do is one write, as the order is consistent. If
you do anything else to that volume, however, (using, for example, rsync -u
to copy additional files), it will copy them on in alphabetical order but
multiple times. You will end up with n number of a-z writes distributed
across 6 minutes of seek time.

Trying to restore alphabetically after doing more than one copy will result
in one of the least optimal restore paths imaginable, with average read
rates dropping by 50% in my case when compared with copying them one at a
time using rsync. I, however, was copying fairly large files. Any number of
smaller sizes or wider distribution of files, or both will decimate the
performance. You could easily create a file structure for which rsync would
blindly shoot back and fourth across the entire tape for every single file.

Basically: batch file copying with rsync, if it always restores
alphabetically, is useless for LTFS volumes.

While I can write a python script to execute rsync on individual files, It
would be nice to be able to pass an argument to rsync
(--maintain-file-order) which would prevent rsync from taking my optimally
ordered list and butchering the LTFS restore with an alphabetical sort.