#337827 rsync should handle new sparse files better with -S

Package:
rsync
Source:
rsync
Description:
fast, versatile, remote (and local) file-copying tool
Submitter:
Goswin von Brederlow
Date:
2010-04-14 12:12:27 UTC
Severity:
wishlist
#337827#5
Date:
2005-11-06 18:39:59 UTC
From:
To:
Hi,

I noticed that the sparse file handling of rsync could use some
improvements when transmitting a new file. It seems to me like rsync
will copy the zeros of a sparse file verbatim over the connection
limiting the transfere to the network bandwith (or cpu power with
-z). This can be greatly increased (a factor of 10 here) by running
"dd if=/dev/zero of=file bs=1 count=1 seek=1000000" beforehand because
then rsync will transmit block matches for every block of zeroes
instead of verbatim data.

My suggestion would be to have an implizit block of zeroes (block
number ~0 or -1) either always or when the -S option is given (needs
client+server support) or to insert a faked block at the end of the
file when generating checksums (only needs client changes).


On a grander scale the rsync protocol could be extended to cover
repetitive blocks and send a block match to the destination file
instead of the source file. One bit in the block number could be
reused for this.

MfG
	Goswin
#337827#10
Date:
2006-10-14 19:29:21 UTC
From:
To:
Hello,

Any progress on this bug ?

The way rsync is handling sparse file is suboptimal. It leaves any
backup policy based on rsync open to a trivial DoS with thinks link the following :

dd if=/dev/zero of=bigfake bs=1k count=1 seek=2000000000

rsync -e ssh -avS bigfake user@localhost:/tmp

At that point you wait for 2TB of unusfull zeros been transferred
between the src-server and the backup_server... Annoying.

I've been beaten by this feature twice already. Students borking some
seek/lseek maths while writing to files... We got several 100GB files
to transfert during the backup at night...

@+,
	Fab
#337827#15
Date:
2006-10-14 20:38:56 UTC
From:
To:
I'm afraid not...
I'll talk to the upstream maintainer to see what possibilities there are
for extending the protocol to handle this.

I understand...

Using -z will speed things up quite a lot, as the zeroes compress well.
However, perhaps a better workaround in the meantime is to exclude
(student) files that are larger than a reasonable amount via the
--max-size option.


Paul Slootman
#337827#20
Date:
2006-10-15 08:42:41 UTC
From:
To:
Paul Slootman a écrit :

Thanks.

Yep, if I can apply this option only on sparse files. It will slow down
quit a bit our backups if we use it per default.

Or ask the student to clean his/her mess. Thank's for the tip nontheless.

@+,

	Fab