#602965 please add --rsyncable to mksquashfs

Package:
squashfs-tools
Source:
squashfs-tools
Description:
Tool to create and append to squashfs filesystems
Submitter:
Kirill Smelkov
Date:
2019-10-14 06:39:03 UTC
Severity:
wishlist
#602965#5
Date:
2010-11-09 21:05:56 UTC
From:
To:
Please add support for rsync-friendly compression mode for mksquashfs.

This should operate the same way like `gzip --rsyncable` does, and so
would help to significantly reduce needed bandwith when synchronizing
versioned filesystem images over network, or significantly reduce disk
space when storing filesystem images under de-duplicating versioning
backup archive such as bup[1].


Thanks,
Kirill


[1] https://github.com/apenwarr/bup/blob/master/README.md
#602965#18
Date:
2019-10-14 05:55:21 UTC
From:
To:
I build Debian Live images each month, and upload them to a remote site.
As you might expect, most of the files are the same.

Using a Debian 9 stack, this is what I observed this month:

  server images:

    Total file size: 116.29M bytes
    Total bytes sent: 68.51M

    Total file size: 224.01M bytes
    Total bytes sent: 149.05M

  desktop (xfce) image:

    Total file size: 487.74M bytes
    Total bytes sent: 151.98M

  desktop (xfce) image w/ lots of big game assets:

    Total file size: 1.76G bytes
    Total bytes sent: 137.67M

    Total file size: 1.77G bytes
    Total bytes sent: 134.26M

You can see that there seems to be an initial flagfall of ~100MB,
after which the large, unchanging game assets mostly "settle" and are
highly rsyncable, even after passing through mksquashfs -comp xz.


Note that:

  1. These numbers include Debian 9 initrd.img files with COMPRESS=xz.
     These are around 20MB, and are *NOT* rsyncable,
     i.e. rsync has to transmit the whole ramdisk every time.
     In Debian 10 buster, the #845034 fix in initramfs-tools 0.131+ fixes this.

  2. These numbers are from mksquashfs 4.3.
     In Debian 11 bullseye, mksquashfs 4.4 should improve
     "rsyncability" due to several changes related to reproducible builds.

Therefore we can reasonably expect to shave a good 40MB+ off that 100MB flagfall.

On this basis, I think an explicit --rsyncable is NOT NEEDED.
It is sufficient to simply upgrade to squashfs 4.4+ and initramfs-tools 0.131+.

It MIGHT further help to set SOURCE_DATE_EPOCH to the same time for both images
(I set mine to TZ=UTC date +%s -d '1 Jan', which is kinda naughty).

It MIGHT further help to use mksquashfs -sort to force a particular packing, but
my initial tests were not promising, so I abandoned this idea.