[The bug is fixed upstream. This is a tracking bug in Debian for this issue]
Hi,
I want to create bit-by-bit reproducible sparse tar archives of the same sparse
input file and find myself unable to do so using gnu tar. I create the sparse
test input like this:
$ echo start > sparsefile
$ truncate -s 4M sparsefile
$ echo middle >> sparsefile
$ truncate -s 8388604 sparsefile
$ echo end >> sparsefile
$ stat -c '%b*%B-%s' -- sparsefile
24*512-8388608
$ sha1sum sparsefile
e4d6196039c65beb8ba41b5b45f404549e41303c sparsefile
The output of stat shows that indeed the file I created contains holes as the
number of allocated blocks (24) is much smaller than the size of the file
(8MiB) when multiplied by the block size (512).
I try to pack up the file `sparsefile` in a tarball using this invocation of
gnu tar:
$ SOURCE_DATE_EPOCH=0 tar --mtime=@0 --pax-option='delete=atime,delete=ctime' --numeric-owner --owner=0 --group=0 --format=pax --sparse -cf sparse-archive1.tar sparsefile
I then repeat the process, only changing the output filename:
$ SOURCE_DATE_EPOCH=0 tar --mtime=@0 --pax-option='delete=atime,delete=ctime' --numeric-owner --owner=0 --group=0 --format=pax --sparse -cf sparse-archive2.tar sparsefile
My expectation is, that `sparse-archive1.tar` and `sparse-archive2.tar` are
identical as they contain the same file, I have $SOURCE_DATE_EPOCH set, I
delete atime and ctime from the pax header, I set the mtime to 0 and force the
owner to be root. But despite all of that, `sparse-archive1.tar` and
`sparse-archive2.tar` differ like this when looking at them using xxd:
@@ -59,23 +59,23 @@
000003a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000003b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000003c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000003d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000003e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000003f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000400: 2e2f 474e 5553 7061 7273 6546 696c 652e ./GNUSparseFile.
-00000410: 3138 3636 3433 372f 7370 6172 7365 6669 1866437/sparsefi
+00000410: 3138 3636 3432 342f 7370 6172 7365 6669 1866424/sparsefi
00000420: 6c65 0000 0000 0000 0000 0000 0000 0000 le..............
00000430: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000440: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000450: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000460: 0000 0000 3030 3030 3636 3400 3030 3030 ....0000664.0000
00000470: 3030 3000 3030 3030 3030 3000 3030 3030 000.0000000.0000
00000480: 3030 3331 3030 3000 3030 3030 3030 3030 0031000.00000000
-00000490: 3030 3000 3031 3334 3236 0020 3000 0000 000.013426. 0...
+00000490: 3030 3000 3031 3334 3232 0020 3000 0000 000.013422. 0...
000004a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000004b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000004c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000004d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000004e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000004f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000500: 0075 7374 6172 0030 3000 0000 0000 0000 .ustar.00.......
The checksum differs because the stored file contains the tar pid 1866437 for
`sparse-archive1.tar` and 1866424 for `sparse-archive2.tar`.
I read https://www.gnu.org/software/tar/manual/html_node/PAX-0.html to find out
that the sparse file is stored using the pattern %d/GNUSparseFile.%p/%f (where
%p is the PID) so that tar implementations that are not aware of gnu extensions
store the sparse file into separate directories.
Workarounds I found so far:
1. unshare the pid namespace
-----------------------------
If I run tar prefixed with
$ unshare --map-auto --map-root-user --fork --pid -- tar ...
Then tar will run inside a new pid namespace and will always run with a pid of
1 in that namespace. To create a new pid namespace without being root, I also
need to unshare the user namespace, so I am likely no longer allowed to write
into the current directory as the unshared user. To work around that I can
let tar write the tarball to stdout and then redirect that into a file. Full
commandline:
$ unshare --map-auto --map-root-user --fork --pid -- tar --clamp-mtime
--mtime=@0 --pax-option='delete=atime,delete=ctime' --numeric-owner --owner=0
--group=0 --format=pax --sparse -cf - sparsefile > sparse-archive.tar
2. use PAX format 0.0
----------------------
The initial format does not embed the PID into the tarball, so I get bit-by-bit
reproducible output if I run tar like this:
$ tar --sparse-version 0.1 --pax-option delete=GNU.sparse.map --clamp-mtime
--mtime=@0 --pax-option='delete=atime,delete=ctime' --numeric-owner --owner=0
--group=0 -- format=pax --sparse -cf sparse-archive.tar sparsefile
Using format 0.0 is probably not recommended for reasons explained in the
documentation I linked above.
3. don't use gnu tar
---------------------
bsdtar from libarchive does not include the PID into the output and uses a
static value of 0 instead:
$ bsdtar --clamp-mtime --mtime=@0 --numeric-owner --uid 0 --gid 0
--read-sparse -cf sparse-archive.tar sparsefile
But this is what the bsdtar man page calls "restricted pax format". If I run
bsdtar with --format pax then the resulting tarball becomes unreproducible
again because in contrast to gnu tar, bsdtar does not have a way to remove the
atime and ctime pax options from the header. Even when setting the atime using
the touch utility, I cannot control the ctime.
In summary, by default gnu tar is unable to produce bit-by-bit identical sparse
tarballs from the same sparse input file.
I attached a patch from upstream (minus the documentation changes) which fixes
the issue.
Thanks!
cheers, josch