First, thanks for diffoscope; it's the best. :)

The context for this bug is in [1].  The basic gist is that I need to
preserve extended attributes to get the proper capabilities on
"/bin/ping" inside a rootfs tarball, so I updated my GNU tar invocation
to include "--xattrs", "--acls", and "--selinux" (just for good
measure).

[1]: https://github.com/debuerreotype/debuerreotype/pull/38

After doing so, I did a quick "diffoscope" of the before/after (which
normally is indispensibly helpful in ensuring that my PR does exactly
what I expect it to and not a single thing more).  I was expecting to
see something showing me that I've got some new extended attributes on
at least "/bin/ping" (and possibly other files), but instead got the
following:

| $ diffoscope travis/20170101/amd64/jessie/rootfs.tar.xz
output/20170101/amd64/jessie/rootfs.tar.xz
| 2018-06-01 17:22:24 W: libarchive: Ignoring malformed pax extended attribute
| 2018-06-01 17:22:24 W: libarchive: Ignoring malformed pax extended attribute
| 2018-06-01 17:22:24 W: libarchive: Ignoring malformed pax extended attribute
| 2018-06-01 17:22:25 W: libarchive: Ignoring malformed pax extended attribute
| 2018-06-01 17:22:25 W: libarchive: Ignoring malformed pax extended attribute
| 2018-06-01 17:22:26 W: libarchive: Ignoring malformed pax extended attribute
| 2018-06-01 17:22:26 W: libarchive: Ignoring malformed pax extended attribute
| 2018-06-01 17:22:26 W: libarchive: Ignoring malformed pax extended attribute
| 2018-06-01 17:22:26 W: libarchive: Ignoring malformed pax extended attribute
|  |############################|  100%
Time: 0:00:10
| --- travis/20170101/amd64/jessie/rootfs.tar.xz
| +++ output/20170101/amd64/jessie/rootfs.tar.xz
| ├── rootfs.tar
| │ ├── filetype from file(1)
| │ │ @@ -1 +1 @@
| │ │ -POSIX tar archive (GNU)
| │ │ +POSIX tar archive

I've uploaded the two .tar.xz files in question to [2] in case you'd
like to reproduce. :)

[2]: https://people.debian.org/~tianon/diffoscope-gnu-tar-xattrs/

If there's something obvious I've missed, I apologize!  I searched the
bugs and the list as much as I could, and couldn't find anything similar
(which was surprising -- I would've thought someone else would've run
into this long before me). O:>


♥,
- Tianon
  4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4

It gets worse -- apparently any combination of "--xattrs" (with or
without "--xattrs-include" values) makes GNU tar's output completely
unreproducible (running it on the same directory twice will provide a
different result).

I've noted this in [2], and provided a full `hexdump` in [3].

[2]: https://github.com/debuerreotype/debuerreotype/pull/38#issuecomment-393992413
[3]: https://gist.github.com/tianon/5bbfe30a4e889dcece42b8d210f8c8f9

(I poked lamby on IRC about where to note this for the reproducible
builds project as a whole, and he told me to just append it to this
bug for now.)


♥,
- Tianon
  4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4

Hi Tianon,

Oh, neat! Hm, can you try:

Hey, that's snazzy!  That gives me a reproducible result. :)

Without:

$ sudo tar --create --directory wtf --numeric-owner --transform
's,^./,,' --sort name --xattrs . | md5sum
7ce7cc6c2ad85b1f6fbe2e061188c7a0  -
$ sudo tar --create --directory wtf --numeric-owner --transform
's,^./,,' --sort name --xattrs . | md5sum
48d6da7b220e95ce913b0ab69e7edcd3  -

With:

$ sudo tar --create --directory wtf --numeric-owner --transform
's,^./,,' --sort name --xattrs
--pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0 . | md5sum
b45e8c1739b34f844731e94afc2ccaac  -
$ sudo tar --create --directory wtf --numeric-owner --transform
's,^./,,' --sort name --xattrs
--pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0 . | md5sum
b45e8c1739b34f844731e94afc2ccaac  -

And after running "sudo setcap cap_net_raw+ep wtf/bin/ping" (to make
sure there are some caps that actually need to be represented):

$ sudo tar --create --directory wtf --numeric-owner --transform
's,^./,,' --sort name --xattrs
--pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0 . | md5sum
ce0aa6ac1b6fc9df117696cc17cb07b3  -
$ sudo tar --create --directory wtf --numeric-owner --transform
's,^./,,' --sort name --xattrs
--pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0 . | md5sum
ce0aa6ac1b6fc9df117696cc17cb07b3  -


♥,
- Tianon
  4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4

Hi Tianon,

 "It's so simple!"  >.>

Glad I could resolve your use-case. However, I assume there is still a
diffoscope bug here?

ie. We certainly shouldn't be emitting "Ignoring malformed pax
extended attribute" to stderr and — as a possible improvement — we
should be emitting that as part of the "proper" output?


Regards,

Chris Lamb wrote:

Gentle ping on this Tianon? :-)


Regards,

Apologies!  Yes, IMO there's definitely a diffoscope bug here.  The
_ideal_ case would be that it shows me useful data about the PaX
headers that don't match (so that I can narrow down why my tarballs
aren't reproducible).

For some slight context, I reach for diffoscope when I have two binary
objects that I expect to be the same and aren't because it's really
good at taking all that binary noise and converting it to something I
can make some sense out of with an eventual fallback to something like
"diff <(hexdump) <(hexdump)" if it can't find any useful signal in the
noise (or doesn't have the necessary tools installed).

In the case of these PaX headers, we get this misleading warning a few
times, then the "file(1)" diff output ("-POSIX tar archive" / "+POSIX
tar archive (GNU)") -- not even a note about how diffoscope couldn't
find any more meaningful differences between the two objects, so I
think this is at least a wishlist bug for diffoscope and possibly a
minor/normal issue depending on how important you think tar files
containing capabilities / extended attributes are. :)


♥,
- Tianon
  4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4

forwarded 900593 https://salsa.debian.org/reproducible-builds/diffoscope/issues/5
thanks

I've forwarded this upstream here:

https://salsa.debian.org/reproducible-builds/diffoscope/issues/5


Regards,