In Debian/testing: $ gbp clone https://salsa.debian.org/debian/fdroidserver.git $ cd fdroidserver $ rm ../fdroidserver_1.0.6.orig.tar.gz* $ git reset --hard; git clean -fdx $ gbp buildpackage -uc -us gbp:info: Creating /export/share/code/debian/fdroidserver_1.0.6.orig.tar.gz gbp:error: Error creating fdroidserver_1.0.6.orig.tar.gz: Pristine-tar couldn't checkout "fdroidserver_1.0.6.orig.tar.gz": xdelta: expected from file (/tmp/pristine-tar.SljdkfANnj/recreatetarball) of length 7557120 bytes xdelta: expected from file (/tmp/pristine-tar.SljdkfANnj/recreatetarball) of length 7557120 bytes xdelta: expected from file (/tmp/pristine-tar.gWN3rKYWgn/recreatetarball) of length 7557120 bytes xdelta: expected from file (/tmp/pristine-tar.EJo9dG6tou/recreatetarball) of length 7557120 bytes xdelta: expected from file (/tmp/pristine-tar.EJo9dG6tou/recreatetarball) of length 7557120 bytes pristine-tar: Failed to reproduce original tarball. Please file a bug report. pristine-tar: failed to generate tarball You can also see this happening here: https://salsa.debian.org/debian/fdroidserver/-/jobs/26312 tar 1.30+dfsg-2 pristine-tar 1.44 git-buildpackage 0.9.9 This all works fine using pristine-tar 1.38 in Ubuntu/xenial. tar 1.29b-1.1 pristine-tar 1.38 git-buildpackage 0.8.12.2 Control: notfound -1 1.38
This is probably some useful new info: https://salsa.debian.org/eighthave/fdroidserver/-/jobs/26316 gbp:error: Error creating fdroidserver_1.0.6.orig.tar.gz: Pristine-tar couldn't checkout "fdroidserver_1.0.6.orig.tar.gz": mkdir /tmp/pristine-tar.SQkF8S8Z0J/workdir/fdroidserver-1.0.6/tests/repo/urzip-; \320\240\320\260\321\205\320\274\320\260\314\201, [r\311\220x\313\210man\312\262\311\252n\311\231f] \330\263\331\212\330\261\330\254\331\212_\330\261\330\256\331\205\330\247\331\206\331\212\331\206\331\210\331\201 \350\260\242\302\267.apk: File name too long at /usr/bin/pristine-tar line 439. pristine-tar: failed to generate tarball The fdroidserver_1.0.6.orig.tar.gz tarball contains a file with difficult unicode chars as a test that fdroidserver supports unicode well. It seems that it also has tested pristine-tar's unicode support.
This seems to be a regression caused by tar. I can successfully checkout the tarball, but with the version tar_1.29b-2. When I install tar_1.30+dfsg-1, I cannot checkout anymore. I guess we'll need to find the real cause in tar then.
Control: reassign 901952 tar 1.30+dfsg-1 I believe I have a repro case extracted from this bug report showing that #897653 was not actually fixed. It can be downloaded from: http://ivyzdokx5queplps.onion/repro.tar.xz It has a number of files generated with tar 1.30, with all combinations of reproducibility related flags and none of them reproduces the tarball produced by the version 1.29. I'm reassigning this to tar.
We need to isolate which upstream commit caused the behavior change. I've got other things to work on that are higher priority to me right now, so help from someone more motivate to chase this down would be appreciated. Bdale
It looks this commit is the problem: http://git.savannah.gnu.org/cgit/tar.git/commit/?id=dee7e3f16e74e07504bb8f4d80426005fe4364ae tar no longer strips a quoting level when reading lines from the manifest file.
here's another case of this bug: https://salsa.debian.org/eighthave/androguard/-/jobs/47277 I'm often working on a mix of Ubuntu/LTS, Debian/stable, and Debian/testing. It was likely that the tarball was added with gbp-import-orig on Ubuntu/xenial, and that build log is from Debian/testing with some sid packages. Here's what's running on Ubuntu/xenial: ii tar 1.29b-1.1 amd64 ii pristine-tar 1.38 amd64 .hc
I can confirm that force-downgrading tar to 1.29b-1.1 on Debian/testing fixes the issue: https://salsa.debian.org/eighthave/androguard/-/jobs/47348
tags 901952 +help thanks I question whether this bug is really release-critical, since the only cases of it I've heard about so far are the two instances pointed to in this bug log where pristine-tar is being used across distributions and versions... I've not seen the behavior in any of my own packaging work. However, if someone wants to work out a way to patch current tar such that it mimics the old behavior when necessary to unpack an older tarball without breaking anything else, I'd be happy to merge such a patch. I mean something like what was done to fix #897653. Bdale
I think this bug does qualify as RC since it can affect any source tarball that was created with an older version of tar. So it is not just that it comes from a different distro/release than buster, but it could be from a package that has a source tarball that hasn't changed since tar 1.30.
this also goes the other way, where tarballs created in tar 1.30 fail to work in pristine-tar when tar 1.29 is installed: https://salsa.debian.org/python-team/modules/libcloud/-/jobs/115815
Hi. A couple Debian users have reported issues using our packaging of tar 1.30 to unpack source packages made with 1.29 and prior stored in git repos using pristine-tar, due to an unfortunate side-effect of http://git.savannah.gnu.org/cgit/tar.git/commit/?id=dee7e3f16e74e07504bb8f4d80426005fe4364ae My thanks to Eric Prestemon for figuring out which commit was at fault, and for his diagnosis and thoughts documented at http://tech.prestemon.com/debugging/2018/08/09/debian-bug-901952.html Is it expected / correct that --unquote and --verbatim-files-from no longer work together? I'm trying to figure out the least obnoxious patch that would enable recovery of pristine-tar archives created with 1.29 and older using 1.30. If this was an unexpected side-effect and there's already an upstream commit I can pull and/or you're willing to suggest a suitable patch, that'd be awesome. Independently, I'd be pleased to have your opinions on Eric's suggestion that pristine-tar switch to using --null in the future (which I don't have control over but could make suggestions to the maintainer about). Regards, Bdale
Hans-Christoph Steiner <hans@eds.org> writes: Unfortunately, the nature of pristine-tar is such that it's somewhat brittle in the face of upstream changes to tar. I don't think it's unreasonable to expect someone working on a package to have a version of tar installed that's at least as new as the one used to create the package. And as long as we have the work-around of temporarily installing an older version of tar to recover source packages created with older archives, none of this should rise to the level of release criticality for the tar package itself in my mind. Having said that, I just forwarded this bug upstream and hope we'll get some useful advice. Bdale
I agree that the issue of older versions not unpacking things made by newer versions is not an important bug. I added that as an FYI. I do think this bug qualifies as RC even though there is a workaround. The workaround is non-trivial and requires Debian Developers to research the issue for a process that is normally entirely transparent. Tools like pristine-tar become much less valuable if they introduce their own restrictions to the packaging process. It changes "use pristine-tar, and things will work better" into "use pristine-tar and you'll have to learn some new workarounds, but it'll probably save you time".
Bdale Garbee <bdale@gag.com> ha escrit: I have only a vague idea about pristine-tar. Does it do extraction using the -T option and a list of members to extract? What other options are used? Yes, that's right. The intent of dee7e3f16e74 was to fix the functionality of the --verbatim-files-from option[1]. That's reflected in the description of that option as well: "-T reads file names verbatim (no escape or option handling)" The --unquote option effectively defeats the purpose of --verbatim-files-from, so I don't see much sense in using the two options together, especially given that --unquote behavior is the default. So, what kind of patch is needed? I would suppose you'd like --verbatim-files-from to not do any option handling, while still unquoting file names. Am I right? I quite agree with the suggestion. Regards, Sergey [1] http://lists.gnu.org/archive/html/bug-tar/2017-10/msg00015.html
I'm looking into #901952 (pristine-tar failing to checkout out old files with non-printable unicode characters) and think that might be solved with the attached patches, by calling tar with --null and giving it a copy of the manifest file that is unescaped and NUL-terminated. What I haven't looked into yet is whether it will break files generated with 1.46, as that the format incompatibly without changing the version of the delta (though perhaps that can be mitigated by an additional variant at checkout time). What are your opinions on that? Is this worth trying to get into buster? Bernhard R. Link
* Bernhard R. Link <brlink@debian.org> [190330 07:58]: Attached version with some small fixes, more tests convering those changes (and compatibility with previous versions), and an additional variant run on checkout time to also handle files commited with 1.46. Bernhard R. Link
Hi all, I'm pretty sure that the fix for bug #933031 (in pristine-tar) works around the issue. Shall we close this bug as won't fix? Paul