I did a tag2upload of 'knot' which has a peculiar upstream tarball
filtering, using gbp import-orig filter.  As a side note: this
configuration is not my invention, and I consider changing it to use
upstream git instead (which has PGP signed tags), although they use git
submodules so not compatible with tag2upload anyway IIRC.

Quoting debian/watch:

Version: 5
Source: https://secure.nic.cz/files/knot-dns/
Matching-Pattern: (?:|.*/)knot(?:[_\-]v?|)(\d\S*)\.(?:tar\.xz|txz|tar\.bz2|tbz2|tar\.gz|tgz)
Dversionmangle: s/\+hotfix//
Pgpsigurlmangle: s/$/.asc/
Uversionmangle: s/-((alpha|beta|rc)\d*)$/~$1/

So that fetches the "real" upstream tarball with PGP signature.  I think
it puts the unfiltered tarball into the pristine-tar branch.

Then there is debian/gbp.conf:

[DEFAULT]
debian-branch = debian/master
pristine-tar = True
upstream-branch = upstream/3.5
upstream-vcs-tag = v%(version)s
sign-tags = True
[dch]
meta = 1
[import-orig]
filter = [
 'configure',
 '*/Makefile.in',
 '*/*/Makefile.in',
 '*/*/*/Makefile.in',
 'install-sh',
 'ltmain.sh',
 'm4/libtool.m4',
 '/src/lib*/version.h',
 'INSTALL',
 'aclocal.m4',
 'ar-lib',
 'depcomp',
 'compile',
 'missing',
 'test-driver',
 'config.guess',
 'config.sub',
 ]
filter-pristine-tar = False

Which filters out files when doing the git import.

Is this mode supported?

I'm having trouble translating the error message I got below into
actions to make tag2upload work for 'knot'.

/Simon

Debian tag2upload service <dgit-owner@debian.org> writes:

Simon Josefsson writes ("Bug#1143850: tag2upload pristine-tar with gbp import-orig filter?"):

OK.

I think the result of this is that the pristine-tar branch specifies
a tarball which is not the one you want to use.

Is that right?  What .orig are you expecting?  How are you expecting
the tag2upload service to generate it?

When we designed the tag2upload pristine-tar support we assumed that
the presence of data for foo_NNN.orig.tar.gz in the pristine-tar
branch meant that you want to use that foo_NNN.orig.tar.gz as your
.orig.tar.gz.

There is not currently any way to disable the pristine-tar support in
git-debpush.  Probably there should be such a thing for use as a
workaround but we should consider how to detect this scenario.
Reading gbp.conf and trying to second-guess its behaviour doesn't seem
like a good idea.

We could detect that the .id file in the pristine-tar data is
different to the upstream/NNNN tag.  But how would we know that the
upstream/NNNN tag is to be preferred?

Ian.

I investigated this properly.  What I wrote previously was wrong.
(I don't use gbp very much so I'm not familiar with its workings.)

Ian Jackson writes ("Bug#1143850: tag2upload pristine-tar with gbp import-orig filter?"):

In particular, ^ I think that comment from me is wrong.  And the rest
of my message followed from it, so was also wrong.


The discrepancy report in the tag2upload log mentions various files,
most of which seem like autoconf output (ie, intermediate build
products).

Arbitrarily picking aclocal.m4 as a litmus test, that file is present
in the following forms of the package:

 * knot 3.5.6-2 in the Debian archive
 * knot_3.5.6.orig.tar.xz in the Debian archive (cause of ^)
 * dgit clone knot (consequence of ^)

It is absent in the following forms of the package:

 * pristine-tar metadata file knot_3.5.6.orig.tar.xz.id
 * v3.5.2 tag (ie, presumably, upstream git)
 * upstream/3.5.2 tag
 * debian/master branch on salsa
 * debian/3.5.6-2 tag (resulted in tag2upload failed job 5733)

I note that the pristine-tar file knot_3.5.6.orig.tar.xz.delta is half
a megabyte, I think because it is representing all these autogenerated
files.  That is the only representation of (say) aclocal.m4 on salsa.

As you may know, it is a principle of the tag2upload system that the
git tree that you upload corresponds precisely to the generated source
package.  Nothing should end up in the unpacked source package that
buildds and users get from archive, that isn't in your git branch.

This is both a very useful promise to the uploading user and others
(it avoids ever having to audit or review a source package, and
eliminates a hidden and rarely-examined data flow) and a necessary
design principle for the bidirectionality of dsc<->git gatewaying.

But, apparently, with this workflow, the source package contains the
automake output, but that isn't represented in git at all (except via
the pristine-tar delta file).

So indeed, that's not supported.  I don't think it *can* sensibly be
supported.  (And I would recommend against this workflow for other
reasons too. [1])

I'm tagging this bug wontfix, so that it can stand as a discoverable
record of the lack of support for this approach (and provide a place
for discussion of possible workarounds, amelioration, or whatever).

But, please be assured that I very much valued your report.  The
wontfix tag should not be taken as any kind of criticism.

Regards,
Ian.

[1] Consider for example the xz backdoor.  If adopted for the Debian
xz package, this workflow would have made the crucial backdoor trigger
entirely invisible to git commands run even by the Debian maintainer
on their packaging branch.  It would only show up if the maintainer
did a source debdiff after building the source package.  This is even
worse than the usual way of using gbp import-orig.