`dgit push-source' handles a .gitignore file in the tarball, which
is not in HEAD in git, by adding a commit:

dgit view: creating patches-applied version using gbp pq
dgit view: creating patch to represent .gitignore changes
Commit patch to update .gitignore

[dgit (16.8) update-gitignore-quilt-fixup]

[dgit-view fb07560] Commit patch to update .gitignore
 2 files changed, 31 insertions(+)
 create mode 100644 debian/patches/auto-gitignore
dgit view: created (commit id fb07560c9f5ec92a4064dac5e2be5ed575c08faf)


git debpush doesn't handle this situation; all I get is a failure
email _after_ the attempted upload:

builder:work$ tag2upload-obtain-origs p=libwww-mechanize-shell-perl v=0.63-1 s=unstable u=20a43e5ec72752935a8c26137148a2beb8d41c29 pristine_tar=1ca16b56ab5a7b07a5ae6d6cd47bde32e31ee364
+ dgit --build-products-dir=../bpd -plibwww-mechanize-shell-perl download-unfetched-origs --write-sha256sums=.git/dgit/t2uoo/origs.sha256sums
canonical suite name for unstable is sid
package in target suite is different upstream version, 0.62
no .origs for package libwww-mechanize-shell-perl upstream version 0.63
# no orig(s) in archive, generating
+ pristine-tar checkout -- ../bpd/libwww-mechanize-shell-perl_0.63.orig.tar.gz
pristine-tar: successfully generated ../bpd/libwww-mechanize-shell-perl_0.63.orig.tar.gz
fatal: path '.gitignore' does not exist in 'caea8b14b213a4a16c5edfb95dca6de58a9d9b56'
cmp: EOF on - which is empty
pristine-tar .id file and ../bpd/libwww-mechanize-shell-perl_0.63.orig.tar.gz disagree about .gitignore ?
tar: 487: Child returned status 1
tar: Exiting with failure status due to previous errors
t2u processor [dgit-repos-server]: failed command: ./ssh-builder 'tag2upload-builder@tag2upload-builder-01.debian.org' 'env -C
/srv/builder.tag2upload.debian.org/tmp/autopkgtest-virt-docker.shared.k0esnzwj/downtmp/work tag2upload-obtain-origs p=libwww-mechanize-shell-perl v=0.63-1 s=unstable
u=20a43e5ec72752935a8c26137148a2beb8d41c29 pristine_tar=1ca16b56ab5a7b07a5ae6d6cd47bde32e31ee364'
tag2upload-obtain-origs failed, failed with error exit status 2


That's a bit unfortunate …



Cheers,
gregor
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEE0eExbpOnYKgQTYX6uzpoAYZJqgYFAmql0OJfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEQx
RTEzMTZFOTNBNzYwQTgxMDREODVGQUJCM0E2ODAxODY0OUFBMDYACgkQuzpoAYZJ
qgZJsQ/5AYuKJQXFSjzlLe2j7WYnQooUBJ2F0CeN5WfXdVIG55rXh2a9RIK44Ief
Lds8mGnxwdKRmL6XneuvLApNDKgkA/ohUbIwGdxGRwBz3OxOZ8dHTEvCG/pPGVIl
An0PBA/qXaCh7bnkft6SWn2xSBJvJZIVEwpikUd1B0OsHIvkoUazxv4JFvb7g7sg
vDhqb8lDEXrZo2HNYac8SAcwO3wvI9HBx7xEcqZ7QIULBjHGNG1Ayb+FN+NossLi
+Xwk5ISM4zlWCZNSX5XJ7CoW0XAIoeFgdQa5OeUaq6EE4pqaelI2oh5cR28niSHB
mkFaQgZcEXUmu3iqE6U17p6M6Qg7Kw7qJcxFaDuAWP+29ep0KKuIcZdpuJR+Qwmh
P2Pak4MhoxAg0bt0aEUCXitAdbV5FYT1c1ddGdHI/KR3JxKYxR7u9pUl70BUmTeQ
+fI165WR+9LNMmMXyG8kGq6GbsBzRWcEaYMJ33A47k2qHEB/ahTqRbRItW2Dwx/f
3AdLm2DkRHLuRK20VvgkJz61Y+liZn0DFB9131N2BuRHkIy0lEHs+7FVMNExhBwq
oywMG+eC6zMZQXoyZr1EvWIRuuN233jbuE+whD73zn93HkyqEC0RixUvGAEjGfia
hsQzdfxYni7PU2ulAqEK5mpfMOUBz6nmcKe9XqVErsb3A6pMm5c=
=PRoQ
-----END PGP SIGNATURE-----

Hi.  Thanks for the report.

gregor herrmann writes ("Bug#1147543: git-debpush stumbles over .gitignore"):
...
...

Here, git-debpush made a tag saying
  upstream-tag=upstream/0.63
  upstream=20a43e5ec72752935a8c26137148a2beb8d41c29

According to the current tag2upload tag spec, that means that the
.orig tarball is supposed to be treesame to that tag and commit.

We could change the spec.

But one of the things that the user is supposed to be able to rely on
is that the source package contents is predictable from the user's git
tree.  That makes it safe for the user to work solely in git.

We could relax this requirement for .gitignores.  But that would mean
that the source package, and the canonical dgit git branch, would
contain arbitrary .gitignore(s) that are not visible in the user's
HEAD.  That seems hazardous.  While a .gitignore is "just" metadata,
it can be quite surprisingly change the way git behaves for a user who
is manipulating the package with git.  See Simon's comment
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=908747#47

Another thing we could do is change the behaviour of git-debpush, so
that it would specify some different tag and tree.  But there is no
such tag and tree in this scenario (and I doubt we'd want git-debpush
to make one) - and git-debpush couldn't even detect this situation
without running pristine-tar locally.

The underlying difficulty is that in this situation, the .gitignore
contents are expressed only in the xdelta diff in the pristine-tar
branch.  In general, this is a very bad way to convey intended source
package contents.

(dgit push doesn't have this difficulty, because it has the tarball
right there already.  Mayabe the user ran pristine-tar to generate it,
but dgit can't detect that and in that case that's the user doing; to
run pristine-tar safely, the user ought to manually do the kind of
checks that the tag2upload service does.  Most users don't, I bet.
See also my comments about the xz-utils attack.)

The other possible workaorund would be to elide the upstream
gitignores when making source packages, and canonical git branches,
like dpkg-source does.  That would be a DFSG violation.  (Yes, I am
saying that dpkg-source is automating DFSG violations - see below!)

Yes.

The underlying problem is #908747.

We have been muddling through with various workarounds (including the
one in dgit push where it makes a .gitignore patch).  But my analysis
above suggests that we have reached the limits of our ability to work
around the problem.

#908747 is 8 years old and such arguments as have been presented in
favour of the current behaviour are speculative, weak, and rebutted.

I think the next step would be to refer #908747 to the Technical
Committee.  I'm not sure I have the effort for that myself,
particularly after the LLM vote, and past harassment on the TC list (!).
But I would be very happy if you were to make that referral.

Ian.

Ian Jackson writes ("Re: Bug#1147543: git-debpush stumbles over .gitignore"):

Adding the appropriate blocking relationship.

Ian.

Ian Jackson writes ("Re: Bug#1147543: git-debpush stumbles over .gitignore"):

I presume the data in this package was made with gbp import-orig.

I wasn't aware that gbp import-orig deletes .gitignore from the
imported upstream git branch by default, when importing tarballs.
Is that really the case?

It seems weird because surely the Debian maintainer generally wants
the benefit of the upstream .gitignore.

So I think I am saying that maybe there is something weird in this
specific package.

(I never use gbp import-orig.)

Ian.

Thanks for the quick and detailed reply!

Right, that was my guess, thanks for the confirmation.

Which wouldn't help when t2u creates the .orig.tar.gz with
pristine-tar either, unless it does some "massaging" afterwords.
Which doesn't sound like a good idea …

Understood.

I'm not sure a change in dpkg-source would help in the case where the
upstream tarball (via pristine-tar) contains a .gitignore and the
packaging git's HEAD doesn't?

Right, sorry for not mentioning this.

TBH, I'm not sure if it does by default. But I've seen many packages
with a debian/gbp.conf which filters out .gitignore (or more .git*
stuff) for gbp-import-orig. And so does our (the Debian Perl Group's)
wrapper dpt-import-orig.

I'm afraid I don't agree; personally I don't want an upstream
.gitignore (which doesn't belong in a release tarball in the first
place IMO) to hide files from me, when I import a tarball into _my_
packaging git repo.

And many others :)

But yes, the situation I keep seeing is caused by (a specific use of)
gbp-import-orig and is not a bug in git-debpush. I just raised this
topic because of the different behaviour from dgit-push-source.


What I found in gbp-import-orig's documentation is an option
      --[no-]filter-pristine-tar
which, if I understand it correctly, also filters out the same file
from the tarball before passing (?) it the pristine-tar. I'm not sure
I like it and haven't tried it but it's there.


Cheers,
gregor