Dear Maintainer,

In Kali Linux, we package an upstream that uses Git LFS to store a big
file (a GeoIP database). The upstream is at:
https://github.com/rsmusllp/king-phisher

In a previous version, upstream used to version the database "as is", it
was a regular file in the Git repo. Then in a subsequent version, they
switched to use Git LFS to store this file.

Gbp doesn't handle this transition well, apparently this is due to the
combination of:
* "gbp clone" disabling Git attributes (hence git lfs)
* however "gbp import-orig" does no such thing

I'm the person who updated this package, so my local copy of
king-phisher doesn't have the git attributes disabled, and everything
works fine with me. However other folks who clone the repo complain, as
it leads to an unclean git checkout, and I don't know what's the way
forward.

For a longer (and hopefully crystal-clear) explanation of the issue, I
prepared a Git repo and a walkthrough to reproduce the issue. There we
go :)

Let's first clone the king-phisher package *before* upstream switched to
Git LFS:

  $ gbp clone https://gitlab.com/arnaudr/king-phisher.git
  $ cd king-phisher
  $ cat .gitattributes
  cat: .gitattributes: No such file or directory
  $ ls -l data/server/king_phisher/GeoLite2-City.mmdb
  -rw-r--r-- 1 arno arno 61615395 Dec 15 11:53 data/server/king_phisher/GeoLite2-City.mmdb

So at this point, the file GeoLite2-City.mmdb is versioned "as is", it
is a regular file.

Now let's update the package to latest Git snapshot:

  $ gbp import-orig --uscan
  gbp:info: Launching uscan...
  Downloading data/server/king_phisher/GeoLite2-City.mmdb (62 MB)
  gbp:info: Using uscan downloaded tarball ../king-phisher_1.15.0+git20221107.orig.tar.xz
  What is the upstream version? [1.15.0+git20221107]
  gbp:info: Importing '../king-phisher_1.15.0+git20221107.orig.tar.xz' to branch 'upstream'...
  gbp:info: Source package is king-phisher
  gbp:info: Upstream version is 1.15.0+git20221107
  gbp:info: Replacing upstream source on 'kali/master'
  gbp:info: Successfully imported version 1.15.0+git20221107 of ../king-phisher_1.15.0+git20221107.orig.tar.xz

The line "Downloading data/server/king_phisher/GeoLite2-City.mmdb (62
MB" comes from git lfs, which is downloading the file. And here's the
situation now:

  $ cat .gitattributes
  *.mmdb filter=lfs diff=lfs merge=lfs -text
  $ cat .git/info/attributes
  cat: .git/info/attributes: No such file or directory
  $ ls -l data/server/king_phisher/GeoLite2-City.mmdb
  -rw-r--r-- 1 arno arno 61615395 Dec 15 11:56 data/server/king_phisher/GeoLite2-City.mmdb

So we can see the git lfs thinggy, and we can see that
.git/info/attributes' doesn't exist (more on that below).

Let's push that work (I prepared a fork to push changes):

 $ git remote add arnaudr2 git@gitlab.com:arnaudr/king-phisher2.git
 $ git push arnaudr2 : --follow-tags
  Locking support detected on remote "arnaudr2". Consider enabling it with:
    $ git config lfs.https://gitlab.com/arnaudr/king-phisher2.git/info/lfs.locksverify true
  Locking support detected on remote "arnaudr2". Consider enabling it with:
    $ git config lfs.https://gitlab.com/arnaudr/king-phisher2.git/info/lfs.locksverify true
  Locking support detected on remote "arnaudr2". Consider enabling it with:
    $ git config lfs.https://gitlab.com/arnaudr/king-phisher2.git/info/lfs.locksverify true
  Locking support detected on remote "arnaudr2". Consider enabling it with:
    $ git config lfs.https://gitlab.com/arnaudr/king-phisher2.git/info/lfs.locksverify true
  Uploading LFS objects: 100% (1/1), 62 MB | 3.4 MB/s, done.
  Enumerating objects: 112, done.
  Counting objects: 100% (82/82), done.
  Delta compression using up to 8 threads
  Compressing objects: 100% (46/46), done.
  Writing objects: 100% (49/49), 19.06 KiB | 19.06 MiB/s, done.
  Total 49 (delta 29), reused 5 (delta 0), pack-reused 0
  remote:
  remote: To create a merge request for pristine-tar, visit:
  remote:   https://gitlab.com/arnaudr/king-phisher2/-/merge_requests/new?merge_request%5Bsource_branch%5D=pristine-tar
  remote:
  remote:
  remote: To create a merge request for upstream, visit:
  remote:   https://gitlab.com/arnaudr/king-phisher2/-/merge_requests/new?merge_request%5Bsource_branch%5D=upstream
  remote:
  To gitlab.com:arnaudr/king-phisher2.git
     c5db68b..dbf4ce7  kali/master -> kali/master
     d9ec6a5..e4e9390  pristine-tar -> pristine-tar
     be63910..f4f0fae  upstream -> upstream
   * [new tag]         upstream/1.15.0+git20221107 -> upstream/1.15.0+git20221107

And now, the issue: when we clone this repo with gbp, the resulting repo
is not clean. Let's try:

  $ gbp clone -v git@gitlab.com:arnaudr/king-phisher2.git
  gbp:debug: ['git', 'rev-parse', '--show-cdup']
  gbp:info: Cloning from 'git@gitlab.com:arnaudr/king-phisher2.git'
  gbp:debug: ['git', 'clone', '--quiet', 'git@gitlab.com:arnaudr/king-phisher2.git']
  gbp:debug: ['git', 'rev-parse', '--show-cdup']
  gbp:debug: ['git', 'rev-parse', '--is-bare-repository']
  gbp:debug: ['git', 'rev-parse', '--git-dir']
  gbp:debug: ['git', 'rev-parse', '--show-cdup']
  gbp:debug: ['git', 'rev-parse', '--is-bare-repository']
  gbp:debug: ['git', 'rev-parse', '--git-dir']
  gbp:debug: Will track branches: ['kali/master', 'upstream', 'pristine-tar']
  gbp:debug: ['git', 'show-ref', '--verify', 'refs/remotes/origin/kali/master']
  gbp:debug: ['git', 'show-ref', '--verify', 'refs/heads/kali/master']
  gbp:debug: ['git', 'show-ref', '--verify', 'refs/remotes/origin/upstream']
  gbp:debug: ['git', 'show-ref', '--verify', 'refs/heads/upstream']
  gbp:debug: ['git', 'branch', 'upstream', 'origin/upstream']
  gbp:debug: ['git', 'show-ref', '--verify', 'refs/remotes/origin/pristine-tar']
  gbp:debug: ['git', 'show-ref', '--verify', 'refs/heads/pristine-tar']
  gbp:debug: ['git', 'branch', 'pristine-tar', 'origin/pristine-tar']
  gbp:debug: ['git', 'show-ref', '--verify', 'refs/remotes/kali/master']
  gbp:debug: ['git', 'config', 'user.name', 'Arnaud Rebillout']
  gbp:debug: ['git', 'config', 'user.email', 'arnaudr@kali.org']
  gbp:debug: ['git', 'ls-tree', '-z', '-r', '-l', 'HEAD', '--']
  gbp:debug: Found non-empty .gitattributes: b'.gitattributes'
  gbp:debug: Configuring Git attributes

  $ cd king-phisher2

  $ git status
  On branch kali/master
  Your branch is up to date with 'origin/kali/master'.

  Changes not staged for commit:
    (use "git add <file>..." to update what will be committed)
    (use "git restore <file>..." to discard changes in working directory)
  	modified:   data/server/king_phisher/GeoLite2-City.mmdb

  no changes added to commit (use "git add" and/or "git commit -a")

  $ cat .gitattributes
  *.mmdb filter=lfs diff=lfs merge=lfs -text
  $ cat .git/info/attributes
  # Added by git-buildpackage to disable .gitattributes found in the upstream tree
  [attr]dgit-defuse-attrs  -text -eol -crlf -ident -filter -working-tree-encoding
  * -export-ignore
  * dgit-defuse-attrs
  $ ls -l data/server/king_phisher/GeoLite2-City.mmdb
  -rw-r--r-- 1 arno arno 61615395 Dec 15 12:12 data/server/king_phisher/GeoLite2-City.mmdb

As we can see above (my interpretation):
* during the 'gbp clone' step, the 'git clone' command will actually
  trigger git lfs, and download the GeoLite2 database (assuming you have
  the package git-lfs installed on your machine).
* then at the end of the gbp clone operation, we can see "Configuring
  Git attributes", and this is when gbp creates the file
  .git/info/attributes
* as a result, the git repo is in an unclean state

To bring back the Git repo in shape, we can either:

1) Undo what gbp just did:

    rm -fr .git/info/attributes

2) Undo what git lfs did:

    $ git checkout data/server/king_phisher/GeoLite2-City.mmdb
    Updated 1 path from the index
    $ cat data/server/king_phisher/GeoLite2-City.mmdb
    version https://git-lfs.github.com/spec/v1
    oid sha256:a253d9cd68fe17b00087da24375f31f07cd4bb3852dc5fe3afe37b8f59e5abd0
    size 61615395

As we can see with option 2), the LFS file becomes a short metadata
file, because that's what's really in the Git repo, before "git lfs"
replaces it with the "real file" that it fetches from somewhere else.

  == Questions

How does the git LFS files should be handled? When "gbp clone" disables
the gitattributes, it disables Git LFS in turn: is it intended, or not?
Does gbp has an opinion on that?  In any case, it seems that disabling
the gitattributes after 'git clone' has run is too late, because the Git
LFS objects were already fetched.

Thanks for reading, and please help me understand how we should handle
those LFS files.

Arnaud

Hi,

Or use `gbp clone --git-defuse-attributes=off ...` ?

Cheers,
 -- Guido

Indeed! Thanks, I didn't notice this option, and I can confirm that it
solves the issue (the exact option name is "--defuse-gitattributes=off"
actually).

I took the time to look a bit at this option and where it comes from
(seemingly, the goal is that gbp is compatible with other tools such as
dgit). There's still something that strikes me.

IIUC, it's *only* "gbp clone" that will defuse the git attributes.

So developer A does "gbp import-orig", brings in the offending
attributes: still they are not defused on their local copy. Then
developer B does "gbp pull", and then no change: gbp does not defuse
attributes during pull, so attributes are still enabled. Then developer
C does "gbp clone", and they possibly get a different working copy,
because in this case only, attributes are defused.

It seems that this behavior is prone to create problems, albeit it might
be detected by CI (not sure what Salsa CI does in details), and it's
also kind of a niche issue anyway.

We took a look at our packaging repos in Kali Linux (~ 600 repos), and
we found 3 of them that are unclean on `gbp clone`. It's solved by
setting --defuse-gitattributes. So we will probably add this argument to
our .mrconfig, and make sure to use it all the time. We'll see where it
gets us.

I'm really not familiar with git attributes, so I don't have any
recommendation, or anything else to say, I'm merely reporting in case it
helps.

Cheers,

Arnaud

On Wed, 21 Dec 2022 15:04:43 +0700 Arnaud Rebillout <arnaudr@kali.org>  wrote:

 > It's solved by setting --defuse-gitattributes.

I meant: setting --defuse-gitattributes=off, of course.

Hi,

Changing git attributes is certainly a problem. If someone wants to send
patches to e.g. warn the user about that, that would be useful. That
said I'd expect maintainers to filter out .gitattributes from imported
tarballs (e.g. via uscan / debian/changelog or gbp itself)

I think what we could do is:

- warn about git found in upsteam repos / tarballs
- warn about changing git attributes
- allow to disable the either of these via d/gbo.conf

Are these repos based on upstream's git or on tarballs that got those
gitattributes imported?

We have some tests that test defusing git attributes in gbp. If there's
any of your use cases missing, feel free to add those. That is often
a good step to document expected behavior.

Cheers,
 -- Guido