Dear Maintainer,

I'm trying to update a package that contains a submodule. The package:
https://salsa.debian.org/arnaudr/golang-github-oschwald-maxminddb-golang-v2/

The watch file is as such:

```
Version: 5

Mode: git
Source: https://github.com/oschwald/maxminddb-golang.git
Matching-Pattern: refs/tags/v@ANY_VERSION@
Git-Modules: all
```

When I run uscan, it downloads the latest version, as expected:

```
golang-github-oschwald-maxminddb-golang-v2_2.1.0.orig.tar.xz -> golang-github-oschwald-maxminddb-golang-v2-2.1.0.tar.xz
golang-github-oschwald-maxminddb-golang-v2-2.1.0.tar.xz
```

However, in the tarball, the submodule directory test-data/ is present
but empty.

Consequently, when I later import the new orig tarball with:

```
gbp import-orig ../golang-github-oschwald-maxminddb-golang-v2_2.1.0.orig.tar.xz
```

The test-data/ directory gets removed.

Am I doing something wrong, or is it a bug?

Thanks,

Arnaud
--- /etc/devscripts.conf ---
Empty.

--- ~/.devscripts ---
DEBRELEASE_DEBS_DIR=$HOME/$(pwd | cut -d/ -f4)/build-area
DEBRELEASE_UPLOADER=dput
DEBCHANGE_AUTO_NMU=no
DEBCHANGE_MULTIMAINT_MERGE=yes
DEBCHANGE_RELEASE_HEURISTIC=changelog  ## default already
DEBSNAP_DESTDIR=.
DEBUILD_LINTIAN_OPTS="--color always -I"
RMADISON_DEFAULT_URL=debian,new

Hi Arnaud,

On Sat, 08 Nov 2025 22:40:46 +0700 Arnaud Rebillout wrote:

 > I'm trying to update a package that contains a submodule. The package:
 >
https://salsa.debian.org/arnaudr/golang-github-oschwald-maxminddb-golang-v2/
 >
 > The watch file is as such:
 >
 > ```
 > Version: 5
 >
 > Mode: git
 > Source: https://github.com/oschwald/maxminddb-golang.git
 > Matching-Pattern: refs/tags/v@ANY_VERSION@
 > Git-Modules: all
 > ```
 >
 > When I run uscan, it downloads the latest version, as expected:
 >
 > ```
 > golang-github-oschwald-maxminddb-golang-v2_2.1.0.orig.tar.xz ->
golang-github-oschwald-maxminddb-golang-v2-2.1.0.tar.xz
 > golang-github-oschwald-maxminddb-golang-v2-2.1.0.tar.xz
 > ```
 >
 > However, in the tarball, the submodule directory test-data/ is present
 > but empty.

You don't seem to be doing anything wrong. I can't reproduce this
behaviour, even cloning your repository and ~/.devscripts configuration.

Can you please run `uscan -v` and post the output (or a link to it)?

Thank you,

Hugh

Hello Hugh,

Thanks for your quick answer!


I tried again to reproduce the issue from a clean "gbp clone" repo, and
I couldn't reproduce it...

But then I can still reproduce it from the git repo that I work with
usually, so I compared the logs from uscan -v, and I understand now that
the difference is due to the presence of the upstream branch.

So here's a script to reproduce the issue (attached as I'm afraid that
the formatting gets messed up if I write it in the email).

SEE ATTACHMENT

What we can see from the logs is that uscan works differently, depending
on whether there's an upstream branch. It seems that without a upstream
branch, it clones the repo, and therefore includes the submodule.
However when there's already a upstream branch, it doesn't run "git
clone", and for some reason it doesn't include the submodule, even
though one can see that it IS included in the upstream branch (just run
"git switch upstream").

Best,

Arnaud

I meant "upstream remote" and not "upstream branch", sorry for the awful
confusion...

Arnaud

On Thu, 20 Nov 2025 17:04:40 +0700 Arnaud Rebillout wrote:
 > On Mon, 17 Nov 2025 10:55:26 +0700 Arnaud Rebillout wrote:
 > > But then I can still reproduce it from the git repo that I work with
 > > usually, so I compared the logs from uscan -v, and I understand now
that
 > > the difference is due to the presence of the upstream branch.
 >
 > I meant "upstream remote" and not "upstream branch", sorry for the awful
 > confusion...

No problem. I knew what you meant to say.

I have a partial patch on Salsa [1] for a similar problem also using the
git upstream mode. [2]

The patch doesn't quite fix your bug, but `uscan -vv` debug output shows
the right actions are happening.

Hugh

[1] https://salsa.debian.org/debian/devscripts/-/merge_requests/594

[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1120533

Thanks for the pointer. I applied your patches, ran with "-vv", here's
what I found.

On
https://salsa.debian.org/arnaudr/golang-github-oschwald-maxminddb-golang-v2/
the directory "test-data" is NOT a submodule.

Background: initially this repository was created with the idea of
keeping upstream git history, so it was a git clone of upstream, however
the git submodule was NOT included. Later on I found out that the
submodule was needed for the tests. So I edited d/watch to switch to git
mode, and I imported a new upstream version. As a result, gbp/uscan
imported the content of this git submodule, however it came in under the
form of this extra commit that sits between the upstream branch and the
debian packaging branch. It's this commit:
11b4dcb8fb045b8a31f32f26b5a5e5ca8f55e2d7.

So the data is here, but it's under the form of a git commit, and not of
a git submodule. "git submodule status" shows nothing.

Now, when I run uscan, when it sees a upstream remote, it assumes the
git submodule is properly configured, and it runs this:

git submodule foreach --recursive git archive --format=tar \
  --prefix=golang-github-oschwald-maxminddb-golang-v2-2.1.0/$displaypath/ \
  --output=../$sha1.tar HEAD

But this command has no effect (it doesn't even fail, it's just no-op).

I tried to find a way out of this situation, but surprisingly it's very
hard, didn't even manage after an hour, I'm giving up for now.

I hope that sheds some light on the issue. If you have any
recommendation, I'm happy to hear it.

Arnaud

On Fri, 21 Nov 2025 15:29:22 +0700 Arnaud Rebillout wrote:

 > Background: initially this repository was created with the idea of
 > keeping upstream git history, so it was a git clone of upstream, however
 > the git submodule was NOT included. Later on I found out that the
 > submodule was needed for the tests. So I edited d/watch to switch to git
 > mode, and I imported a new upstream version. As a result, gbp/uscan
 > imported the content of this git submodule, however it came in under the
 > form of this extra commit that sits between the upstream branch and the
 > debian packaging branch. It's this commit:
 > 11b4dcb8fb045b8a31f32f26b5a5e5ca8f55e2d7.
 > [...]
 > I tried to find a way out of this situation, but surprisingly it's very
 > hard, didn't even manage after an hour, I'm giving up for now.
 >
 > I hope that sheds some light on the issue. If you have any
 > recommendation, I'm happy to hear it.

I have spent a lot of time investigating this issue and had the same
problem, even with my own packages.

The solution is to use a Git clone of the upstream repository as the
starting point for creating Debian packaging.

Otto Kekäläinen has a written a guide that works well, even with
submodules. [1]

Note that when calling `git clone`, you need to add
`--recurse-submodules` to the command line too.

I did a quick test using Otto's instructions on your repository. The
result is at [2] and work as expected with uscan.

Let me know how you go.

Hugh

[1] https://optimizedbyotto.com/post/debian-packaging-from-git/

[2] https://salsa.debian.org/hmc/golang-github-oschwald-maxminddb-golang-v2

[2] is not reachable, maybe it's private?

Ideally I wanted to be able to fix the existing git repo, rather than
restart from scratch. The repo I gave in this bug report was under my
own namespace (because edited for the purpose of this bug report), but
in fact I have two repositories that are facing this issue, and already
uploaded, so I'd prefer to avoid force-pushing.

But this morning I spend a couple of hours on that again, failed again,
I keep banging my head against a wall it feels, so at this point any
solution will do. And thanks for spending the time on this issue :)

Best,

Sorry, that should be fixed now.

To be honest, I'm not sure the issue is fixable. There is no common
commit history for Git to work with, so even merging upstream and Debian
repositories is not possible.

     git add remote ...
     git merge <upstreamvcs>

If it were possible, Git would read the submodule information.

While it's a pain to start over, it is probably the least painful option
right now. Plus, you can more forwards.

I'll keep thinking about this.

Hugh

So I `git clone` your repo, then I found out that... I need to pass
--recurse-submodules, of course.

So I `git clone --recurse-submodules` and all looks good. The
`test-data` directory is populated, and I get some output with `git
submodule status`, showing that the submodule was properly initialized.

Now let's try an update this repo, we're going to update from git HEAD
since there's no new upstream release:

```
# create upstream branch
git branch upstream 8b6ffca

# edit d/watch to track git head
Mode: git
Source: https://github.com/oschwald/maxminddb-golang.git
Matching-Pattern: HEAD
Git-Modules: all
Git-Pretty: 2.1.0+git%cd.%h

# commit
git add debian
git commit -m "Update watch file"
```

Then I run `uscan`, it gets me the right tarball, including the
submodule. All good.

Now, I want to import that in my git repo, and also track upstream git
history:

```
# Add and fetch upstream remote
git remote add upstreamvcs https://github.com/oschwald/maxminddb-golang.git
git fetch upstreamvcs

# Import
gbp import-orig
../golang-github-oschwald-maxminddb-golang-v2_2.1.0+git20251120.4d0333b.orig.tar.xz
--upstream-vcs-tag=4d0333b --debian-branch=debian/latest
```

And... Alas! gbp added the test-data directory to the "New upstream
version" commit, and now the submodule is gone, `git submodule status`
returns nothing. We're back to the beginning.

Did I do something wrong?

Off-Topic: I had to be careful to add the upstream vcs only AFTER
running uscan, otherwise I hit #1120533 (uscan doesn't fetch the right
git HEAD), which doesn't seem to be fixed in devscripts 2.25.27.

Best,

Arnaud

I updated the test repository with debian/gbp.conf.

What I had in the test repository debian/watch is fine:

====
Version: 5

Mode: git
Source: https://github.com/oschwald/maxminddb-golang.git
Matching-Pattern: refs/tags/v@ANY_VERSION@
Git-Modules: all
====

Excellent, although you don't need to do this, as we don't want a source
tarball (see below).
Try: `gbp import-orig --uscan` for automation.

Try what I said above. Also, try following Otto's guide, even just from
your local machine. It's quick to do, and you can skip a lot of steps,
since you only want the branch setup.

Yes, #1120533 is fixed in the devscripts repository but not yet
uploaded. If you use tags, the patch isn't required. But just in case,
you can run PERL5LIB=/path/to/devscripts/clone/lib/
/path/to/devscripts/clone/scripts/uscan.pl

Hugh

Yes, but I want to test importing a new orig tarball, and there's no
such, since your test repo is based on the latest release already.
That's why I switched to git HEAD, but that makes the example more
complicated.

Ok, so to keep it simple, I pushed yet another git repo :) ->
https://salsa.debian.org/arnaudr/golang-github-oschwald-maxminddb-golang-v2-with-submodule

This is just your test repo, but with the packaging commits based on the
previous tag (v2.0.0), and with the d/changelog edited accordingly. I
also created the upstream/latest branch.

So you can get it with:

```
git clone --recurse-submodules
https://salsa.debian.org/arnaudr/golang-github-oschwald-maxminddb-golang-v2-with-submodule.git
cd golang-github-oschwald-maxminddb-golang-v2-with-submodule
git branch --track upstream/latest origin/upstream/latest 
```

At this point, the submodule is setup, the two branches are setup,
d/watch and d/gbp.conf are configured. It seems to me that we're ready
to import the latest upstream version 2.1.0?

Not yet, `gbp import-orig --uscan` fails, rightfully so, because we
didn't setup the remote for upstream, and it's needed for the gbp
upstream-vcs-tag option. So let's configure the remote:

```
git remote add upstreamvcs https://github.com/oschwald/maxminddb-golang.git
git fetch upstream vcs
```

This time, `gbp import-orig --uscan` does its job, it imports versions
v2.1.0, however... It replaces the submodule with a git commit.

Please tell me if I'm stubbornly missing something.


Ah, you're right, sorry about that. Now I'm using latest devscripts from
git, to have the latest changes that you committed.

Best,

Arnaud

Hi Arnaud,

Sorry for my delayed reply.

I believe this is a bug in gbp import-orig. When it imports the tarball,
it clobbers the submodule somehow.

I believe I've found the solution, and it looks to me like we have been
over-thinking this.

Like you, I started a new repository based on the v2.0.0 tag.

git clone --recurse-submodules
https://salsa.debian.org/hmc/maxminddb-golang.git
git switch upstream/latest
git remote add upstreamvcs https://github.com/oschwald/maxminddb-golang.git
git fetch --recurse-submodules upstreamvcs
git merge v2.1.0
git switch debian/latest
uscan -v
# git push --all
# git push --tags

For me, this sequence updates the upstream repository and generates the
correct tarball while keeping `git submodule show` enabled.

Let me know how you go with this.

Hugh

Hello Hugh,

I have the same guess, but didn't check gbp's code.
to update to 2.1.0 is missing?

On my side, I have found two solutions.

1) If the goal is to have submodules and track upstream git history,
then I can use a "pure git" workflow. No uscan, no gbp, just two remotes
(salsa and upstream). When I want to update my repo, I git fetch and git
merge.

That's roughly what's described at
https://go-team.pages.debian.net/workflow-changes.html#wf-2017-11-upstream-branch,
section "Packaging new upstream release". However this workflow never
really took off in the Go packaging team.

2) If the goal is to use uscan and gbp, then I can just NOT track
upstream git history. So not upstream remote, then uscan will use "git
ls-remote" and get the submodule just fine. gbp imports it in a commit
when it changes. It seems to work well.

I decided to go for solution 2, because the 2 packages maxminddb-golang
and mmdbinspect are maintained within the Go team, and teammates will
basically do "gbp import-orig --uscan" to update those repositories. So
I need this command to work out of the box. I don't care too much about
git history (it's nice to have and it's standard practice in the team,
but in this particular case we'll do without).

One last thought on that: I really don't know what's the best way to
approach packaging when upstream uses git submodules. You surely put
more thoughts than me on that topic. But if you think that the right
approach is to also have the submodules proper on salsa.debian.org, then
that kind of clashes with tools like debcheckout / gbp-clone that don't
setup submodules by default (TBH I didn't check, maybe this kind of
things is being discussed and adjusted). It means that co-worker that
want to work on my submodule-enabled packaging repo won't get their
local git repo properly setup when they clone it, and good luck figuring
out about the submodules.

On that note, well, thanks a lot for spending so much time on that. I
let you decide what to do with this bug report.

Thanks again,