Hi!

Yesterday I've finally given tag2upload a try with the cubeb package.
It's wonderful!

Still, I've encountered a surprising issue related to git submodules.

cubeb upstream uses git submodules, but they are not needed in Debian.
In my tarball-based workflow, I've been able to ignore them, as the
tarball generated by Git(Hub) creates empty directories where submodules
should be. Fine.

When switching to git-debpush, I've also switched how I get upstream
code. Since I package Git snapshot, I followed the instructions in
dgit-maint-debrebase and now fetch the code from the upstream Git.

With this change, though, submodules are now properly represented in the
source. I still ignored them.

When pushing with git-debpush I then received an email with the
following error:

    Format `3.0 (quilt)', need to check/update patch stack
    canonical suite name for experimental is rc-buggy
    dgit (build host): found orig(s) in .. missing from build-products-dir, transferring:
    dgit (build host): hardlinked orig from ..: cubeb_0.0~git20250529.78ee5f0+ds.orig.tar.xz
    dgit: split brain (separate dgit view) may be needed (--quilt=gbp).
    examining quilt state (multiple patches, gbp mode)
    dpkg-source: warning: cubeb-0.0~git20250529.78ee5f0+ds/debian/rules does not exist
    dpkg-source: warning: unknown information field 'Build-Driver' in input data in general section of control info file
    dgit (build host): base trees orig=25938cea9885d0da060b o+d/p=cf711e3c67958fc2bf9b
    dgit (build host): quilt differences: src:  ## orig ##     gitignores:  == orig ==
    dgit (build host): quilt differences:      HEAD ## o+d/p               HEAD == o+d/p
    dgit:  cannot represent change: not a plain file (000000->160000): cmake/sanitizers-cmake
    dgit:  cannot represent change: not a plain file (000000->160000): googletest
    dgit:  cannot represent change: not a plain file (000000->160000): src/cubeb-coreaudio-rs
    dgit:  cannot represent change: not a plain file (000000->160000): src/cubeb-pulse-rs

    dgit (build host): error: HEAD has changes to .orig[s] which are not representable by `3.0 (quilt)'

Ok, I just repacked the source removing the submodules with `git rm`,
but I found this error kinda odd, especially as I had been able to build
the package locally just fine.

Is this expected? How am I supposed to deal with git submodules?

Bye!

Note:
  I'm cloning this bug a couple of times for clearly defined
  action items.  Let us leave this original bug #1107219 for user
  support, and discussion of further options.

Andrea Pappacoda writes ("Bug#1107219: git-debpush: upload failing when source has submodules"):

I'm glad you're enjoying it :-).

Sadly, git submodules are a complete disaster.  I wrote a blog post
with a rant:  https://diziet.dreamwidth.org/14666.html

But, I think our tooling could have served you better.  Let me try to
give you some advice, and discuss possible improvement options.

Aha.

Yes, that is the workflow we would normally recommend.  It is more
faithful and more traceable to upstream.  Unfortunately the
faithfulness also exposes you to the submodules.

This is a reasonable thing for you to have attempted, even though it
was never going to work.

Our tooling ought to have detected this problem much sooner; and,
I think there are opportunities for making the workaround more
convenient.

tag2upload uses dgit to do a lot of the work, and this error is
basically #726953 "dgit fails with submodules".

I don't know your local building workflow, but many approaches for
local builds omit certain important consistency checks.

One of the key properties of our git transition tools, like tag2upload
and dgit, is that the source package (.dsc) is *completely identical*
to the (canonical) git view.  That makes it possible to replace .dsc
with git without changing what source code you get, to convert in both
directions, and so on.

For a git submodule it's not quite clear what "identical" means here,
but it probably means the submodule is populated.  This is discussed
at much greater length in another bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=726953#64

With our current tooling, I suggest treating the submodule the same
way we would treat nonfree materials.  Ie, this part of the workflow
manpage:

https://manpages.debian.org/bookworm/dgit/dgit-maint-debrebase.7.en.html#DFSG-non-free:_When_upstream_tags_releases_in_git

It's not quite clear to me what you mean by "repacked".  With
tag2upload you do not, in fact, ever need to have a terball on your
system.  So you don't need to run git deborig.

As for our tooling and docs:

 * git-debpush should definitely reject a submodule.
   That would have detected the problem locally.  (clone -2)

 * git-debrebase should detect upstream trees that have
   submodules and call that situation a snag.  (clone -3)

 * #726953 discusses more comprehensive submodule support
   for dgit (which, if implemented, would extend to tag2upload too).

 * Is there good tooling for doing DFSG-filtering entirely in git?
   Eg is there some git tooling that will read uscan config?

 * We need workflow manpages for tag2upload.  Separately we want more
   opinionated documentation of a git-first flow - that does not
   involve gbp import-orig.

 * I'm tempted to suggest an interim version of #726953 that makes it
   possible to simply filter out submodules during canonicalisation.

   That would improve convenience when upstream has submodules but the
   Debian package wants to ignore all of them completely.  How common
   do we think this situation is?

   (Note that it's easy to speak of doing this "as a quilt mode" or
   "in quilt fixup" but it's not a "3.0 (quilt)" thing - it might
   apply to native format source packages.)

Ian.

Hi Ian, thank you for the detailed reply.

Nice read :)

I haven't had a terrible experience when using submodules upstream, but
I agree that their implementation is not perfect, and using them can be
finicky.

I usually use `dgit sbuild`, but this time I felt particularly naughty
and used `sbuild` directly (because I also wanted to build with
uncommitted changes in a patches-unapplied project).

Makes sense.

Mh. I wouldn't mind if the canonical representation were just an empty
directory (after all, a plain git clone results in an empty directory,
at least when looking at the regular files creating by Git). But I guess
that the internal git representation is different.

Yeah, that's exactly what I did. I'm still used to tarballs, that's why
I used the "repacked" terminology; sorry for that!

As far as I know, no. Honouring d/copyright's Files-Excluded is the main
thing I miss from the regular tarball workflow.

+1 on both!

As mentioned above, I'd personally be OK with this solution. Submodules
are usually not used in Debian packaging (well, *I've* never used them).

Thanks again for your work! Bye :)

Hello,

Yes.

Yes.

If you're using a merging workflow then you can delete the offending
files once and then merging new upstream versions won't bring them back.
But otherwise, no, this is best done manually at present, and it's a
hassle.

We were just talking elsewhere about the problem of producing more and
more documentation :)

Unfortunately though I think you're probably right that we need another
workflow manpage to write down things like "treat submodules as a +ds
filter, i.e., like +dfsg".

But how about we try writing the opinionated documentation first?
I am slightly hopeful that would be enough and we wouldn't need a
tag2upload workflow manpage too.

I would assume that it is not that common, but I don't know, we would
need to ask people in packaging ecosystems that vendor a lot,
presumably.