Dear Maintainer, dear Guido,

At Kali Linux we use gbp to maintain our packages. We have around 500
packages total, among which 40 are forked from Debian. Kali Linux being
a rolling distribution, those 40 packages are kept in sync with their
counterpart in Debian testing, meaning that we update them often.

We use the same workflow for pretty much all of our packages: 3 git
branches (packaging, upstream and pristine-tar), and no upstream git
history. We use "gbp import-orig" for all the Kali specific packages,
and "gbp import-dsc" for all the Debian forks.

All in all, gbp works great for us, but nothing is never perfect :)


    THE PROBLEM

For our Debian forks, we actually have 4 branches (or 2 when it's a
native package): we have the packaging branch (named "kali/master"), the
usual upstream and pristine-tar branches, but we also have the "upstream
distro" branch (named "debian") that contains the unmodified source from
Debian, as imported by "gbp import-dsc".

Just to clarify, let me describe how we maintain such a 4-branches git
repo:

* Step 1.  $ gbp import-dsc --debian-branch=debian
  -> ie. update the 3 branches upstream, pristine-tar and debian.

* Step 2.  $ git merge debian
  -> ie. merge the debian branch into the kali/master branch. This is
     when we resolve our merge conflicts, since the Kali changes live
     on the kali/master branch.

(Note: this workflow was mentioned in https://bugs.debian.org/670612)

This workflow works very well, and keeping our Debian forks in sync with
Debian is fairly straightforward.

Our issue is that the 4th branch (the "debian" branch) lives out of the
realm of gbp. Therefore it's ignored by the gbp commands that deal with
git branches, and we always need additional steps to handle this branch.
In particular:

* gbp clone: Won't setup this branch, one has to do it manually:

    gbp clone
    cd <pkg>
    git branch debian origin/debian

* gbp pull: Won't pull it, one has to do it manually:

    gbp pull
    git checkout debian
    git merge
    git checkout -

  (or: pull all branches)

    gbp pull --all

* gbp push: Won't push it, one has to do it manually:

    gbp push
    git checkout debian
    git push
    git checkout -

  (or: push with git)

    git push origin : --follow-tags

  (or: if the --tips option in #908204 was implemented, we could do:)

    gbp push --tips

These additional steps are trivial, but they are also easy to forget.
One thing that keeps biting me is that I often forget to pull the
"debian" branch. So I work on the package, update it, make it ready for
upload, and at the last moment, when I want to push all the git
branches, git refuses, because I worked with an outdated branch and my
modifications are not fast-forwardable. So I have to restart from zero,
or force push, or whatever.


    THE TENTATIVE IMPROVEMENTS

So I've been thinking hard about ways to prevent that to happen, and
more generally, about how I could get gbp to clone/pull/push this 4th
branch automatically, so that there's no need for additional steps.

Here are some ideas, starting with the worst one:

  1) the "upstream distro" concept

An ambitious and complicated way could be to teach gbp about the concept
of an "upstream distro" (meaningful only for Debian derivatives), and
define in the gbp.conf what is the "upstream-distro-branch", the
"upstream-distro-tag", and more if needed. And then let gbp do the right
thing (TM). In other words, hammer a new concept into gbp.

IMO: way too complicated.

  2) the "extra branche(s)" idea

Instead, a lighter approach could be to teach gbp about "extra
branches", without defining what those branches are for. "gbp clone"
would setup those branches, and "gbp pull" would pull it. That's super
easy to implement. The complicated part is that "gbp push", as far as I
can see, would not know what to do with this branch, due to the way it
works. And therefore it wouldn't push it. Unless the option "--tips"
mentioned in #908204 is implemented, and in such case we could document
that gbp will push the extra branches only when --tips is enabled.

It's not super elegant (and maybe not super useful) to introduce an
option for extra branches, and then say that it's only supported by
clone and pull, but not supported by push. So I'm not super convinced
myself...

  3) more hooks

I've noticed that some gbp actions have a "--postxxx" argument, to allow
users to run hooks. That could help!

There's already a "--postclone" option. I can add this snippet in
debian/gbp.conf, and it works:

    [clone]
    postclone = git branch debian origin/debian

If I had a "--postpull" option, I could make sure that the "debian"
branch is automatically pulled. This is something I'd configure in the
debian/gbp.conf for all of Kali's Debian forks.

Same with a "--postpush" option.

And while we're at it, a "--postimport" for the "gbp import-dsc" command
would also be welcome, so that I can display a friendly message such as
"please merge the debian branch into kali/master, resolve conflicts and
commit".


    THE CONCLUSION

Is there any of the ideas above that make sense for gbp? Extra branches,
or more hooks? Is there any other options that I didn't think about? If
you think something could be done in gbp, I'd be happy to implement it.

Thanks for reading,

  Arnaud

Hi Arnaud,
more hooks could be a mitigation of the situation having support for
additional branches is useful too (that could also help some workflows
that maintain the dfsg clean and non-dfsg clean sources in the same
repo).

I miss one point in your enumeration: syncing with changes from Debian's
salsa which should be part of the picture as well - because in that
situation you want the gbp.conf that Debian ships and merge the result
into the derivative's branch.

So given that I wonder if having the concept of a "downstream" or
"derivative" within gbp wouldn't be worthwhile to look at? I know there
are more people using gbp for derivative work (including me) so making
this easier (without regressing in the "pure" Debian case would make
sense to me. This could also include more explicit tracking or a
remote for the derivative and salsa.

Cheers,
 -- Guido

We don't really use Salsa. For most of our Debian's forks, we just have
minor changes to maintain, so we don't need to know about Debian's git
history. So we just use import-dsc to import new versions from Debian.

The only exception I know of it the debian-installer. For this package
we do fetch changes from Salsa, as it's sometimes convenient to package
a git snapshot. In that case, we just do plain git commands, eg. "git
fetch salsa && git merge salsa/master".

Does that answer your question? I'm not sure I understood exactly what
you asked.

The concept of "downstream" would make sense indeed. I can do a bit of
work on my side to see how it would look like, in order to be usable by
Kali. Then I'll come back here to report what I found. No ETA though,
this is something that I'll do in the background when times allow ;)


Cheers,