Hi,

I'm in the process of converting my last packages still maintained in
svn to git :-) (nvidia-cuda-toolkit and nvidia-graphics-drivers, both in
non-free)

The upstream tarballs contain binary blobs, and cannot be maintained in git
(or with pristine-tar) due to their incredible size, so I'm going to use
overlays and keep only debian/* in git.
There is one tarball per architecture, so we have to deal with multiple
.orig-*.tar.gz components as well.

For some parts of the packaging work (e.g. working with patches) I need
to create a temporary tree including unpacked upstream tarballs
overlayed with the debian directory and copy back the modifications to
debian/, this was previously done with svn-do.

(I always build the packages in pbuilder environments, never on the host,
so there is always exporting used for building and I usually don't need
(or want) a (non-temporary) full source tree on the host)

The following command works as a start:

gbp buildpackage --git-export=WC --git-builder='env debian_chroot="gbp-do" ${SHELL}' --git-postbuild='cp -vuapf "${GBP_BUILD_DIR}/debian/"* "debian/"' --git-notify=off

the other relevant options come from debian/gbp.conf:
[buildpackage]
overlay = True
export-dir = ../build-area
create-orig = False
pristine-tar = False
compression = gz
tarball-dir = ../tarballs-nvidia-cuda-toolkit
component = [ 'ppc64el', ]


It would be great if such functionality could be added as a new
subcommand. Or do you have a better suggestion how to implement such a
workflow?


Here are the issues I've encountered so far:

* only the main .orig.tar.gz is unpacked, but the sub-components .orig-*.tar.gz are not


* since exporting does not preserve timestamps, always the full debian/
  tree will be copied back (unlike svn-do)


* in case of failure, the --git-builder command is used as a python
  format string:

...$ gbp buildpackage --git-export=WC --git-builder='env debian_chroot="gbp-do" ${SHELL}' --git-postbuild='cp -vuapf "${GBP_BUILD_DIR}/debian/"* "debian/"' --git-notify=off
gbp:info: Exporting 'WC' to '/path/to/package/build-area/package-tmp'
gbp:info: Moving '/path/to/package/build-area/package-tmp' to '/path/to/package/build-area/package-1.0'
(gbp-do)...$ exit 1
exit
Traceback (most recent call last):
  File "/usr/bin/gbp", line 151, in <module>
    sys.exit(supercommand())
  File "/usr/bin/gbp", line 147, in supercommand
    return module.main(args)
  File "/usr/lib/python3/dist-packages/gbp/scripts/buildpackage.py", line 550, in main
    )(dir=build_dir)
  File "/usr/lib/python3/dist-packages/gbp/command_wrappers.py", line 248, in __call__
    Command.__call__(self, list(*args))
  File "/usr/lib/python3/dist-packages/gbp/command_wrappers.py", line 192, in __call__
    self._log_err()
  File "/usr/lib/python3/dist-packages/gbp/command_wrappers.py", line 145, in _log_err
    log.err(self._format_err())
  File "/usr/lib/python3/dist-packages/gbp/command_wrappers.py", line 159, in _format_err
    err_reason=self.err_reason)
KeyError: 'SHELL'


* Is it possible to use an upstream-version subdirectory for the
  tarball-dir? E.g.
  tarball-dir = ../tarballs-nvidia-cuda-toolkit/%{upstream-version}


Andreas

control: clone -1 -2
control: retitile -2 "gbp buildpackge --git-overlay ignores additional tarballs"

Hi,

Let's use a new bug number for this one. Overlay mode used to be the
problem child in gbp since I'm not using it myself. I added a first test
for it in 2017 to not break it completely:

https://github.com/agx/git-buildpackage/blob/master/tests/component/deb/test_buildpackage.py#L179

The code really didn't handle additional tarballs. Fix forthcoming.
Cheers,
 -- Guido

control: clone -1 -2
control: retitle -2 Prevent expansion of '{}' in Command errors

Hi,

Sad but true. Thanks!
 -- Guido

Hi Andreas,
thanks for your report. You've found several issues, see below:

I'm not familiar with svn-do so please bear with me. Looking at:

https://manpages.debian.org/stretch/svn-buildpackage/svn-do.1.en.html

we could add a "gbp overlay-do". Let's use this bug number for sorting
this out.

This is true and I've cloned the bug to get a new number.

I'm not sure what you mean here. Preserve timestamps of what exactly the
exported debian dir? The copying back would be done by gbp-do? I you
think there's s.th. wrong already (even with a gbp overlay-do) please
clone this bug so we can work out the details there.

This is sad but true and I've cloned the bug to get a new number.

We're not doing any expansion on the tarball-dir. If this is a desireable
feature let's please move this into a separate wishlist bug.

Cheers,
 -- Guido

source code for svn-do is here:
https://anonscm.debian.org/viewvc/collab-maint/deb-maint/svn-buildpackage/trunk/contrib/svn-do?revision=23023&view=markup

You'll have at least one user that will test "gbp overlay-do" (although
I doubt that I'll use something else than the default "$SHELL" command)

I noticed that this is a "normal" issue (not related to my attempt to
get something svn-do alike) the other day when I tried to build
nvidia-cuda-toolkit on the ppc64el porterbox and half the tree was
missing, just didn't get around to factor this out as a separate issue.

I don't think git archive has the opportunity to "preserve" timestamps
from the WC (seems to be one timestamp for all files, taken from newest
file (or commit), didn't work out the details) ... but maybe that's
something that can be "fixed" while copying back.
I got the "cp -vuapf X Y" from svn-do, maybe something else can be used
to only copy back files that actually changed content (not just
timestamp). And deleting disappeared files could be considered as well.

What I was thinking about here was that "cp -vuapf X Y" lists all files
as copied back (or probably excluding the one that contributed the
newest timestamp), making that output just noise, so the -v could be
dropped as well. (With svn-do I usually got a list of files that were
actually changed, plus some symlinks)

wasn't sure if I had caused this by some "unsupported" type of command ...

Not sure if I want to pursue this ... depends on how I'm going to handle
the storage backend of the tarball-dir (for now it will stay on svn, and
per-upstream-version subdirectories would have allowed to do more sparse
svn checkouts, but I'd need to implement something in the package for that).
But since we can reproducibly build the upstream tarballs, we don't even
need to go through a central repository any more.
(btw, the last upstream releases of nvidia-cuda-toolkit were about
2.5 GB of blobs per release)


Andreas

control: tag -1 +help
control: severity -1 wishlist

Hi,
something out that works for you and keep the bug report up to date with
that? If something is missing in existing tools let me know and once we
have something working we can look into merging it in. Either in
examples/ or as an "official" command.

Cheers,
 -- Guido