I'd like to have "git import-orig" filter out all the files that are
listed in debian/clean, without having to keep the lists synchronized.

That way, if i notice that the upstream tarball is injecting some sort
of additional pre-built artifact into the tarball, i can keep it out of
our revision control *and* ensure that it gets rebuilt during a build
from source.

(This is motivated by discussion with Andreas Metzler about building
GnuPG documentation artwork artifacts, see
https://alioth-lists.debian.net/pipermail/pkg-gnupg-maint/2024-May/009340.html
, and of course also by the recent xz incident, where malware was
expressed the tarball that had not been committed to revision control)

background:  i prefer debian packaging linked to the upstream revision
control, but also being able to tie our work to formally released
tarballs, if the upstream project ships them.  I'm relying on gbp
import-orig with an appropriately configured debian/gbp.conf to import
cryptographically signed upstream tarball releases while pointing back
to upstream's revision control tags.

One example workflow that i would like to be able to have easily at my
disposal as a maintainer is to tell that things are in the tarball that
are *not* in the upstream revision control system (the other direction:
looking for things in upstream revision control that didn't get shipped
in the tarball, is interesting, but a separate question).

After having verified the cryptographic signatures of the upstream
tarball, and the upstream release tag, and doing "gbp import-orig", it's
nice to be able to do (for example):

    git diff --stat gnupg-2.2.43..upstream/2.2.43

This helps me identify artifacts that we should probably be re-building
from source.

By including these generated artifacts in debian/clean, i can ensure
that during the standard debian build process, they will be necessarily
re-generated (because we run "debian/rules clean" before building).  But
if i'm including them in debian/clean, then there's no point in keeping
them in the git packaging directory either.  and i would prefer to avoid
synchronizing debian/clean and debian/gbp.conf's import-orig.filter list.

Hi,

On Wed, May 15, 2024 at 06:58:22PM -0400, Daniel Kahn Gillmor wrote:
[..snip..]

Great! This matches my preferred way too.

Wouldn't d/copyright's `Files-Excluded:` work here too? I'm using that
for similar purposes as it even allows to use `gbp import-orig --uscan`
and have things filtered out. `debian/clean` could parse the pattern from
there.

I think what you propose is doing it the other way around: Have gbp
run `debian/rules clean` to have a programatical way of filtering?

Cheers,
 -- Guido

Hi Guido--

☺  Thanks for walking through the options here with me!

Hm, I don't know what the semantics are for Files-Excluded, or what
other side effects they have.  The documentation for the
machine-readable copyright format:

https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/

doesn't even include the word "Excluded", let alone "Files-Excluded"
(see #685506, sigh).  According to
https://wiki.debian.org/UscanEnhancements#Deleting_Files_using_Files-Excluded_field_in_debian.2Fcopyright
the Files-Excluded field actually affects the tarball by causing uscan
to re-pack it without those files.

Doing the tarball re-packing would mean breaking the upstream tarball's
cryptographic signature, so i'm not sure i want to do that.  The goal
here is to increase attributability and provenance, and breaking the
upstream cryptographic signature seems to work against that goal.

Hm, while i depend on gbp for my regular packaging workflow, one of the
things i like about it is how it wraps itself around other packaging
workflows.  If i remove debian/gbp.conf from my package's source, the
source can still build just fine using dpkg-buildpackage or debuild.
I'd like to keep that property.

right, that would do the job, and is probably the more principled way to
do it than merely parsing debian/clean.  It would work regardless of
whether the packaging used debhelper or not.  Does that seem like a
plausible way to operate gbp import-orig?

Hi dkg,

But you'd break that when filtering out files? I think what keeps me
confused: the tarball uploaded to Debian is the filtered one and hence
has a different checksum, no?

I understand that point.

It would also have the upside that packages invoking `dh_clean … path1
path2` would still work.

Another reason to not parse debian/clean verbatim is that we'd also need
to support dh's substitution variables and would forever need to follow
what dh does (and we might even need to pay attention to the dh compat
level of the package) as otherwise things would break on people.

That would be an approach. Implementation wise the "tricky" bit is
that you don't have debian/ on the upstream branch you want to filter so
dh_clean or `debian/rules clean` won't work as is . So we'd need to
overlay that (which is certainly doable, just wanted to point it out).

So that's a lot of effort for s.th. that can already be done via either
gbp.conf or FilesExcluded. I'm not against it, just looking at the pros
and cons.

Cheers,
 -- Guido

hm, i don't think so, because we use
import-orig.filter-pristine-tar=False.  This lets me preserve both the
upstream signature and the git history, and to compare the upstream
tarball with the git tag using git as well.  In the gnupg2 package, we
currently have this in debian/gbp.conf:

---------
[DEFAULT]
debian-branch = debian/unstable
upstream-branch = upstream-2.2
pristine-tar = True
upstream-vcs-tag = gnupg-%(version)s

[import-orig]
filter = [
 'aclocal.m4',
 'build-aux/compile',
 'build-aux/config.rpath',
 'build-aux/depcomp',
 'build-aux/install-sh',
 'build-aux/missing',
 'build-aux/mkinstalldirs',
 'build-aux/texinfo.tex',
 'ChangeLog',
 'config.h.in',
 'configure',
 'doc/gnupg.info*',
 'doc/*.pdf',
 'doc/*.png',
 'INSTALL',
 'm4/iconv.m4',
 'm4/intdiv0.m4',
 'm4/intl.m4',
 'm4/lock.m4',
 'm4/printf-posix.m4',
 'm4/size_max.m4',
 'm4/uintmax_t.m4',
 'm4/wint_t.m4',
 '*/*/Makefile.in',
 '*/Makefile.in',
 'Makefile.in',
 'po/*.gmo',
 'po/Makefile.in.in',
 'po/stamp-po',
 'regexp/_unicode_mapping.c',
 'regexp/UnicodeData.txt',
 'common/audit-events.h',
 'common/status-codes.h',
 'ChangeLog-2011',
 '*/ChangeLog-2011',
 'tests/*/ChangeLog-2011',
 ]
filter-pristine-tar = False
---------


So what i'm asking is to be able replace that big import-orig.filter
array with something that knows how we do cleaning, so that when i
improve our cleaning, it improves the filter for the next import as
well.

you've convinced me that running the clean target is better than trying
to parse debian/clean :)

ah, yes, i see the complication here.

right, i see tradeoff you're describing, and if you decide this is too
much complication for gbp, i'm willing to just keep the two lists
(debian/clean and debian/gbp.conf's import-orig.filter) in sync more or
less manually.

But i thought it wouldn't hurt to ask -- it'd certainly be nice for
anyone working on the GnuPG packaging (or any other packaging which
covers a similar upstream) to have a simpler packaging maintenance
workflow.

Thanks for thinking this all through with me here, Guido!

Hi,

Ah...now I get it. You filter the branch but let pristine-tar take care
of restoring the original tarball.

[..snip..]

Great! Let's try that path then.

I'm fine with the complication. Just wanted to point out possible
implementation pitfalls. Just to be sure: It's unlikely that I'll be
able to look at an implementation myself near term.

Thanks for explaining your workflow!
 -- Guido