Hello,

there was some discussion on debian-devel@l.d.o about enabling uscan to
remove files from an upstream tarball automatically.  In this discussion
(first mentioning was here:

http://lists.debian.org/debian-devel/2012/08/msg00406.html

) the idea occured that it makes perfectly sense to mention files which
are excluded from the tarball inside the debian/copyright file.  From my
perspective this makes sense in any case (independently whether uscan
does any automatic job here or not.)  Please note that this problem is
somehow connected to bug #561494.

I would propose the following addition to

http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
----------------------- 8< ---------------------------------------------
Files-Excluded
--------------

Whitespace-separated list: list of patterns indicating files removed
from upstream source.

# begin copy of Files field description

Filename patterns in the Files field are specified using a simplified
shell glob syntax. Patterns are separated by whitespace.

Only the wildcards * and ? apply; the former matches any number of
characters (including none), the latter a single character. Both match
slashs (/) and leading dots, unlike shell globs. The pattern *.in
therefore matches any file whose name ends in .in anywhere in the source
tree, not just at the top level.

Patterns match pathnames that start at the root of the source tree.
Thus, "Makefile.in" matches only the file at the root of the tree, but
"*/Makefile.in" matches at any depth.

The backslash (\) is used to remove the magic from the next character;
see table below.

Escape sequence	Matches
\*	star (asterisk)
\?	question mark
\\	backslashAny other character following a backslash is an error.

This is the same pattern syntax as fnmatch(3) without the FNM_PATHNAME
flag, or the argument to the -path test of the GNU find command, except
that [] wildcards are not recognized.

# end copy of Files field description

The field is optional and should be specified in connection with the
Source field.
----------------------- >8 ---------------------------------------------

Kind regards

        Andreas.

user debian-policy@packages.debian.org
usertags 685506 normative discussion
retitle 685506 copyright-format: new Files-Excluded field
thanks

Le Tue, Aug 21, 2012 at 02:34:50PM +0200, Andreas Tille a écrit :

Hi Andreas,

thank you for your work on this.  I think that it is a good idea to open a bug
as a point for tracking further discussion.

At this moment, I would recommend to look how the situation evolves in the next
months, in particular:

 - How the idea is adopted,
 - if un-anticipated shortcomings are found, and
 - if other tools (git-import-orig for instance) implement the parsing if this
   new field.

Then, once the usage of this field is established, let's re-open the discussion
of the integration in the next revision of the format.

Have a nice day,

Thanks for enhancing the metadata.

I personally will migrate any of my get-orig-source scripts to use
exclusion enabled removal uscan from the moment where this might become
possible.

While the idea for this new field was born from the intended usage in
uscan I would like to stress the fact that I see a perfectly valid
reason to use this totally independently as a documentation means.  So
this bug is somehow unrelated to any usage in tools (even if admittedly
it perfectly would help adopting it).  Please see the suggestion as any
other field in debian/copyright which is not necessarily used by any
tool.

Kind regards

      Andreas.

[..]

Wouldn't it be easier to say "Same as ``Files'', except that it may
only occur once."?

This probably belongs into the "Header paragraph (once)" section as
an additional bullet point "Files-Excluded: optional".


Cheers,
gregor

Yes.
My motivation to create a copy was that changes might be done more
easy.  If there will be no change between what I marked as copy I
would fully agree.

Yes.  (I was not aware of this header-body structure so I just missed
this way better description.)

Thanks for the enhancements

       Andreas.

Hi,

FWIW, prior to DEP-5 adoption, I actually also made a proposal akin to
this:
http://lists.debian.org/debian-devel/2009/12/msg00046.html
which, at the time, was rejected on the ground that documented removed
files is not explicitly mandated by Policy.

I actually used it in some of my packages :

Removed-Files: yorick/lbfgs*
Rationale: Not DFSG
 Those files are not used in the plug-in. They bear no clear copyright
 statement.

Removed-Files: idl/*
Rationale: Not DFSG
 The idl/ directory of the original source contains files meant for
 building a plugin for a commercial product. These files are not used
 in the Debian package and some have a dubious copyright statement,
 presumably non-DFSG.

(Its possible that some of it remains but I believe I removed it when
switching to the adopted standard with routine uploads. See:
http://packages.debian.org/changelogs/pool/main/y/yorick-optimpack/yorick-optimpack_1.3.1+dfsg1-1/yorick-optimpack.copyright)

I think having a "Rationale" is good. It helps the (next?) maintainer
decide whether the restrictions still apply to the new upstream release.

To me, Removed-Files sounds clearer: those files have been removed
from the package, not simply excluded from... perhaps the copyright
notice above? Files-Excluded could be interpreted as "the following
applies to all except those files". No strong feelings about this
though, you may paint the bike shed whatever color you see fit ;-)

Hope this helps.

Kind regards, Thibaut.

I don't agree: I find it better to allow several such paragraphs, when
the rationale behind removing several files differs, as in this
real-life example:

Removed-Files: yorick/lbfgs*
Rationale: Not DFSG
 Those files are not used in the plug-in. They bear no clear copyright
 statement.

Removed-Files: idl/*
Rationale: Not DFSG
 The idl/ directory of the original source contains files meant for
 building a plugin for a commercial product. These files are not used
 in the Debian package and some have a dubious copyright statement,
 presumably non-DFSG.

http://packages.debian.org/changelogs/pool/main/y/yorick-optimpack/yorick-optimpack_1.3.1+dfsg1-1/yorick-optimpack.copyright

Of course this goes together with the idea of documenting the
rationale behind removing a file.

As exemplified, the Rationale would have a short and a long
escription. The short description should be standardized with a few
typical values:
 Not DFSG
 Big but not used
 other
purely for statistical purposes. Not sure whether that's useful.

Kind regards, Thibaut.

* Thibaut Paumard <thibaut@debian.org>, 2013-01-16, 14:23:

What would "Big but not used" be doing in a _copyright_ file?!

I hope this makes everybody realise that debian/copyright is a very bad
place for repack instructions.

Ok; that would mean defining a new paragraph type for
Copyright-Format 1.x; adding a field to the Header paragraph can be
tried "for free" before changing the spec.
(Just saying, not that this means we can't do it.)

Currently CF1.0 also says for the Source Field in the Header
paragraph:
"If the upstream source has been modified to remove non-free parts,
that should be explained in this field."
so just using this seems easier.

I guess it boils down to how much detail we need (your "Rationale");
while I understand your point, so far I was quite happy with writing
one free-form line in a (Header) Comment field and be done with it.

Cheers,
gregor

Hi,
the option of:

Files-Excluded:
    file_a
    file_b
    file_c
    ...
Comment: The following files were removed because
  file_a: Binary with no source
  file_b: non-free license
  file_c: just cruft
  ...

This is not that nice but should work.

Kind regards

      Andreas.

Le Tue, Dec 03, 2013 at 07:19:23PM +0100, Andreas Tille a écrit :

Hi Andreas,

I think that the introduction of new fields would be under the scope of a
version update of the copyright format.  The Policy bug for adding
Files-Excluded is actually there already.

http://bugs.debian.org/685506

Ideally, such a version update would come with a systematic review of all
the other bugs related having "copyright-format" in their subject, and a
survey of the other extra fields currently in use would be neat as well.

However, if this does not happen because nobody has time to do the work, I
personally would be fine with a 1.1 update that only contains the
Files-Excluded addition.  This field was not my favorite solution to the
problem, but experience showed that your solution was implemented, used,
and unchallenged.

Have a nice day, and thanks for your perseverance.

Hi Charles,

Ahhh, it was soooo long ago that I even forgot this bug.  Next time I'll
reread the documentation I have written in Wiki myself better.  Sorry for
the noise.

The compensation will be a saving of work when there is a need for removing
files from upstream. :-)

Kind regards

       Andreas.

Le Tue, Dec 10, 2013 at 05:35:08PM +1100, Ben Finney a écrit :

Hi Ben,

http://bugs.debian.org/685506 tracks the proposal of adding a Files-Excluded
in the next version of the specification.

Your comment implies that the definition of the Source field should be changed
together with the addition of Files-Excluded, and I think that it is totally
doable.

People who like the information to be in debian/copyright worked on an
implementation that is used and now supported in devscripts.  In contrary,
people who like the information to be somewhere else, however good are their
reasons, did not produce a viable alternative.  Unless there is a concrete
commitment for creating a robust and well-accepted alternative, I think that
there is no point discussing the issue further.

Have a nice day,

Hi Joachim,

Good.

Hmmmm, I can not confirm.  Did you tried the mothur package?

Cool!

I have changed this at

https://wiki.debian.org/UscanEnhancements

(please double check and change if needed) and I also keep the bug report
in CC.

Many thanks for your work on this

     Andreas.

Hi,

Am Freitag, den 21.03.2014, 14:17 +0100 schrieb Andreas Tille:

No, not until now. My code happened to add a "./" in front of every
path, but that package excludes ".*", so all files were removed. Fixed
and covered by the test suite.

Greetings,
Joachim

Hi Joachim,

Hmmmmm, I guess something remains wrong.  While uscan says:

Newest version on remote site is 1.33.3, local version is 1.33.0+dfsg
 (mangled local version number 1.33.0)
 => Forcing download as requested
-- Downloading updated package Mothur.1.33.3.zip
-- Repacking from zip to .tar.xz
-- Successfully downloaded updated package Mothur.1.33.3.tar.xz
    and removed 134 files from it in mothur_1.33.3+dfsg.orig.tar.xz
-- Scan finished


the files that need to be removed are remaining inside the package.
(all *.o files are there, __MACOSX dir remained).

Kind regards

     Andreas.

Hi,

Am Freitag, den 21.03.2014, 22:19 +0100 schrieb Andreas Tille:

Did you check Mothur.1.33.3.tar.xz or mothur_1.33.3+dfsg.orig.tar.xz – a
mistake I also did earlier today:

$ apt-get source mothur
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
HINWEIS: »mothur«-Paketierung wird betreut im »Svn«-Versionsverwaltungssystem auf:
svn://anonscm.debian.org/debian-med/trunk/packages/mothur/trunk/
Es müssen 918 kB an Quellarchiven heruntergeladen werden.
Holen: 1 http://http.debian.net/debian/ sid/main mothur 1.33.0+dfsg-1 (dsc) [2.103 B]
Holen: 2 http://http.debian.net/debian/ sid/main mothur 1.33.0+dfsg-1 (tar) [902 kB]
Holen: 3 http://http.debian.net/debian/ sid/main mothur 1.33.0+dfsg-1 (diff) [13,9 kB]
Es wurden 918 kB in 0 s geholt (1.095 kB/s).
dpkg-source: Information: mothur wird nach mothur-1.33.0+dfsg extrahiert
dpkg-source: Information: mothur_1.33.0+dfsg.orig.tar.xz wird entpackt
dpkg-source: Information: mothur_1.33.0+dfsg-1.debian.tar.xz wird entpackt
dpkg-source: Information: makefile.patch wird angewandt
dpkg-source: Information: no_lcurses.patch wird angewandt
dpkg-source: Information: drop_sse_option.patch wird angewandt
dpkg-source: Information: catchall_fix.patch wird angewandt
dpkg-source: Information: build_without_tty.patch wird angewandt
dpkg-source: Information: uchime_link_dynamically wird angewandt
dpkg-source: Information: uchime_hardening.patch wird angewandt
dpkg-source: Information: spelling.patch wird angewandt
/tmp $ cd mothur-1.33.0+dfsg/
/tmp/mothur-1.33.0+dfsg $ ~/build/devscripts/devscripts/scripts/uscan.pl --verbose --force-download --repack --repack-compression xz
-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
   opts=dversionmangle=s/(\+dfsg|[~\+]repack)// http://www.mothur.org/wiki/Download_mothur    /w/images/.*/.*/Mothur\.([\d\.]+)\.zip
-- Found the following matching hrefs:
     /w/images/6/65/Mothur.1.33.3.zip (1.33.3)
     /w/images/9/99/Mothur.1.32.1.zip (1.32.1)
     /w/images/b/bc/Mothur.1.31.2.zip (1.31.2)
     /w/images/d/d3/Mothur.1.30.2.zip (1.30.2)
     /w/images/a/a5/Mothur.1.29.2.zip (1.29.2)
     /w/images/6/68/Mothur.1.28.0.zip (1.28.0)
     /w/images/c/cb/Mothur.1.27.0.zip (1.27.0)
     /w/images/2/20/Mothur.1.26.0.zip (1.26.0)
     /w/images/7/7b/Mothur.1.25.1.zip (1.25.1)
     /w/images/0/09/Mothur.1.24.1.zip (1.24.1)
     /w/images/3/33/Mothur.1.23.1.zip (1.23.1)
     /w/images/4/4b/Mothur.1.22.2.zip (1.22.2)
     /w/images/6/64/Mothur.1.21.1.zip (1.21.1)
     /w/images/4/42/Mothur.1.20.3.zip (1.20.3)
     /w/images/0/08/Mothur.1.19.4.zip (1.19.4)
     /w/images/9/96/Mothur.1.18.1.zip (1.18.1)
     /w/images/9/9b/Mothur.1.17.3.zip (1.17.3)
     /w/images/8/8c/Mothur.1.16.1.zip (1.16.1)
     /w/images/a/a5/Mothur.1.15.0.zip (1.15.0)
     /w/images/2/2d/Mothur.1.14.0.zip (1.14.0)
     /w/images/0/05/Mothur.1.13.0.zip (1.13.0)
     /w/images/4/4d/Mothur.1.12.3.zip (1.12.3)
     /w/images/b/b9/Mothur.1.11.0.zip (1.11.0)
     /w/images/6/64/Mothur.1.10.0.zip (1.10.0)
     /w/images/e/e5/Mothur.1.9.0.zip (1.9.0)
     /w/images/e/ea/Mothur.1.8.0.zip (1.8.0)
     /w/images/e/e8/Mothur.1.6.0.zip (1.6.0)
     /w/images/7/7a/Mothur.1.4.2.zip (1.4.2)
     /w/images/f/f3/Mothur.1.3.0.zip (1.3.0)
     /w/images/a/a9/Mothur.1.2.0.zip (1.2.0)
     /w/images/3/33/Mothur.1.1.0.zip (1.1.0)
Newest version on remote site is 1.33.3, local version is 1.33.0+dfsg
 (mangled local version number 1.33.0)
 => Forcing download as requested
-- Downloading updated package Mothur.1.33.3.zip
-- Repacking from zip to .tar.xz
-- Successfully downloaded updated package Mothur.1.33.3.tar.xz
    and removed 156 files from it in mothur_1.33.3+dfsg.orig.tar.xz
-- Scan finished
$ tar taf ../Mothur.1.33.3.tar.xz|wc -l
961
$ tar taf ../mothur_1.33.3+dfsg.orig.tar.xz|wc -l
805

Gruß,
Joachim

Hi,

see below ...

Hmmm, that's strange.  At my side the orig.tar.xz is a symlink to the
other file.  I'm using

    debian/rules get-orig-source

(which does the same as you above).  BTW, I personally see no point in
keeping a non-stripped archive - it is perfectly fine to have only the
result of the stripped download.

No idea how I can help you reproducing the behaviour at my side.

Kind regards

         Andreas.

Hi,

Am Freitag, den 21.03.2014, 23:24 +0100 schrieb Andreas Tille:

In debian/rules I see no call to “repack”, that might be the reason; but
I won’t look into the details (what happens with which flags, and why)
today.

Greetings,
Joachim

Hi Joachim,

$ svn diff
Index: rules
===================================================================
--- rules       (Revision 16448)
+++ rules       (Arbeitskopie)
@@ -36,4 +36,4 @@
 #         Alternatively you can use
 #          . debian/get-orig-source
 get-orig-source:
-       uscan --verbose --force-download --repack-compression xz
+       uscan --verbose --force-download --repack --repack-compression xz


This also became necessary with the current devscripts implementation.
My original code realised that since some files were removed a
repackaging is simply *required* and thus this option was redundant.

So, we actually are using the same command with different results which
is really strange.

Kind regards

       Andreas.

Hi,

Am Freitag, den 21.03.2014, 23:31 +0100 schrieb Andreas Tille:

so starting in an empty directory (maybe that makes a difference), if
you do

$ apt-get source mothur
$ cd mothur-1.33.0+dfsg/
$ path-to-current/devscripts/scripts/uscan.pl --force-download --repack --repack-compression xz
$ tar taf ../Mothur.1.33.3.tar.xz|wc -l
$ tar taf ../mothur_1.33.3+dfsg.orig.tar.xz|wc -l

what do you get?

Greetings,
Joachim

Hi,

forget about this - I was using the wrong uscan.  Sorry for the noise.

BTW, I do not see any sense in having the original *.zip, a xz
compressed tar.xz and the stripped +dfsg.orig.tar.xz.  IMHO the latter
is fully sufficient and the intermediate result (.tar.xz) could go away.

Kind regards

      Andreas.

In-Reply-To=<20140321105301.GE18098@an3as.eu>

Package: debian-policy
Followup-For: Bug #685506

Joachim Breitner wrote:

Andreas Tille wrote:
"foo/bar.js */foo/bar.js".

Imagine an upstream providing two implementations, a default non free
"imp.c" and a free alternative "gpl/imp.c". The maintainer cannot
remove the former while keeping the latter anymore.

I call this less flexible, but I may miss your point.


The problem that you are trying to solve also affects license
attribution.
In your example of an upstream accidentally including a ".mytags" file
next to "pkg-1.0/" in the tarball, then specific Files patterns will
probably fail to match the new paths, so the whole package will be
considered as covered by the default license (Files: *).
If a work-around is ever adopted, it should be a dep5 update, not
create a divergence specific to Files-Excluded.


The Files and Files-Excluded fields share the same pattern syntax.
The new schem implies that:
- "a" matches "foo/a" in Files-Excluded, not in Files
Such an inconsistency create confusion, and will probably delay
acceptance of Files-Excluded into dep5.

Moreover, I consider that existing inconsistencies like
- "[ab]" matches "a" in Files-Excluded, not in Files
- "{a,b}" matches "a" in Files-Excluded, not in Files
- "a/" matches subdirectory "a" in Files-Excluded, not in Files
should be removed.

Imagine for example that upstream relicenses files under the "subdir"
directory. The maintainer will probably replace
  Files-Excluded: subdir/

  Files: *
  License: GPL-3
with
  Files: *
  License: GPL-3

  Files: subdir/
  License: GPL-2
This is valid, but last stanza is silently ignored.
I think that forcing users to write
  Files-Excluded: subdir*
or the more accurate "subdir subdir/*" from the beginning avoids
problems in the long term.


Thanks for making Files-Excluded a reality!
Please CC me as I am not subscribed to the list.

Hi,

the mothur issue is settled but I think there is a new problem with
ordinary *.tgz files now:  Please try:

  apt-get source gnumed-client
  cd <unpackdir>
  debian/rules get-orig-source

I get:

$ debian/rules get-orig-source
uscan --verbose --force-download --repack-compression xz
-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
   opts="dversionmangle=s/\+dfsg//g" http://www.gnumed.de/downloads/gnumed-versions.txt    http://www.gnumed.de/downloads/client/[\d\.]+/gnumed-client\.([\d\.]+)\.tgz
-- Found the following matching hrefs:
http://www.gnumed.de/downloads/client/1.4/gnumed-client.1.4.7.tgz (1.4.7)
Newest version on remote site is 1.4.7, local version is 1.4.6+dfsg
 (mangled local version number 1.4.6)
 => Forcing download as requested
-- Downloading updated package gnumed-client.1.4.7.tgz
Cannot determine compression method of gnumed-client.1.4.7.tgz at /home/andreas/bin/uscan line 1969.
make: *** [get-orig-source] Fehler 255


I also got the message

  Cannot determine compression method of ... at /home/andreas/bin/uscan line 1969.

for an ordinary *.tgz without any attempt to unpack (for gnumed-server actually).

Any clue what might went wrong?

Kind regards

       Andreas.

Agreed.  We shouldn't be introducing divergence between how Files: and
Files-Excluded: are interpreted.

Cheers,

Hi,

I realised that the latest devscripts upload now also includes the
--compression option.  Thanks for all your work on this.  While I think
that even the previous version fixed #685506 I wonder whether you kept
this bug open intentionally.  From my point of view it can be closed.

I have adapted the wiki page about uscan enhancements[1] since all major
points seem to be solved now.  The only remaining item was to enable a
trigger for "tar --exclude-vcs" which now should probably go to
mk-origtargz.  I just think that it is sensible to have an option to
drop all the VCS stuff inside upstream tarballs.

I'm going to file a bug report about this if you will not beat me with
implementing this in Git first.  I think the implementation is pretty
easy and any patch of mine will just pester you with my naming choice
which has not proven the best one in the past and thus I would hesitate
to send a patch.

Kind regards and thanks for all your work on this

      Andreas.

[1] https://wiki.debian.org/UscanEnhancements

Hi,

Am Montag, den 12.05.2014, 09:59 +0200 schrieb Andreas Tille:

I agree, that makes two of us, closing.

Again, I’m opposed to silently do stuff with user’s data that he did not
explicitly asked for. What if upstream decides that he wants the git
repo in the tarball (after all, it is a distributed VCS, so why not
distribute it) and the build process uses it to calculate, say, the
version number and a change log at built time.

I don’t mind an option "--exclude-vcs" to mk-origtargz (and uscan) that
works like "--exclude-file .git --exclude-file _darcs" ...

It’s not as trivial as you think it might be. Currently, mk-origtarz
uses "tar --delete" and never actually unpacks the tarfile. But
--exclude-vcs does not work with --delete:

$ tar tf foo.tar
foo/
foo/.git/
foo/.git/a
foo/b
$ tar --delete --file foo.tar --exclude-vcs
$ tar tf foo.tar
foo/
foo/.git/
foo/.git/a
foo/b

It would probably work easiest by simply listing the various glob
patterns in mk-origtargz directly, like in the "origtarz" script, or use
@Dpkg::Source::Package::tar_ignore_default_pattern (from libdpkg-perl).

Do you also want options --exclude-backups, --exclude-caches,
--exclude-caches-all, --exclude-caches-under, .... :-)

Greetings,
Joachim

Control: reopen -1

I think that you closed the wrong bug: this one is about documenting the
Files-Excluded field in the specification of machine-readable debian/copyright
files, which is probably the next thing to do now that it has been implemented
in devscripts.

Have a nice day,

Ahhh, thanks for watching us.  I added a line to

https://wiki.debian.org/UscanEnhancements

explaining the nature of this bug.

Kind regards

      Andreas.

Dear Customer,

Your item has arrived at January 25, but our courier was not able to deliver the parcel.

You can find more details in this e-mail attachment!

Kind regards,
Raul Oneill,
USPS Delivery Clerk.

On 2012-08-21 14:34:50 +0200 Andreas Tille <tille@debian.org> wrote:> I
would propose the following addition to
It appears as if Files-Excluded in d/changelog has become a de facto
standard in the meantime, so I'd like to ping the Policy Team and ask
whether/when we can expect an official blessing of this standard :-)

I've been updating my crufty packages and lintian has repeatedly
reported debian-rules-contains-unnecessary-get-orig-source-target. Most
of my targets were indeed simple calls to uscan, but I used to maintain
a dedicated d/files-excluded file (which I'd pass on with the
--copyright-file option) just for the Files-Excluded data.

I moved everything to d/copyright now, but having this sanctioned
officially would be welcomed by us pedants.

Regards,
Christian

Hello,

Contrary to an older e-mail in this bug, the consensus among the Policy
Editors is that we can add new, optional fields to the copyright-format
spec without bumping its version number.  This is because the addition
of optional fields is backwards compatible.

I have not read the whole thread, but a quick scan suggests that all we
are waiting for here is for someone to write a patch against current
Policy's master branch.

Hello,

I really miss the mention of Files-Excluded in CF1.0 and when searching
found this bug.

Is this truly the case that all that's needed is a new patch? Can we get
an official ACK from one of the policy editors? I'd be happy to re-write
the original patch to apply against HEAD if that's all that is required.

Hello,

Well, it would need seconding, but otherwise, ACK.

Thank you for your interest.

Hi,

Sean Whitton <spwhitton@spwhitton.name> writes:

Gentle ping to Policy editors for that seconding :-)  It would be really
nice to move this info from tribal knowledge to documentation.

Best,
Nicholas

Hello,

There's no patch to be seconded ...?

For whoever is going to write it, remember to talk about:
 * Files-Excluded-$component
 * Files-Included
which are even less-known bits of the "specification" (Files-Included is
also *very* new, in Debian times).

I need to admit that I'm not up for such a patch despite the fact
that I was involved into Files-Excluded invention.  I'm lagging
way too much behind my usual Debian work (and this is for a reason
which I announced some time ago in debian-private.

Ahhh, nice to learn about this.  I never heard about it before.

Kind regards

     Andreas.

Hello again,

Well, it took me longer than I anticipated to get around to this; nevertheless,
below is my first draft at a patch to document Files-Excluded. It's my first
time writing for Debian Policy, so please let me know if I didn't use an
appropriate convention somewhere.

---
 copyright-format-1.0.xml | 52 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

diff --git a/copyright-format-1.0.xml b/copyright-format-1.0.xml
index b8df359..102b7fa 100644
--- a/copyright-format-1.0.xml
+++ b/copyright-format-1.0.xml
@@ -246,6 +246,11 @@
             <link linkend="copyright-field">Copyright</link>: optional.
           </para>
         </listitem>
+        <listitem>
+          <para>
+            <link linkend="files-excluded-field">Files-Excluded</link>: optional.
+          </para>
+        </listitem>
       </itemizedlist>
       <para>
         The <varname>Copyright</varname> and <varname>License</varname>
@@ -672,6 +677,53 @@ Copyright 2009, 2010 Angela Watts</programlisting>
       </para>
     </section>

+    <section id="files-excluded-field">
+      <title><varname>Files-Excluded</varname></title>
+      <para>
+        Whitespace-separated list: list of patterns indicating files that have
+        been removed from the original upstream source distribution. The
+        syntax for the patterns describing these files is identical to that
+        of the <varname><link linkend="files-field">Files</link></varname>
+        field.
+      </para>
+      <para>
+        Often upstream developers include additional files in their source
+        distribution files that are unwanted by Debian and/or unneeded for
+        use in building a Debian package, such as:
+        <itemizedlist>
+          <listitem>
+            <para>
+              Source files that do not meet the DFSG;
+            </para>
+          </listitem>
+          <listitem>
+            <para>
+              Pre-compiled executable binary or other non human-editable files;
+            </para>
+          </listitem>
+          <listitem>
+            <para>
+              Files intended for use with other platforms.
+            </para>
+          </listitem>
+        </itemizedlist>
+      </para>
+      <para>
+        These types of files, or any others that Debian does not want to
+        include in our archive, must be stripped from the upstream tarball
+        prior to uploading. The <varname>Files-Excluded</varname> field serves
+        to document the removal of these files from the original upstream
+        source. This allows others to understand or audit how the source
+        distribution in Debian is derived from the upstream source.
+      </para>
+      <para>
+        Additionally, once documented in this manner, various tools such as
+        <literal>uscan</literal> or <literal>mk-origtargz</literal> can use
+        this information as instructions on how to automatically repack an
+        upstream source distribution into one suitable for use within Debian.
+      </para>
+    </section>
+
   </section>
   <section id="license-specification">
     <title>License specification</title>

Am Sat, Nov 13, 2021 at 11:21:26PM -0500 schrieb Joe Nahmias:

I like this patch!

Kind regards

      Andreas.

Hi,

For your information:

In https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000771
is help requested from people of pseudo-package www.debian.org


Groeten
Geert Stappers

May I add, that Joe's patch is lacking any mention of Files-Included,
as recommended by Mattia in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=685506#187
?


Holger

Hello Joe,

Thanks for the patch.  Here's a review:

The reason we remove these is also DFSG.
on other platforms.  It's typically only done if the files are huge.  So
please remove this one from the list.

How about just saying: we always remove non-DFSG files if the package is
intended for main or contrib, and sometimes there are other files that
are removed for technical reasons or because they are both unneeded and
very large, and both these sorts of removal are documented using this
field?

How about "are stripped" not "must be stripped", as the normative stuff
is explained more precisely over in the Policy manual itself?

Nice.

This feature is orthogonal to the machine readable copyright format.

It should be possible to use it with the plain old copyright format too,
otherwise we are kind of renegating on our promise that the machine
readable copyright format be optionnal.

Cheers,

Hi,

Am Thu, Jan 27, 2022 at 02:52:18PM -0700 schrieb Sean Whitton:

I like this wording.  I'm not sure whether we might add what I'm doing
in practice:  If there is the need to remove non-DFSG files it might
make sense to use other files that are typically unused for Debian and
thus are just bluring the content of the tarball.

I'm unsure whether we should really add this to policy but this is what
I'm doing.

+1

Thanks a lot for working on this in Debian policy

     Andreas.

Hi,

Am Thu, Jan 27, 2022 at 11:25:31PM +0100 schrieb Bill Allombert:

That's true for the introductory sentense.  Its not really fair to cut
the real point

+        to document the removal of these files from the original upstream
+        source. This allows others to understand or audit how the source
+        distribution in Debian is derived from the upstream source.

which should be mentioned in policy.

Did we ever promised this?

I personally would rather think that excluding files from the upstream
source is a pretty good reason to make DEP5 mandatory *for these cases*.
Besides a sensible way of documentation it saves maintainer time to
simply use uscan to obtain a proper tarball.

Kind regards
    Andreas.

Yes we did.

The issue is that mk-origtargz does not actually need any field from
DEP5, it only needs the extra fields it specifies. It would be more
sensible for mk-origtargz to get this information from debian/watch
or debian/mk-origtargz.

While the new copyright format is part of the policy package, it is
a separate specification and is not intented for package to put
configuration information there.

From the point of view of binary package, the information is quite often
meaningless since there is not explanation why a file was removed,
files that are removed are often regenerated at built time and shipped
and the file not removed from the source might still not be shipped in
any binary package generated from it. Important feature removal should
rather be documented in README.Debian.

So it is quite sufficient for this data to be in the source package
only.

Cheers,

Hi Bill,

Am Fri, Jan 28, 2022 at 10:00:36AM +0100 schrieb Bill Allombert:

Any links to refresh my mind?

I can't parse this.  Do you mean `man mk-origtargz` is reading fields
that are not mentioned in DEP5.  I'd say DEP5 is just older than
Files-Excluded and the good thing about DEP5 is that it *enabled* other
tools to parse those fields.

I do not see any need for a new file debian/mk-origtargz for a solved
problem.  I also do not think that debian/watch is a good place since it
defined where to get a file from and how to detect a new version.  From
my point of view debian/copyright is the perfect place to express what
we need to exclude for copyright reasons.

The Files-Excluded field is not necessarly configuration information -
it documents in a machine-readable format what is excluded.  The
advantage of a machine-readable format is that machines can read it
... which is done by mk-origtargz (and not only by this - also some
UDD gatherers are parsing the licenses).

I do not see any conflict in mentioning sensible information in
README.Debian in addition to debian/copyright.

Sure - but this is no reason to mention it in d/copyright as well.

What I'm actually imagine is that the time I need to write this kind of
mails I could perfectly use to convert 2-3 d/copyright files from old
format to DEP5.  And now I'm wondering why I'm not doing so ...

Kind regards

       Andreas.

On Fri, Jan 28, 2022 at 12:41:39PM +0100, Andreas Tille wrote:
)

mk-origtargz ignore all the fields that are currently
mentionned in DEP5. The only fields that are used are the ones it
introduces. So in that sense it is not an application of DEP5.

Do not slow down on my account!

Cheers,

Am Fri, Jan 28, 2022 at 12:56:56PM +0100 schrieb Bill Allombert:

When doing the review updating and fixing some issues.

Kind regards
     Andreas.