Hi,

Thanks for aiming at providing a reliable way to clean up upstream
tarballs with regard to non-DFSG compliant material.

It would be nice if one could also alter specific files within the
upstream tarball, e.g., by removing nonfree ICC profiles from images as
recently discussed on debian-devel. [0]

To do so, it would be convenient to execute arbitrary code during the
repack, e.g. via a --command switch allowing to execute things like:

	exiftool -icc_profile"-<=" $listoffiles

where $listoffiles would refer to some files within the upstream
tarball.

The sound of “execute arbitrary code during the repack” makes me feel
uneasy, but I’d like to make it possible for such argument to the
--command switch to be defined into the debian/ directory (e.g., a
debian/upstream/repack.commands file), and possibly deactivate this
behavior by default if there are security concerns (one then would need
to call mk-origtargz and uscan with a --i-trust-the-debian-dir switch in
order to take this file into account).

	0: https://lists.debian.org/debian-devel/2014/05/thrd2.html#00312

Regards

David
--- /etc/devscripts.conf ---

--- ~/.devscripts ---
DEBSIGN_KEYID=0xB82A217AFDFE09F2

Hi David,

I have been working on uscan related topic recently :-).  Since it is a
bit rough, my latest work is living in a branch off alioth's main branch
(multitar).  So far, multiple upstream tarball, separate URL search for
SIG file, better uupdate integration have been done.

As I see https://bugs.debian.org/748474 , this seems to be something
easily handled by the released code as is. (Maybe there is some corner
case where we may need to address if you wish to use command such as
"gbp import-orig --uscan" which I have not looked into yet.

Just create a script debian/uupdate and create watch file like

version=3
http://www.example.org/downloadguide \
        foo/bar-([\d.]+).tar.xz debian debian/uupdate

The content of debian/uupdate can be:

-------------------
#!/bin/sh
# Assume recent uscan v=3 "--upstream-version <version> <path>"
while [ "$1" ]; do
    case $1 in
    --upstream-version|-v)
        shift; NEW_VERSION="$1"
        ;;
    --force-bad-version|-b|--find|-f)
        :
        ;;
    esca
    shift
done
# ignore <path> and rely on ../ being the path to the orig.tar.xz
# using source package name $SPKG from debian/changelog
# This makes it more robust for newer uupdate in future
SPKG=$(dpkg-parsechangelog -SSource)
cd ..
# Now ${SPKG}_${NEW_VERSION}.orig.tar.xz is upstream tarball
mv -f ${SPKG}_${NEW_VERSION}.orig.tar.xz ${SPKG}_${NEW_VERSION}.orig.tar.xz.orig
tar --xz -xf ${SPKG}_${NEW_VERSION}.orig.tar.xz
# make any modification to files in ${SPKG}-${NEW_VERSION}
# exiftool -icc_profile"-<=" $listoffiles
tar --xz -cf ${SPKG}_${NEW_VERSION}.orig.tar.xz ${SPKG}-${NEW_VERSION}
cd -
uupdate --no-symlink --upstream-version ${NEW_VERSION} ../${SPKG}_${NEW_VERSION}.orig.tar.xz
-------------------

As for "gbp import-orig --uscan", this needs to be updated to cope with
multiple upstream tarballs.  Then we just need to make it honor
debian/uupdate in the watch file at that time.

Regards,

Osamu

Hi,

After thinking about xpi-repack, this ICC_PROFILE may be another one to
solve nicely for many people to clean upstream as easy as possible.
debian/uupdate approach is not easy to use.

Here is my memo:

As for the action to take,
https://lists.debian.org/debian-devel/2014/05/msg00812.html
https://github.com/danwent/Perspectives/pull/120
These are good guide.

As I checked my packages
$ find . -regextype posix-extended -iregex '.*\.(jpg|png)' -exec sh -c 'identify -verbose "$0" | grep -i copyright && echo "$0"' {} \; 2>/dev/null
    Copyright: Public Domain
http://creativecommons.org/licenses/publicdomain/
./ddp/ddp/release-notes/png/caution.png
    Copyright: Public Domain
http://creativecommons.org/licenses/publicdomain/
./webwml/manuals/release-notes/png/caution.png
    Copyright: Public Domain
http://creativecommons.org/licenses/publicdomain/
./ibus/github/ibus/ibus-hangul/setup/ibus-setup-hangul.png
    Copyright: Public Domain
http://creativecommons.org/licenses/publicdomain/
./ibus-hangul/osamu/ibus-hangul-keep/setup/ibus-setup-hangul.png
    Copyright: Public Domain
http://creativecommons.org/licenses/publicdomain/
./ibus-hangul/ibus-hangul/setup/ibus-setup-hangul.png
$

So not all ICC profile data are bad :-)

I wonder if we can add features like Files-Excluded ...
Files-ICC-Removed:
Files-ICC-Removed-<component>:

But this may be over-complication and sticking with adding a feature to
run a packaged script as originally proposed.

At any rate, if --report is used, this part of script is not run.
So security concern can be addressed for cases where uscan are run by
the server environment.

As for the copyright checker as presented with "find ..." in the above,
adding this to the debmake package may be more helpful.

Osamu

Please respond to 748474@bugs.debian.org
Please drop       458789@bugs.debian.org

Hi,

i.e., never trust codes from the internet :-)

https://bugs.debian.org/458789

This is about running script while searching the new version string.
That is handled with mangling rules under safe substitution method.

Adding script running capability beats the whole purpose of
  sub safe_replace ($$)
to ensure safety of this part of action.  (uscan --report ...)

Please read Jakub's message: https://bugs.debian.org/cgi-bin/458789#34

I agree with him and I marked this #458789 as wontfix.

This is another issue.  repacking of tarball is done by mk-origtargz
called from uscan.  See the bug report:
https://bugs.debian.org/cgi-bin/748474

This talks about running script during repacking of tarball
This part of code will not be run if --report is used.  All packaging
crawling usages of uscan under the server environment do not come here.

So I am not all that negative about adding some support for this
feature of running script when repackaging.  (David!)

I was initially thinking to add a watch line option pointing to a
script... but as I read your message, your idea is better direction.

As the policy indicates, the get-orig-source provide downloading feature
too.  So there is an feature overlap with uscan.  We do not need
"download" part.  If there is the "clean-orig-source" target exist,
that is what mk-origtargz should run, I think.

This "clean-orig-source" target maybe better interface than setting
script name in watch file.  We can test script independently without
reading manual :-) Just "./debian/rules clean-orig-source"

Let's see what other people say.

Osamu

Osamu,
	with the get-orig-source I think I really messed up my explanation of
how I see things.

	Once a maintainer has dtermined  that the upstream download method is
too whacky to expect uscan to cope, and has to resort to a
get-orig-source target in the debian/rules file, then yes uscan is out
of the picture for downloading the tarball.

	However uscan is also used in say the DDPO reports to identify when a
new version is available. That function of uscan cannot be handled by
debian/rules but uscan could offer a wrapper around the get-orig-source
target to make that infomration available via the debian/watch interface.

	Does that make my point clearer?

Hi,


Watch file to identify if there is new version or not
get-orig-source to download etc.

Hmmmm, that's easy addition.  Question is is it valuable. Probably.

Let me think a bit more.

(clean-orig-source idea may not be too far as this report....)

Some download script run after --report barrier ...  interesting
thought.

Osamu