It would be useful if uscan had the ability to use external scripts to
handle checking and downloading the newest upstream source where
upstream doesn't use a conventional release management scheme. It
could allow the flexibility of allowing a package maintainer determine
how to best check for and/or download a new upstream source package,
and alleviate the problem of adding complicated code within uscan
that's difficult to maintain.

It could possibly help people who are facing similar problems as
reported in bugs #279236, #351218, and #395439. I have yet another
problem where no-ip upstream has a webpage that displays there latest
package, but there tarball link is always the same name. The webpage
is at http://www.no-ip.com/downloads.php?page=linux

Then there's also the problem with downloading a package. Sometimes a
package is not packed using bzip2 or gzip. Sometimes it's not
delivered using ftp or http. Sometimes the upstream source needs to
have "CVS" directories, binaries, or other things removed. It would be
great if uscan could do all this automatically, even if it means using
an external script.

What I would personally like to do is have a watchfile that has this snippet.

# Run an external script for checking and/or downloading new upstream source
use-check-script=debian/path-to-script \
use-check-script-options="a b c ..." \
use-download-script=debian/path-to-script \
use-download-script-options="a b c ..."

Thus, uscan could be used without seriously having to change how
everyone uses uscan (hopefully) and when these options are used in a
watchfile, uscan will handle them accordingly (again, hopefully and
ideally without forcing any changes on how a user currently runs
uscan).

Hello,

What about contacting and suggest them to use a proper version-based naming?

Quoting myself:

I strongly *object* to adding such a feature because it is very likely that
such scripts can't be accessed by DEHS. Either because the foo package
providing the bar tool is not installed or as simple as the fact that DEHS
_only_ extracts the watch file from the .diff and *not* the whole .orig.

Why are two options needed for this?
E.g. use-download-script="debian/path-to-script a b c ..." \


Sincerely,

Raphael Geissert <atomo64@gmail.com> writes:

What would be really useful is if uscan could unpack the contents of the
downloaded tarball, run a custom script in it, and then repack the
tarball. You can find an example of such a script at [1].

DEHS and similar services only need to check for new versions, but not
really repack them. So this option certainly shouldn't be enabled by
default.

[1]
http://git.debian.org/?p=pkg-multimedia/ffmpeg-debian.git;a=blob;f=debian/strip.sh;hb=HEAD

I think the script should be called with the new version of the package
that was detected and downloaded. This way the script can cope with
changes to the structure of the upstream sources.

If such a thing was already available, please let me know.

uscan already provides a similar feature (but no unpacking is done):
http://.../file-... debian path/to/script

Please refer to uscan(1)

Cheers,

 - This mail is a HTML mail. Not all elements could be shown in plain text mode. -

Hi,
Welcome to the world of Beauty. Visit the site
http://reallifelabs.com
where you will find the best beauty facts and makeup tips for both male & female.
Regards ....
Shilpa

Hi,

Let me propose a new generic mangle rule: qx/script/

This mangles by feeding the target $string into the STDIN of script in
the debian directory and reading its STDOUT back into the target
$string.

This works somewhat like a Perl command thus I used qx as the initiating
command.

    $script = qx"debian/script";

This is applicable to all mangle rules :-)  including pagemangle I
proposed in https://bugs.debian.org/797787 Message #15. (Disregard the
old patch in Message #5 and #10)  The exact patch will be posted there
later.

At least this and pagemangle facilitate users to setup a watch file for
the following bug reports:

 #451051 Please support parsing .lsm files
 #458789 Please allow framework for using external scripts
 #614802 watch file syntax for taking version from timestamp
         ==> The last one needs a new mangling rule after the download.

This is a very generic tool to address many random feature requests
which devscripts maintainer can not be keeping up with.

I just made a proof of concept code snippet which changes the start of
uscan safe_replace($$) as follows:
---------
sub safe_replace($$) {
    my ($in, $pat) = @_;
    $pat =~ s/^\s*(.*?)\s*$/$1/;

    $pat =~ /^(s|tr|y|qx)(.)/;
    my ($op, $sep) = ($1, $2 || '');
    my $esc = "\Q$sep\E";
    my ($parsed_ok, $regexp, $replacement, $flags);

    if ($op eq 'qx') {
        my $input = $$in;
        my $output;
        # script name: [a-zA-Z0-9][-_a-zA-Z0-9]* (any quotation dropped)
        $pat  =~ s/^qx[^-_a-zA-Z0-9]*([a-zA-Z0-9][-_a-zA-Z0-9]*)[^-_a-zA-Z0-9]*$/debian\/$1/;
        if ( -x $pat) {
            spawn(exec => $pat,
            from_string => \$input,
            to_string => \$output,
            wait_child => 1);
            $$in = $output;
            return 1;
        } else {
            print STDERR "$progname warning: missing executable $pat: $!\n";
            return 0;
        }
    } elsif ($sep eq '{' or $sep eq '(' or $sep eq '[' or $sep eq '<') {
...
---------

So we can use this as:
  opts="pagemangle=qx/script/;suffix=+dfsg1;dversionmangle=s/+dfsg\d*$//" ...

There are relatively strict limitations to the script name for the safe
operation.

This uses Dpkg::IPC and I tested the above code in a mock code
environment and works nicely.

Regards,

Osamu

On Sat, Sep 05, 2015 at 11:13:39PM +0900, Osamu Aoki wrote:
...

Oops.

opts="pagemangle=qx/script/,suffix=+dfsg1,dversionmangle=s/+dfsg\d*$//" ...

Osamu

Hi Osamu!

* Osamu Aoki <osamu@debian.org>, 2015-09-05, 23:13:

I'm afraid this is no-no.

Please (and services like mentors.debian.net) run "uscan
--report-status" on untrusted source packages. This change would 
introduce arbitrary code execution vulnerability.

Heh, the whole point of safe_replace() is to defuse Perl regexes, which
normally let you execute arbitrary code.

Hi,

This is an expected response :-)

True.  I agree such functionality should be blocked under --dehs and
--report-status.  After all any script can be executed by placing it in
place of uupdate for non- --dehs and --report-status cases.

I am curious that level of limitation is enough or not.

 rule may not contain any expressions which have the potential to
 execute code (i.e. the (?{})  and  (??{}) constructs are not supported)

I am not going push this any more.  Let me get back with multitarball
thing.

Osamu

Osamu,
	I was just thinking about this because the internet is the modern day
equiavlent of the Wild West.

	It seems the bug report is fairly old and quite a lot of what is talked
about has been implemented. However the essential issue is that some
packages just do not fit into the normal mode. "ksh" is an example. (The
project is dead upstream so it deserves no consideration for that as
well as being odd.) Such projects usually have a get-orig-source target
in their rules file as specified in the policy:
http://www.debian.org/doc/debian-policy/ch-source.html.

	So what we could do is build on that. The "custom" type watch file
would run the "get-orig-source" target and treat a specified directory
as the remote web page and parse that directory as if it were a web page.

Please respond to 748474@bugs.debian.org
Please drop       458789@bugs.debian.org

Hi,

i.e., never trust codes from the internet :-)

https://bugs.debian.org/458789

This is about running script while searching the new version string.
That is handled with mangling rules under safe substitution method.

Adding script running capability beats the whole purpose of
  sub safe_replace ($$)
to ensure safety of this part of action.  (uscan --report ...)

Please read Jakub's message: https://bugs.debian.org/cgi-bin/458789#34

I agree with him and I marked this #458789 as wontfix.

This is another issue.  repacking of tarball is done by mk-origtargz
called from uscan.  See the bug report:
https://bugs.debian.org/cgi-bin/748474

This talks about running script during repacking of tarball
This part of code will not be run if --report is used.  All packaging
crawling usages of uscan under the server environment do not come here.

So I am not all that negative about adding some support for this
feature of running script when repackaging.  (David!)

I was initially thinking to add a watch line option pointing to a
script... but as I read your message, your idea is better direction.

As the policy indicates, the get-orig-source provide downloading feature
too.  So there is an feature overlap with uscan.  We do not need
"download" part.  If there is the "clean-orig-source" target exist,
that is what mk-origtargz should run, I think.

This "clean-orig-source" target maybe better interface than setting
script name in watch file.  We can test script independently without
reading manual :-) Just "./debian/rules clean-orig-source"

Let's see what other people say.

Osamu

Osamu,
	with the get-orig-source I think I really messed up my explanation of
how I see things.

	Once a maintainer has dtermined  that the upstream download method is
too whacky to expect uscan to cope, and has to resort to a
get-orig-source target in the debian/rules file, then yes uscan is out
of the picture for downloading the tarball.

	However uscan is also used in say the DDPO reports to identify when a
new version is available. That function of uscan cannot be handled by
debian/rules but uscan could offer a wrapper around the get-orig-source
target to make that infomration available via the debian/watch interface.

	Does that make my point clearer?

Hi,


Watch file to identify if there is new version or not
get-orig-source to download etc.

Hmmmm, that's easy addition.  Question is is it valuable. Probably.

Let me think a bit more.

(clean-orig-source idea may not be too far as this report....)

Some download script run after --report barrier ...  interesting
thought.

Osamu