Some watch files are now broken because GitHub archive URLs now include the
full ref rather than the tag name. It would be great if lintian could warn when this was the case.

See e.g. the watch file for jupyter-core:

https://qa.debian.org/cgi-bin/watch?pkg=jupyter-core

which reports the current upstream version as refs/tags/4.7.1

Hi Jelmer,

I saw the traffic on IRC where someone suggested we replace

    .*archive/v?([0-9.]*).tar.gz

with

    .*archive/.*/v?([0-9.]*).tar.gz

to fix at least 1,500 affected packages. Unfortunately, that may not
work for jupyter-core, which does not prefix tags with a "v" and for
which "(.*)" catches the slash (or maybe even slashes).

As a tool without network access, Lintian is not well positioned to
figure out, in general, whether a URL/regex combination works. Would
it be okay if Lintian instead issues two now classification tags?

The first would occur once per source. It shows the watch file URL and
the regular expression for HTML parsing, possibly followed by "debian
update" (or similar). The second tag would occur once for each of the
options selected, i.e. multiple times. Armed with that information,
the Janitor could probe the URL and figure out which parts need
fixing.

The watch file version is already available in UDD, as you know, so
you could reconstruct the watch file and perhaps even enlist 'uscan'
to help you.

The parsing for these components is in place. If it is time sensitive,
I could provide the new tags via UDD within 48 hours. What do you
think? Thank you!

Kind regards
Felix Lechner

I started a branch for lintian-brush here:
https://salsa.debian.org/chronitis/lintian-brush/-/tree/github-archive-url

(using a nonexistant lintian tag, so having a real one would definitely
be a first step).

However, it turned out to be a bit more complex than I first thought (or
hoped):

* Lots of unrelated test cases get broken (since it rewrites their watch
  files)
* Lots of different ways of spelling the match pattern - amongst my
  there were at least three (and subvariants of each)

    - .*/archive/v([0-9.]+)    # now matches nothing
    - .*/archive/(.+)          # now matches refs/tags/x.y.z
    - .*/archive/@ANY_VERSION@ # now matches nothing

  and the discussion on IRC suggested other cases too (adding a wildcard
  for the new /refs/tags/ part, just matching @ANY_VERSION@.tar.gz, etc.
* Unpreservable formatting in several of the test cases I was using
  (continuation lines in comments?)
* What new pattern to actually write? The initial idea was just to
  literally replace /archive/ with /archive/refs/tags/, which _should_
  meet the idea of being conservative about what to fix (but might still
  collide with hand-written fixes for this issue like ./archive/.*/v...

I _think_ a good indicator for lintian (and a fixer) would be if the
matching expression contains "archive" followed by no wildcard pattern
before the capturing group for the version.

Let me know if this makes sense to develop further.

Gordon

Hi Felix,
I was hoping that lintian could verify that there is at least
something after "/archive/" in the matching pattern that could
match slashes without relying on the main regex group - that could be
done without querying GitHub. That said, that code would have to be
updated if GitHub changes again in the future and it may be somewhat
tricky code.

The offer for informational tags is appreciated, but as you say - the
data is already available in UDD so just providing the pure uscan
contents wouldn't help much.

The alternative is to just let lintian-brush work without a signal
from lintian, and gradually grind through the archive. That'll work
too, though it'll take a few months - and we lose the verification
from lintian after the fix.

Jelmer

Thanks! I've merged your branch with some additional changes:

* reformatted the watch files in the examples to use a format that
  debian/watch can preserve
* changed the logic to follow your last suggestion

Ideally the other ways of formatting debian/watch would be handled
too, but that's something that is a work in progress and needs to be
fixed in debmutate.watch.

Jelmer

Hi Jelmer,

Could Lintian-brush or the Janitor do so, when Lintian provides the string?

Kind regards
Felix Lechner

It could, but at that point there isn't much value in having lintian
involved - the janitor could just get that directly from UDD.

The main value in having lintian in the loop here is:

 * lintian actively goes out and discovers issues (especially on fully
   built packages), so that lintian-brush runs can be prioritized.
 * so we can verify that issues that lintian found and lintian-brush
   thought it fixed are actually fixed

Not all issues can be fixed by the janitor either, so it would be
useful to have a lintian tag for those packages that are affected but
can't be fixed.

Hi Jelmer,

That might be better. According to mapreri:

Kind regards
Felix Lechner