#951386 uscan: fails to downloads from a “dumb” HTTP git repository

Package:
devscripts
Source:
devscripts
Description:
scripts to make the life of a Debian Package maintainer easier
Submitter:
nicoo
Date:
2021-01-13 23:39:02 UTC
Severity:
important
#951386#5
Date:
2020-02-15 19:20:08 UTC
From:
To:
Dear maintainers,

While updating debian/watch for package xe, I discovered that uscan fails
to download packages, in mode=git, when given a repository served over the
“dumb” HTTP(S) transport (in particular, without support for shallow clones):
I would expect uscan to fallback to a non-shallow clone in this setting.


Best,

  nicoo

- -- System Information:
Debian Release: bullseye/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 5.4.0-3-amd64 (SMP w/4 CPU cores)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_WARN, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages devscripts depends on:
ii  dpkg-dev              1.19.7
ii  fakeroot              1.24-1
ii  file                  1:5.38-4
ii  gnupg                 2.2.19-1
ii  gnupg2                2.2.19-1
ii  gpgv                  2.2.19-1
ii  libc6                 2.29-10
ii  libfile-homedir-perl  1.004-1
ii  libfile-which-perl    1.23-1
ii  libipc-run-perl       20180523.0-2
ii  libmoo-perl           2.003006-1
ii  libwww-perl           6.43-1
ii  patchutils            0.3.4-2+b1
ii  perl                  5.30.0-9
ii  python3               3.7.5-3
ii  sensible-utils        0.0.12+nmu1
ii  wdiff                 1.2.2-2+b1

Versions of packages devscripts recommends:
ii  apt                         1.8.4
pn  at                          <none>
ii  curl                        7.67.0-2
pn  dctrl-tools                 <none>
ii  debian-keyring              2020.02.02
ii  dput-ng [dput]              1.29
pn  equivs                      <none>
ii  libdistro-info-perl         0.23
ii  libdpkg-perl                1.19.7
ii  libencode-locale-perl       1.05-1
ii  libgit-wrapper-perl         0.048-1
ii  libgitlab-api-v4-perl       0.23-1
ii  liblist-compare-perl        0.53-1
ii  liblwp-protocol-https-perl  6.07-2
ii  libsoap-lite-perl           1.27-1
ii  libstring-shellquote-perl   1.04-1
ii  libtry-tiny-perl            0.30-1
ii  liburi-perl                 1.76-1
ii  licensecheck                3.0.42-1
ii  lintian                     2.50.0
ii  man-db                      2.9.0-2
ii  patch                       2.7.6-6
ii  python3-apt                 1.8.5
ii  python3-debian              0.1.36
ii  python3-magic               2:0.4.15-3
ii  python3-requests            2.22.0-2
pn  python3-unidiff             <none>
ii  python3-xdg                 0.26-1
ii  strace                      4.26-0.2
ii  unzip                       6.0-25
ii  wget                        1.20.3-1+b2
ii  xz-utils                    5.2.4-1+b1

Versions of packages devscripts suggests:
ii  adequate                                  0.15.2
ii  autopkgtest                               5.11
pn  bls-standalone                            <none>
pn  bsd-mailx | mailx                         <none>
ii  build-essential                           12.8
ii  check-all-the-things                      2017.05.20+nmu1
pn  cvs-buildpackage                          <none>
ii  debhelper                                 12.9
pn  devscripts-el                             <none>
ii  diffoscope                                136
pn  disorderfs                                <none>
ii  dose-extra                                5.0.1-14+b2
pn  duck                                      <none>
pn  faketime                                  <none>
pn  gnuplot                                   <none>
pn  how-can-i-help                            <none>
pn  libauthen-sasl-perl                       <none>
pn  libdbd-pg-perl                            <none>
pn  libfile-desktopentry-perl                 <none>
pn  libnet-smtps-perl                         <none>
pn  libterm-size-perl                         <none>
ii  libtimedate-perl                          2.3100-1
pn  libyaml-syck-perl                         <none>
pn  mozilla-devscripts                        <none>
pn  mutt                                      <none>
ii  openssh-client [ssh-client]               1:8.1p1-5
ii  piuparts                                  1.1.1
ii  postgresql-client-11 [postgresql-client]  11.6-2~sid1
ii  quilt                                     0.65-3
ii  ratt                                      0.0~git20180127.c44413c-2+b10
pn  reprotest                                 <none>
pn  svn-buildpackage                          <none>
pn  w3m                                       <none>

- -- no debconf information
-----BEGIN PGP SIGNATURE-----

iQJFBAEBCgAvFiEEU7EqA8ZVHYoLJhPE5vmO4pLV7MsFAl5IRGYRHG5pY29vQGRl
Ymlhbi5vcmcACgkQ5vmO4pLV7Mum2A//cPo0dBAcIHrbzCahyjoZkqa9fV0BCLhA
7UiP5iY04O1AnM5EoeanKOMwfLAOJ53j9iocSlLrLfyUQYtIDS+tzIqn+dSW6Ih2
s83+wPwOtMx2MMPdQaBBT/yIP3sTHfAXol46abRMRJ/c8cUg7Eliah/mqp9RbTvS
kaoSe0+h6RejfR54uBVcRdm8xshnQoUJBpSG37ssxJnnGeu66A1KOQbQLHvzF9Z+
AO+Y7fM7fCptGvA9YAO93a+WCcD0IjXq3EtjRWV2xvl/kTvbTeLcL1WpBwXYyDoZ
hYPo3pvnQIbRsNXFrK3v9jAmmGuo74I2IXbXOxRzFlM0TE/H3IUl/pzkhUlja0Ox
hMBNwZWAN7RKi4gqTgas1SMNljx1tsHFWFr6XxjmkJ1rfE6Urk/t2eL1Gi5jjUqg
LEKNECV1scRT69rCPvEpEoAVRfh5hCbsjonPOKzu0XMvNPfnL7JSxPag38wkgIy4
tnrTRpOF0YoTvudCcl+fjfacrEIu9Z8P8EZq7GeGK6y+vinPfJ0+BxafE/Y8cJOJ
HwFWHM++C8xKaQCtfPctsZEzPacjXqpPty1rdEbd94im/oM/grCJp5qhee+EP+Ce
N9RgeCNslG9MmcsVjTB4ShyTkXTcJCxjFmgwl6Agn+NBZ4OONaWZXCpm7v4Q0Pbf
XglrkRXbEOQ=
=Ryfz
-----END PGP SIGNATURE-----

#951386#10
Date:
2020-03-09 20:42:37 UTC
From:
To:
Hi,

I'm unable to reproduce this bug:

$ uscan -v --force-download
uscan info: uscan (version 2.20.2) See uscan(1) for help
uscan info: Scan watch files in .
uscan info: Check debian/watch and debian/changelog in .
uscan info: package="xe" version="0.11-4" (as seen in debian/changelog)
uscan info: package="xe" version="0.11" (no epoch/revision)
uscan info: ./debian/changelog sets package="xe" version="0.11"
uscan info: Found upstream signing keyring: debian/upstream/signing-key.asc
uscan info: Process watch file at: debian/watch
    package = xe
    version = 0.11
    pkg_dir = .
uscan info: opts: mode=git, pgpmode=gittag
uscan info: line: https://github.com/leahneukirchen/xe.git
refs/tags/v(\d\S+)
uscan info: Parsing mode=git
uscan info: Parsing  pgpmode=gittag
uscan info: line: https://github.com/leahneukirchen/xe.git
refs/tags/v(\d\S+)
uscan info: Last orig.tar.* tarball version (from debian/changelog): 0.11
uscan info: Last orig.tar.* tarball version (dversionmangled): 0.11
uscan info: Execute: git ls-remote https://github.com/leahneukirchen/xe.git
uscan info: Found the following matching refs:
     refs/tags/v0.11 (0.11)
     refs/tags/v0.10 (0.10)
     refs/tags/v0.9 (0.9)
     refs/tags/v0.8 (0.8)
     refs/tags/v0.7.0 (0.7.0)
     refs/tags/v0.6.1 (0.6.1)
     refs/tags/v0.6 (0.6)
     refs/tags/v0.5 (0.5)
     refs/tags/v0.4 (0.4)
     refs/tags/v0.3 (0.3)
     refs/tags/v0.2 (0.2)
     refs/tags/v0.1 (0.1)
     HEAD ()
     refs/heads/master ()
     refs/pull/1/head ()
     refs/pull/1/merge ()
     refs/pull/2/head ()
     refs/pull/3/head ()
     refs/pull/3/merge ()
     refs/pull/4/head ()
     refs/pull/4/merge ()
     refs/pull/5/head ()
     refs/pull/5/merge ()
uscan info: Looking at $base = https://github.com/leahneukirchen/xe.git with
    $filepattern = refs/tags/v(\d\S+) found
    $newfile     = refs/tags/v0.11
    $newversion  = 0.11
    $lastversion = 0.11
uscan info: Upstream URL(+tag) to download is identified as
https://github.com/leahneukirchen/xe.git refs/tags/v0.11
uscan info: Filename (filenamemangled) for downloaded file: xe-0.11.tar.xz
uscan info: Newest version of xe on remote site is 0.11, local version
is 0.11
uscan info:    => Package is up to date for from
https://github.com/leahneukirchen/xe.git refs/tags/v0.11
uscan info:    => Forcing download as requested
uscan info: Downloading upstream package: xe-0.11.tar.xz
Clonage dans le dépôt nu '../xe-temporary.3071220.git'
remote: Enumerating objects: 18, done.
remote: Counting objects: 100% (18/18), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 18 (delta 0), reused 12 (delta 0), pack-reused 0
Dépaquetage des objets: 100% (18/18), 17.82 Kio | 912.00 Kio/s, fait.
uscan info: Successfully downloaded package: xe-0.11.tar.xz
gpgv: Signature faite le dim. 05 nov. 2017 21:03:23 CET
gpgv:                avec la clef RSA C9C0E3814184EAFF
gpgv: Impossible de vérifier la signature : Pas de clef publique
uscan die: OpenPGP signature did not verify. at
/usr/share/perl5/Devscripts/Uscan/Output.pm line 58.

#951386#17
Date:
2021-01-13 23:36:11 UTC
From:
To:
Hi Xavier,
- first, I got confused and gave you a watch file that did *not*
  exhibit the bug, instead of my package's original d/watch ;
- second, when I looked at the bug since, I couldn't reproduce either...
  because the watch file was pointed at git.vuxu.org, which implemented
  git's “smart” HTTP transport since.

Sorry for the super-slow reply, it took be a good while to realise what happened

So, here is a script that's a reproducing testcase; it relies on having a
copy of the xe repo (git.vuxu.org/xe) on a “dumb” HTTP remote, here just
https://nicolas.braud-santoni.eu/tmp/xe.bare

This is the output I get from it here:

	$ git clone https://salsa.debian.org/Debian/xe.git
	Cloning into 'xe'...
	remote: Enumerating objects: 642, done.
	remote: Counting objects: 100% (642/642), done.
	remote: Compressing objects: 100% (277/277), done.
	remote: Total 642 (delta 374), reused 616 (delta 355), pack-reused 0
	Receiving objects: 100% (642/642), 136.34 KiB | 293.00 KiB/s, done.
	Resolving deltas: 100% (374/374), done.

	$ cd xe

	$ sed -i s,git\.vuxu\.org/xe/,nicolas.braud-santoni.eu/tmp/xe.bare/, debian/watch

	$ cat debian/watch
	version=4
	opts="mode=git, pgpmode=gittag" \
https://nicolas.braud-santoni.eu/tmp/xe.bare/ refs/tags/v(\d\S+)

	$ uscan --force-download -v
	uscan info: uscan (version 2.20.5) See uscan(1) for help
	uscan info: Scan watch files in .
	uscan info: Check debian/watch and debian/changelog in .
	uscan info: package="xe" version="0.11-5" (as seen in debian/changelog)
	uscan info: package="xe" version="0.11" (no epoch/revision)
	uscan info: ./debian/changelog sets package="xe" version="0.11"
	uscan info: Found upstream signing keyring: debian/upstream/signing-key.asc
	uscan info: Process watch file at: debian/watch
	    package = xe
	    version = 0.11
	    pkg_dir = .
	uscan info: opts: mode=git, pgpmode=gittag
	uscan info: line: https://nicolas.braud-santoni.eu/tmp/xe.bare/ refs/tags/v(\d\S+)
	uscan info: Parsing mode=git
	uscan info: Parsing  pgpmode=gittag
	uscan info: line: https://nicolas.braud-santoni.eu/tmp/xe.bare/ refs/tags/v(\d\S+)
	uscan info: Last orig.tar.* tarball version (from debian/changelog): 0.11
	uscan info: Last orig.tar.* tarball version (dversionmangled): 0.11
	uscan info: Execute: git ls-remote https://nicolas.braud-santoni.eu/tmp/xe.bare/
	uscan info: Found the following matching refs:
	     refs/tags/v0.11 (0.11)
	     refs/tags/v0.10 (0.10)
	     refs/tags/v0.9 (0.9)
	     refs/tags/v0.8 (0.8)
	     refs/tags/v0.7.0 (0.7.0)
	     refs/tags/v0.6.1 (0.6.1)
	     refs/tags/v0.6 (0.6)
	     refs/tags/v0.5 (0.5)
	     refs/tags/v0.4 (0.4)
	     refs/tags/v0.3 (0.3)
	     refs/tags/v0.2 (0.2)
	     refs/tags/v0.1 (0.1)
	     HEAD ()
	     refs/heads/debian/sid ()
	     refs/heads/pristine-tar ()
	     refs/heads/upstream ()
	     refs/heads/v0.11-5 ()
	     refs/tags/debian/0.11-2 ()
	     refs/tags/debian/0.11-3 ()
	     refs/tags/debian/0.11-4 ()
	     refs/tags/debian/0.11-5 ()
	     refs/tags/debian/v0.10-1 ()
	     refs/tags/debian/v0.11-1 ()
	uscan info: Looking at $base = https://nicolas.braud-santoni.eu/tmp/xe.bare/ with
	    $filepattern = refs/tags/v(\d\S+) found
	    $newfile     = refs/tags/v0.11
	    $newversion  = 0.11
	    $lastversion = 0.11
	uscan info: Upstream URL(+tag) to download is identified as    https://nicolas.braud-santoni.eu/tmp/xe.bare/ refs/tags/v0.11
	uscan info: Filename (filenamemangled) for downloaded file: xe-0.11.tar.xz
	uscan info: Newest version of xe on remote site is 0.11, local version is 0.11
	uscan info:  => Package is up to date from:
	             => https://nicolas.braud-santoni.eu/tmp/xe.bare/ refs/tags/v0.11
	uscan info:  => Forcing download as requested
	uscan info: Downloading upstream package: xe-0.11.tar.xz
	Cloning into bare repository '../xe-temporary.21614.git'...
	fatal: dumb http transport does not support shallow capabilities
	uscan error: Command failed (git clone --bare --depth=1 -b v0.11 https://nicolas.braud-santoni.eu/tmp/xe.bare/ ../xe-temporary.21614.git)
	$ rm -rf /tmp/tmp.FFg74iljuz