Hello, I would guess what happened here is that git-deborig made a .tar.xz but tag2upload-obtain-origs was expecting to find a .tar.gz so it failed. git-deborig doesn't have a way to force .gz compression so I think we need to give it a new option to do that. Ian, I'll add that to git-deborig, perhaps you could handle the changes to tag2upload-obtain-origs?
Sean Whitton writes ("Bug#1121367: tag2upload-obtain-origs: Needs to ask git-deborig for .gz compression"):
the .orig, after it has found that there ia an existing .orig in the
archive, but that it couldn't obtain it.
The thing is, I think that nowadays we should almost always be able to
get the actual orig from the archive, from incoming if necessary.
I think this was job 1851? I looked at the transcript and I see it
trying
https://incoming.debian.org/debian-buildd/pool/main/g/golang-github-package-url-packageurl-go/golang-github-package-url-packageurl-go_0.1.3.orig.tar.gz
amonst other URLs. Evidently it got 404, but that URL works for me
now.
I looked at tracker and
https://tracker.debian.org/news/1692742/accepted-golang-github-package-url-packageurl-go-013-1-source-all-into-unstable/
The Accepted mail there is Date: Tue, 25 Nov 2025 09:10:14 +0000.
The t2u failure report for job 1851 [1] is dated Nov 25 09:45:36 GMT.
So IDK what happened here.
Having said all that, I think this is a reasonable way to try to make
this work despite all the headwinds.
Sure. Would you do the test case? Ie hand me the branch with the new
test failing. (I think the test case is perhaps the hardest part
here, but if we don't write it, our iteration cycle will involve t2u
server-side deployment *and* waiting for a failure.)
Regards,
Ian.
[1] Please always quote the job number.
Hello, Okay, I'll give it a shot. ACK.
1) I did 'git-debpush' for a NEW upload, causing something not in the Debian archive at all to show up on dgit, but the source-ful upload from tag2upload succeding on your side but failing on the upload queue side. I upload my amd64 build at the same time as I do the git-debpush, and since gbp defaults to *.gz and tag2upload defaults to *.xz there is no filename clash in the upload, so my amd64 upload "wins" and the source-only upload gets discarded eventually. 2) This package was ACCEPTED by ftp-masters into the archive, and BEFORE the orig.tar file appeared on deb.debian.org etc I did a source-only -2 upload of the package. That failed below, and I was somewhat ready for this to fail, because I wanted to see how things behaved in this situation. So maybe you want to change something, but I'm not convinced that it is really necessary. It may be okay to treat this as "user confused, move along". I'm not even sure what you could actually do better? The problem in 2) is a race condition where I tag2upload'ed before the orig.tar was available. Is it possible for your queue daemon to notice this situation, and just re-try 6 hours later? Re 1) I have used this a couple of times now. It leads to things not in the archive showing up on browse.dgit.debian.org. This is open for abuse, any DD could start to spam Salsa with millions of git repos and valid tag2upload signed tags, and you would import all of it. I have been hoping you wouldn't notice, or care, because I find the ability to make a immutable (sort of) record of the -1 NEW upload in dgit useful. /Simon Sean Whitton <spwhitton@spwhitton.name> writes:
Hi. Thanks for the explanations. I guess that our beta phase is
precisely when you should be carrying out experiments like this...
Simon Josefsson writes ("Bug#1121367: tag2upload-obtain-origs: Needs to ask git-deborig for .gz compression"):
In the dgit data model, this is an acceptable out-of-course situation.
Unfortunately, we must work with the existing archive machinery which
is very asynchronous and hard to deal with. So skew like this cannot
be entirely prevented.
This skew *is* undesirable so we try to minimise it. Sadly I don't
think there is a reasonable way for git-debpush to detect this
particular situation, without having it query the archive API, which
we don't want it to do because it's supposed to be a pure git tool.
(I would be open to reconsidering this principle TBH. A call to the
ftpmaster API service wouldn't be a lot of code nor a lot of data
transfer, and it would jsut be part of the checks, not entangled with
the tag creation functionality.)
Why did you make these two uploads at the same time? You explain what
you did but not whether you did it deliberately :-).
Our recommendation for NEW is to use dgit. "dgit push-built",
specifically. This fits roughly ito the same space as git-debpush,
except that of course you have to build binaries. "dgit sbuild" and
"dgit pbuilder" will do that for you. Presumably you have a suitable
chroot setup.
(We don't like that we need wrappers for these build commands, but
they're quite shallow wrappers.)
I think if we did what Sean suggests in this bug report, this would
work (at least, almost all of the time) - because t2u would be able to
reproduce the gbp-generated .orig.tar.gz.
You make a couple of other points about handling of NEW in *.dgit.d.o:
Was this package wholly-NEW, or only NEW due to new binary packages?
The system is supposed to hide wholly-NEW packages from the public git
view on browse.dgit.d.o and git.dgit.d.o until they are ACCEPTed.
If that isn't working it needs fixing.
TBH I think this would be an attack that we expect a DD not to
conduct. So that's outside our threat model. If it were to happen,
we would manually delete the bullshit and get DAM to eject the
perpetrator.
That the -1 upload is visible to authorised users, while in NEW, via
push.dgit.d.o, is deliberate, for the reasons you gave and that I've
snipped.
HTH.
Ian.
Ian Jackson <ijackson@chiark.greenend.org.uk> writes: FWIW, as a user I would prefer if you stay out of using ftpmaster API service, because I think dgit has a good place in a future git-oriented Debian but the ftpmaster APIs less so. Oh. My goal was to get the package source code into dgit (to make it possible to build packages directly from dgit), and pushing a signed tag to Salsa does that easily for me. I went from gbp-build-package+debsign+dput directly to tag2upload, so I never integrated dgit into my workflow. I have more NEW uploads coming up, so I will try to use 'dgit push-built' next time, I don't recall if I ever got dgit to build packages for me. Without parsing gbp-related files, which I think you want to avoid? That would be nice. Wholly NEW. I didn't actually check but merely assumed that it automatically appeared on browse.dgit.d.o. I have several NEW uploads coming, I can try to use the same method again and report back. Nothing beats throwing things at live systems. /Simon
Simon Josefsson writes ("Re: Bug#1121367: tag2upload-obtain-origs: Needs to ask git-deborig for .gz compression"):
Right.
Please let us know your experience.
I think we are probably OK parsing at least some gbp-related files.
But I don't think that's needed here, since the tag2upload service can
tell fromk the ftpmaster API that a .gz is wanted.
:-).
Ian.
Simon Josefsson <simon@josefsson.org> writes: 'golang-github-cyclonedx-cyclonedx-go' via tag2upload yesterday, and it is in the NEW queue through my separate amd64 upload. I can't find it on browse.dit.d.o. So presumably you do have it, but it is hidden? /Simon
Simon Josefsson writes ("Bug#1121367: tag2upload-obtain-origs: Needs to ask git-deborig for .gz compression"):
Yes:
$ dgit --for-push clone golang-github-cyclonedx-cyclonedx-go
canonical suite name for unstable is sid
fetching existing git history
no version available from the archive
Package not found in the archive, but has allegedly been pushed using dgit.
Perhaps the upload is stuck in incoming. Using the version from git.
HEAD is now at 14b3187 Update changelog for 0.9.3-1 release
dgit ok: ready for work in golang-github-cyclonedx-cyclonedx-go
$ git ls-remote git+ssh://dgit@push.dgit.debian.org/dgit/debian/repos/golang-github-cyclonedx-cyclonedx-go.git
14b31877c7aefeefda1c7b1a225a3b152894916d HEAD
14b31877c7aefeefda1c7b1a225a3b152894916d refs/dgit/sid
14b31877c7aefeefda1c7b1a225a3b152894916d refs/heads/master
...
It's hidden from public view but available to people whose ssh key has
been enrolled.
Ian.
Hello, To my mind the main thing about this principle is that we don't deal with Debian-specific stuff locally. A query to the ftpmaster API does not seem like a major violation of that, especially since, as you say, it would be part of the checks. I'm not sure we want to add this now but noting this for the future.
Ian Jackson <ijackson@chiark.greenend.org.uk> writes: remember do NEW uploads like this next time, without tag2upload. /Simon jas@kaka:~/dpkg/golang-github-openvex-go-vex$ dgit push-built --new -Cgolang-github-openvex-go-vex_0.2.7-1_amd64.changes Format `3.0 (quilt)', need to check/update patch stack canonical suite name for unstable is sid no version available from the archive nothing quilty to commit, ok. checking that golang-github-openvex-go-vex_0.2.7-1.dsc corresponds to HEAD dpkg-source: varning: extraherar osignerat källkodspaket (/home/jas/dpkg/golang-github-openvex-go-vex/../golang-github-openvex-go-vex_0.2.7-1.dsc) dpkg-source: info: extraherar golang-github-openvex-go-vex i unpacked dpkg-source: info: packar upp golang-github-openvex-go-vex_0.2.7.orig.tar.gz dpkg-source: info: packar upp golang-github-openvex-go-vex_0.2.7-1.debian.tar.xz ../golang-github-openvex-go-vex_0.2.7-1_amd64.changes already has appropriate .orig(s) (if any) gpg: VARNING: mottagare (-r) angivna utan att använda publik nyckel-kryptering gpg: WARNING: server 'gpg-agent' is older than us (2.2.27 < 2.4.7) gpg: Note: Outdated servers may lack important security fixes. gpg: Note: Use the command "gpgconf --kill all" to restart them. gpg: problem with fast path key listing: IPC-parameterfel - ignored gpg: VARNING: mottagare (-r) angivna utan att använda publik nyckel-kryptering gpg: WARNING: server 'gpg-agent' is older than us (2.2.27 < 2.4.7) gpg: Note: Outdated servers may lack important security fixes. gpg: Note: Use the command "gpgconf --kill all" to restart them. gpg: problem with fast path key listing: IPC-parameterfel - ignored gpg: Signatur gjord tis 25 nov 2025 17:32:33 CET gpg: med EDDSA-nyckeln A3CC9C870B9D310ABAD4CF2F51722B08FE4745A2 gpg: issuer "simon@josefsson.org" gpg: Korrekt signatur från "Simon Josefsson <simon@josefsson.org>" [förbehållslös] gpg: Signatur gjord tis 25 nov 2025 17:32:32 CET gpg: med EDDSA-nyckeln A3CC9C870B9D310ABAD4CF2F51722B08FE4745A2 gpg: issuer "simon@josefsson.org" gpg: Korrekt signatur från "Simon Josefsson <simon@josefsson.org>" [förbehållslös] varning: Du verkar ha klonat ett tomt arkiv. Räknar upp objekt: 2076, klart. Räknar objekt: 100% (2076/2076), klart. Deltakomprimering använder upp till 16 trådar Komprimerar objekt: 100% (835/835), klart. Skriver objekt: 100% (2076/2076), 849,71 KiB | 60,69 MiB/s, klart. Totalt 2076 (delta 1207), återanvände 2047 (delta 1198), paket-återanvända 0 (från 0) remote: Analyserar delta: 100% (1207/1207), klart. remote: To /srv/dgit.debian.org/dispatch-dir/distro=debian/repos/_tmp/golang-github-openvex-go-vex_prospective remote: * [new reference] 0dd3f524959e858cc5c65fc105bcc75a6c0c12e0 -> refs/dgit/sid remote: * [new tag] b5722091e4e5728f4152f1a58e629fda3c623870 -> archive/debian/0.2.7-1 remote: * [new tag] 0a756b02c518e3d58d3d2cc3cd52698c64a072a1 -> debian/0.2.7-1 remote: To /srv/dgit.debian.org/dispatch-dir/distro=debian/repos/_tmp/golang-github-openvex-go-vex_prospective remote: * [new branch] 0dd3f524959e858cc5c65fc105bcc75a6c0c12e0 -> master To git+ssh://push.dgit.debian.org/dgit/debian/repos/golang-github-openvex-go-vex.git * [new reference] 0dd3f524959e858cc5c65fc105bcc75a6c0c12e0 -> refs/dgit/sid * [new tag] archive/debian/0.2.7-1 -> archive/debian/0.2.7-1 * [new tag] debian/0.2.7-1 -> debian/0.2.7-1 signfile dsc ../golang-github-openvex-go-vex_0.2.7-1.dsc Simon Josefsson <simon@josefsson.org> gpg: VARNING: mottagare (-r) angivna utan att använda publik nyckel-kryptering gpg: WARNING: server 'gpg-agent' is older than us (2.2.27 < 2.4.7) gpg: Note: Outdated servers may lack important security fixes. gpg: Note: Use the command "gpgconf --kill all" to restart them. gpg: problem with fast path key listing: IPC-parameterfel - ignored fixup_buildinfo ../golang-github-openvex-go-vex_0.2.7-1.dsc ../golang-github-openvex-go-vex_0.2.7-1_amd64.buildinfo signfile buildinfo ../golang-github-openvex-go-vex_0.2.7-1_amd64.buildinfo Simon Josefsson <simon@josefsson.org> gpg: VARNING: mottagare (-r) angivna utan att använda publik nyckel-kryptering gpg: WARNING: server 'gpg-agent' is older than us (2.2.27 < 2.4.7) gpg: Note: Outdated servers may lack important security fixes. gpg: Note: Use the command "gpgconf --kill all" to restart them. gpg: problem with fast path key listing: IPC-parameterfel - ignored fixup_changes dsc ../golang-github-openvex-go-vex_0.2.7-1.dsc ../golang-github-openvex-go-vex_0.2.7-1_amd64.changes fixup_changes buildinfo ../golang-github-openvex-go-vex_0.2.7-1_amd64.buildinfo ../golang-github-openvex-go-vex_0.2.7-1_amd64.changes signfile changes ../golang-github-openvex-go-vex_0.2.7-1_amd64.changes Simon Josefsson <simon@josefsson.org> gpg: VARNING: mottagare (-r) angivna utan att använda publik nyckel-kryptering gpg: WARNING: server 'gpg-agent' is older than us (2.2.27 < 2.4.7) gpg: Note: Outdated servers may lack important security fixes. gpg: Note: Use the command "gpgconf --kill all" to restart them. gpg: problem with fast path key listing: IPC-parameterfel - ignored Successfully signed dsc, buildinfo, changes files Uploading golang-github-openvex-go-vex using ftp to ftp-master (host: ftp.upload.debian.org; directory: /pub/UploadQueue/) running protected-distribution: warn before uploading to distributions where a special policy applies running checksum: verify checksums before uploading running suite-mismatch: check the target distribution for common errors running gpg: check GnuPG signatures before the upload Uploading golang-github-openvex-go-vex_0.2.7-1.dsc Uploading golang-github-openvex-go-vex_0.2.7.orig.tar.gz Uploading golang-github-openvex-go-vex_0.2.7-1.debian.tar.xz Uploading golang-github-openvex-go-vex-dev_0.2.7-1_all.deb Uploading golang-github-openvex-go-vex_0.2.7-1_amd64.buildinfo Uploading golang-github-openvex-go-vex_0.2.7-1_amd64.changes dgit ok: pushed and uploaded 0.2.7-1 jas@kaka:~/dpkg/golang-github-openvex-go-vex$