#1127825 Erroneous taints when archive doesn't process queue for too long

#1127825#5
Date:
2026-02-11 14:00:45 UTC
From:
To:
Debian tag2upload service [11/Feb 12:01pm GMT] wrote:

tag2upload job #2730.

This is a bug in dgit-repos-server on gideon, I think.

I'm really surprised we haven't seen it sooner than now.

#1127825#10
Date:
2026-02-11 14:46:20 UTC
From:
To:
Hi, Reinhard.  We are investigating an anomaly with dgit/tag2upload
relating to golang-github-microsoft-didx509go.

I don't think you did anything wrong, but I'd like you to confirm some
things to make sure I understand the order of events.


This package currently doesn't work with dgit clone.  The dgit-repos
server seems to have erroneously thought the upload was rejected.

It looks like you did this upload

https://deb.debian.org/debian/pool/main/g/golang-github-microsoft-didx509go/golang-github-microsoft-didx509go_0.0.3-1.dsc
https://tracker.debian.org/news/1655777/accepted-golang-github-microsoft-didx509go-003-1-all-source-into-unstable/

The .changes was generated "Sat, 09 Aug 2025 08:25:35 -0400" ie
2025-08-09T12:25:35Z.  The .dsc contains this

  Dgit: 8a259de4d07ac3fc487b1e0c5ce7585e1f074097 debian archive/debian/0.0.3-1 https://git.dgit.debian.org/golang-github-microsoft-didx509go

so I think you used dgit to do the upload.

Q. to Reinhard: Can you confirm that you used dgit to do this upload?
Do you recall anything going wrong (or weirdly) with it?


It appears that at 2025-08-09T16:37:27Z, the dgit-repos service
decided that your package had been REJECTed.  There is an entry in our
taint database with that timestamp, for the commit names above, saying:

  tag archive/debian/0.0.3-1 referred to this object in git tree but
  all previously pushed versions were found to have been removed from
  NEW (ie, rejected) (or never arrived)

I don't think this can be right.  The package was in fact ACCEPTed
(see URL above) at 2025-08-15T18:00:15Z.  But dgit-repos cannot tell
the difference between a package that is REJECTed and one that is
still en route.  (This is IMO a brokenness in the legacy Debian
Archive.)  I conjecture that the tarballs version of the upload was
delayed somehow.

Q. to Reinhard: do you still have copies of the mails from the archive
systems, particularly the ones saying "Processing of ... .changes"
"golang-github-microsoft-didx509go is NEW" ?  If so can you please
forward them to here to this bug.


I looked on the server and at a similar time we have
golang-github-microsoft-go-winio too.

Q. to Reinhard: If you are able, can you confirm that you did these
two packages roughly the same way and with the same experience ?


Subject to confirmation from Reinhard I propose to manually fix this
up on the server, by restoring the two repositories (from "_garbage")
and by removing the taint indications from the database.


Thanks, and sorry for the inconvenience.

Ian.

#1127825#15
Date:
2026-02-11 14:46:20 UTC
From:
To:
Hi, Reinhard.  We are investigating an anomaly with dgit/tag2upload
relating to golang-github-microsoft-didx509go.

I don't think you did anything wrong, but I'd like you to confirm some
things to make sure I understand the order of events.


This package currently doesn't work with dgit clone.  The dgit-repos
server seems to have erroneously thought the upload was rejected.

It looks like you did this upload

https://deb.debian.org/debian/pool/main/g/golang-github-microsoft-didx509go/golang-github-microsoft-didx509go_0.0.3-1.dsc
https://tracker.debian.org/news/1655777/accepted-golang-github-microsoft-didx509go-003-1-all-source-into-unstable/

The .changes was generated "Sat, 09 Aug 2025 08:25:35 -0400" ie
2025-08-09T12:25:35Z.  The .dsc contains this

  Dgit: 8a259de4d07ac3fc487b1e0c5ce7585e1f074097 debian archive/debian/0.0.3-1 https://git.dgit.debian.org/golang-github-microsoft-didx509go

so I think you used dgit to do the upload.

Q. to Reinhard: Can you confirm that you used dgit to do this upload?
Do you recall anything going wrong (or weirdly) with it?


It appears that at 2025-08-09T16:37:27Z, the dgit-repos service
decided that your package had been REJECTed.  There is an entry in our
taint database with that timestamp, for the commit names above, saying:

  tag archive/debian/0.0.3-1 referred to this object in git tree but
  all previously pushed versions were found to have been removed from
  NEW (ie, rejected) (or never arrived)

I don't think this can be right.  The package was in fact ACCEPTed
(see URL above) at 2025-08-15T18:00:15Z.  But dgit-repos cannot tell
the difference between a package that is REJECTed and one that is
still en route.  (This is IMO a brokenness in the legacy Debian
Archive.)  I conjecture that the tarballs version of the upload was
delayed somehow.

Q. to Reinhard: do you still have copies of the mails from the archive
systems, particularly the ones saying "Processing of ... .changes"
"golang-github-microsoft-didx509go is NEW" ?  If so can you please
forward them to here to this bug.


I looked on the server and at a similar time we have
golang-github-microsoft-go-winio too.

Q. to Reinhard: If you are able, can you confirm that you did these
two packages roughly the same way and with the same experience ?


Subject to confirmation from Reinhard I propose to manually fix this
up on the server, by restoring the two repositories (from "_garbage")
and by removing the taint indications from the database.


Thanks, and sorry for the inconvenience.

Ian.

#1127825#20
Date:
2026-02-11 14:48:26 UTC
From:
To:
Sean Whitton writes ("Bug#1127673: dgit-repos-server: fatal: detected dubious ownership in repository at '$repos/_empty'"):
...

This is very odd.  Literally `$repos` !

It does the same thing with any repo that doesn't exist.  I think it
is trying to substitue an empty tree (for Reasons) but failing.

I think we haven't see this usually because it does not happen with
dgit-repos-policy-debian as the policy hook - unless something else
has gone wrong.  See my other mail.

Ian.

#1127825#25
Date:
2026-02-11 15:51:29 UTC
From:
To:
Ian Jackson writes ("Bug#1127673: dgit-repos-server: fatal: detected dubious ownership in repository at '$repos/_empty'"):
...
...

I have just noticed that 2025-08-09 was the date of the trixie
release.  IIRC the whole legacy Debian Archive system was suspended
for several days for release admin.  I think that would explain
everything.

I'll wait a day or two for Reinhard's reply but I think I'm fairly
confident that this explains everything.

It looks from timestamps in the repos/ directory on gideon like only
these two packages were affected.

AFAIAA this kind of breakage hasn't happened before, but we should
perhaqps consider a way to avoid it happening again.  I'm not sure
what that would look like.

Ian.

#1127825#30
Date:
2026-02-12 11:42:06 UTC
From:
To:
Curiously, I'm not able to find the "Proccessing of ... " email, but I
did find these two emails:


 From	Debian FTP Masters
To	Reinhard Tartler, Debian Go Packaging Team
Date	2025-08-09 17:51
Subject golang-github-microsoft-didx509go_0.0.3-1_multi.changes is NEW

binary:golang-github-microsoft-didx509go-dev is NEW.
binary:golang-github-microsoft-didx509go-dev is NEW.
source:golang-github-microsoft-didx509go is NEW.

Your package has been put into the NEW queue, which requires manual
action
from the ftpteam to process. The upload was otherwise valid (it had a
good
OpenPGP signature and file hashes are valid), so please be patient.

Packages are routinely processed through to the archive, and do feel
free to browse the NEW queue[1].

If there is an issue with the upload, you will receive an email from a
member of the ftpteam.

If you have any questions, you may reply to this email.

[1]: https://ftp-master.debian.org/new.html
      or https://ftp-master.debian.org/backports-new.html for *-backports

And:

 From	Debian FTP Masters
To	Reinhard Tartler, Debian Go Packaging Team
Date	2025-08-15 14:00
Subject golang-github-microsoft-didx509go_0.0.3-1_multi.changes ACCEPTED
into unstable

Thank you for your contribution to Debian.

Accepted:

Format: 1.8
Date: Sat, 09 Aug 2025 08:25:35 -0400
Binary: golang-github-microsoft-didx509go-dev
Source: golang-github-microsoft-didx509go
Architecture: all source
Version: 0.0.3-1
Distribution: unstable
Urgency: medium
Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
Changed-By: Reinhard Tartler <siretart@tauware.de>
Closes: 1110630
Description:
  golang-github-microsoft-didx509go-dev - Resolver for DID:x509 method
implemented in Go (library)
Changes:
  golang-github-microsoft-didx509go (0.0.3-1) unstable; urgency=medium
  .
    * Initial release (Closes: #1110630)
Checksums-Sha1:
  79318a2acdb3480d99ebe80e434ebd8c15a72e96 2513
golang-github-microsoft-didx509go_0.0.3-1.dsc
  9f045d5825a50b5b5a8db446b244b4f2f1054770 20803
golang-github-microsoft-didx509go_0.0.3.orig.tar.gz
  148622318d3ea0cf05eb051f963d19a7ce1c6f98 3864
golang-github-microsoft-didx509go_0.0.3-1.debian.tar.xz
  fa04190225a696d59903a9f4756809dc33f5e3b8 16816
golang-github-microsoft-didx509go-dev_0.0.3-1_all.deb
  a8299b1ca3f3a72634cbc56ed9369c48802a7324 6578
golang-github-microsoft-didx509go_0.0.3-1_amd64.buildinfo
Checksums-Sha256:
  8cb8020d0addb5c0b56537cc7cc7c37874fb7493dec616ee754122f870743a8c 2513
golang-github-microsoft-didx509go_0.0.3-1.dsc
  1b77016ee09e809ecf6f16694131c03fbf8caf7f56f95a04dd75d6002d4c33c7 20803
golang-github-microsoft-didx509go_0.0.3.orig.tar.gz
  e4786202fb0760b22312eb6b397455708b92b1924f3f56226ba8e4143adf18e3 3864
golang-github-microsoft-didx509go_0.0.3-1.debian.tar.xz
  38bd2ce56714a177a8df8e8656b533b904c1aadcdba1f1bc7876f302a703ee29 16816
golang-github-microsoft-didx509go-dev_0.0.3-1_all.deb
  8414316c70198cb047570e1132bf9bfc259316a041d9b8cba27d3374b5ae9ec4 6578
golang-github-microsoft-didx509go_0.0.3-1_amd64.buildinfo
Files:
  d59451630193f6ec74af39cb8c9d4963 2513 golang optional
golang-github-microsoft-didx509go_0.0.3-1.dsc
  be658bb10204015f000adf49951e7511 20803 golang optional
golang-github-microsoft-didx509go_0.0.3.orig.tar.gz
  54740bcb8227eafd7cb78c4e41937268 3864 golang optional
golang-github-microsoft-didx509go_0.0.3-1.debian.tar.xz
  167057c9797b71d91c599812bb9b6ab4 16816 golang optional
golang-github-microsoft-didx509go-dev_0.0.3-1_all.deb
  668bf7158d5a028c66fd355ea960df82 6578 golang optional
golang-github-microsoft-didx509go_0.0.3-1_amd64.buildinfo
-----BEGIN PGP SIGNATURE-----

iQJIBAEBCgAyFiEEMN59F2OrlFLH4IJQSadpd5QoJssFAmiXPuUUHHNpcmV0YXJ0
QHRhdXdhcmUuZGUACgkQSadpd5QoJsuv6g/+OXALRupiNiHOiubKha3tBSeMVWv0
lRRVoIYfw9/TLfuN1FdnS+OZB4QQQoQlYn+5p7HgZZOy/5PoGjxEHw/MwggbZA5N
t2HbXjZ5MU4rdAPf9eN1wIU0EcJRtISoXaHywS8y50mWg+F146uLGpiVjIyPnXJw
KT7pMayw7VZLh4lK3iViQ15zpOgNjSYCfJEYDkizWXmysHTP8sV2BbP3zQy2+nGi
9jRL/8N5DlXs+gzetkRcBCQpbd56FSrVde3L3/4FZCeIzpkEGzguRrRyx4IwKmV9
S4/BXH4SpDzeqVDFxHB3i5zuraw67XnFV/s9AnIrFJGQDnKe0LPi29E2yUptE2LG
Qck2bh7hxN99FNrjzWBwpL73XJMALdHfLGxFwFPE1Iboi1gUidhUNt+d+hC/2IHB
qwOXeEVKE/r/gh9d1b6pgBZu8pJlMQ7SOho8OTa9EE21jOgjAJ2QHoLp9R5qyJ+d
2aYS8v9LlJyxRCO+p99AWR69/UHt5f5s87fQRIcHL8IrwzA9felpbyqJzWjz7iaM
V2KL+zD0eHH2JvHMrJFQ4BbmW/gsoOEmrxFNaCQ6nVYndoUpOsqfGdaShzcu50E3
2Z/l/kl5LtOaamd8EOefOc959GeSunKTl1fGypEIyQTlGfPsKv2EpIgCS+GUyHUf
TrMMrjUhnrXoNTk=
=+8se
-----END PGP SIGNATURE-----


Forthermore, I have on the machine I used for the upload:


siretart@x1:/srv/scratch/packages/go-team/golang-github-microsoft-didx509go
$ cat .git/dgit/tag.signed.tmp
object 8a259de4d07ac3fc487b1e0c5ce7585e1f074097
type commit
tag archive/debian/0.0.3-1
tagger Reinhard Tartler <siretart@tauware.de> 1754742335 -0400

golang-github-microsoft-didx509go release 0.0.3-1 for unstable (sid)
[dgit]

[dgit distro=debian split --quilt=gbp]
-----BEGIN PGP SIGNATURE-----

iQJIBAABCgAyFiEEMN59F2OrlFLH4IJQSadpd5QoJssFAmiXPrgUHHNpcmV0YXJ0
QHRhdXdhcmUuZGUACgkQSadpd5QoJssJCg//VsVf5y3v58sprAi2axSGvXHehNMp
G+3r7B6W8Qi5Of/HHU4mct/BMRuP2CeQjFBjyy2xRhQoRYYM9oyeoN4K2PnMGMXb
qP8MgzK9FikJS5JiAZKjfFAK3GbyXc4/V0GeY5lQWm6m2cD/cU04rYRozw4ktHF6
6NkW7mN/rxCESLlp67agzCZrh4n7s3GpdC2gQrHOJD/j8+Kh2DSBBJAWMr31WANX
woFNyr66ht65pjqSsO+5Ie1wRTlPcv3I4IkEaePo7M5h1UF47jUk9LCw+TAYL1f0
z4P2CAQKBnHeNIzZcybCvfwY8CyhzyIgTHJvRhTsVJpt5C/0qR2Tv1t8H/vbFLVH
mi1sz7ZvvEHYwjSJ2ulAFwMFF9JWbGTYoUILOo5MMwHPgdQ/iCqmyv6DL9b+8862
vGZveYzljwVJqa41xapFKpFna5i0OSjzqS94ZhNX11t9+kkJGCme0WCVxgAZZsTO
fM6O6XvcA2d74aWfB9YLwcktaeHKeKDPd8P6Bfq0A+MHZg9laVmOmf8zvujM8Z53
wIN6s7GHY4/tK6/mIpg09mhv661YU6Q998aWfRta1sKEDqUnMLGMK8CZkx87UXtQ
nx2OsPI8EZFbfC8cBcwzSrU736PM+U507l5cc6sjw98Gm1gxG4O0xZQbT5roKxG4
wGyx4/vseaxATd8=
=KaZq
-----END PGP SIGNATURE-----

siretart@x1:/srv/scratch/packages/go-team/golang-github-microsoft-didx509go
$ git log -1
commit 34ef290c409daa0cc2aa92b4b7379535692186de (HEAD -> debian/sid,
tag: debian/0.0.3-1, origin/debian/sid)
Author: Reinhard Tartler <siretart@tauware.de>
Date:   Sat Aug 9 08:18:49 2025 -0400

     Update to jwx v2 API


This confirms that I did use dgit for the upload, and I don't recall any
issues with it.

Hope that helps!

#1127825#35
Date:
2026-02-13 16:08:08 UTC
From:
To:
Bug gardening:

Ian Jackson writes ("Re: Bug#1127673: dgit-repos-server: fatal: detected dubious ownership in repository at '$repos/_empty'"):
...

The desire to somehow avoid this happening again (likely, in two
years' time) is going to be bug #-2

    Erroneous taints when archive doesn't process queue for too long

The broken state on the server for these two packages is this bug
which I am therefore retitling

    #1127673
    golang-github-microsoft-didx509go and golang-github-microsoft-go-winio in bad state on dgit-repos


That the system is trying to access a bogus repository path will be #-3

    If repo doesn't exist, d-r-s can try to access '$repos/_empty'

Which I'm setting to "minor" because it doesn't seem to happen except
in quite strange situations where things have already gone wrong,
and the consequence is it bombing out rather than making anything worse.


Ian.