Stuart Prescott writes ("Re: [tag2upload 2971] irrecoverable sasview 6.1.2-3"):

Hi.  Thanks for the report.  I'm sorry that this broke.  (I think this
is the first failure we've see at this point.)

Unfortunately, you'll need to bump the version and add a new changelog
stanza.  A retry with the same version will fail.

Sean, I think we should have dgit, at least in tag2upload builder
mode, retry this itself for a bit.

That probably applies to the git push, too.

Thanks,
Ian.

This bug #1129363 and #1125487 are both about making dgit push more
reliable, especially in the tag2upload builder.

#1125487 is the full-on version which should do a very good job of
retrying whenever it can, but it's a lot of work.

This bug #1129363 is about doing ad-hoc retries of the actual output
delivery commands (git push and dput).  I have a branch in the works
which does that.  I'll regard those changes as closing this bug.

Those change would probably not in fact have avoided the failure of
job 2971, because simply rerunning dput after a partial upload doesn't
work, when running in ftp mode against real Debian infra.

But I did some tests, and dput-ng seems to be able to resume (or
overwrite) a failed upload to real Debian infra, when running in sftp
mode.  So we should use that.  I'm cloning #1129363 as -2 for that.

We're already using dput-ng.  sftp mode would require enrolling the
builder's ssh key on the archive queued host, and changing the dput
config or target host (somewhere where it will affect the t2u service
but not normal user use of dgit).

Ian.

This is sent to two bugs:

 #1133774 [dput-ng] sftp method silently overwrites files
 #1130552 [dgit-infrastructure] want tag2upload to use dput-ng in sftp mode

tag2upload would like a way of calling dput that will DTRT if the
network craps out halway through and the same command is run again.
I think probably many humans would like that too :-).

I'm not sure precisely what scenario is being described as "uploads
were sometimes rejected because their .orig.tar had already been
removed" since the sftp method behaviour doesn't seem to involve
removing anything.

The whole dput (whether ftp or sftp) and queue daemon protocol is
rather janky.  I don't know if it's possible to achieve my "retries
work every time" goal.

Perhaps using sftp mode isn't the right answer for t2u but it's
difficult to see how to do better with anonftp.  It would be a shame
if changes to dput-ng made it harder rather than easier to make
tag2upload more reliable in the face of network trouble.

Sorry that I don't have good answers...

Ian.

The sftp method itself doesn't involve removing anything, but the system
as a whole does.

The target of sftp will typically be some kind of queue.  The thing
processing the queue typically removes the upload when it's finished.
If you have a shared queue directory (as I think most of these things
do, with the exception of Launchpad), then that means that things can go
wrong if you have two uploads in the queue at the same time that both
mention the same .orig.tar file in their .changes.  Having dput fail all
but one of the uploads in that situation seems strictly better to me,
since it allows outer systems to retry cleanly rather than apparently
succeeding and then resulting in emailed error messages later.

Having per-upload queue directories is strictly better for this purpose,
but it's also rather a lot of effort to retrofit, perhaps even
infeasible.  And historically it hasn't been entirely without its
downsides, since it normally means that people need to resume large
uploads from scratch.  (Of course, t2u offers a quite different solution
to that problem.)

I'm certainly not trying to make t2u's life harder.

With my proposal, would it be sufficient for t2u to just use `dput
--force`?  At the moment I believe that just has the effect of ignoring 
a pre-existing .upload file, which shouldn't make any difference for
t2u, but it would communicate the "no really, I want to overwrite the
previous attempt" intent to dput; and if a user needs to communicate the
same intent outside of t2u, then they very likely also have a .upload
file lying around, so piggybacking on the existing --force option seems
like the right thing to do.

Colin Watson writes ("Bug#1130552: Wanted: idempotent dput (but that may be impossible)"):

Ah, I see.

In principle a protocol that would work reliably would be:

 * Consumer doesn't delete .origs when it processes an upload
 * Instead, origs with "old enough" dates are deleted
 * Use some mode of sftp that overwrites files in place without
   truncating so that (i) timestamp is updated due to write() calls
   (ii) but file contents is not changed (since we write the same data)

Right.

Well, t2u doesn't actually *solve* this problem.  It moves the problem
from the user to the tag2upload service.  The t2u service has
hopefully-better connectivity, but when it does fail it is much more
annoying since you have to burn the version number.

One other possible option which would solve the problem just for t2u
would be for the tag2upload service to have a pet instance of the
queue daemon.  But really I think we ought to be able to get to a
situation where entities that need to do uploads can do so reliably
despite the fact that the public internet is not reliable.

For me the question with this is, what is special about tag2upload
that this option is correct ?

I think what we have here is a difference of likelihood, but not a
difference of fundamental principles.

Presumably debusine is troubled by #1133774 because it's normal with
debusine to make many similar uploads, perhaps even simultaneously.
Presumably they have different versions, but they can of course have
the same upstream version so the same origs.

Whereas tag2upload is troubled by the inability to retry the dput
because redoing the whole t2u processing is annoying and
labour-intensive, and the official archive isn't likely to see
concurrent or nearly-concurrent uploads of a package with the same
.orig.

(This orig stuff causes other kinds of races with tag2upload, too -
archive processing delays and the general nontransactional nature of
the archive can mean that the origs that the user intended is hard for
the t2u service to even find.  We have largely worked around these.)

If this analysis is right then maybe whether overwriting is a good
idea is a property of the upload target.  (If we can't do a *proper*
job without unreasonable effort, which I am quite prepared to
believe.)

Ian.

I came across
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=849251#15
in which Ansgar mentions dput's rsync mode and suggests it might be a
good option for another user.  I don't know whether upload.debian.org
supports rsync.  The expected rsync.upload.debian.org CNAME doesn't
exist.

Ian.