In an integration involving multiple uploads of the same source package
from Debusine to reprepro at similar times, we found that dput-ng's sftp
method is apparently willing to silently (or at least without a hard
error) overwrite files on the destination, whereas the ftp method will
fail in that situation.  This meant that uploads were sometimes rejected
because their .orig.tar had already been removed.

The ftp method's behaviour seems to be an emergent property of typical
anonymous FTP setups rather than deliberate behaviour of the dput-ng
method, but nevertheless I think it would be more useful if the sftp
method behaved the same way.  It isn't 100% obvious how to do this in a
non-racy way with paramiko since the `put` method doesn't support
exclusive opens, but `put` is just a wrapper around some lower-level
methods and `open` supports "x" in the mode argument, so it should be
possible.

(It might make sense for --force to override this.  It's a bit of an
extension from how it's currently defined - "Force an upload, even if
the upload log exists already" - but it seems to me that you'd want to
override my proposed behaviour in the same sorts of situations.)

This is sent to two bugs:

 #1133774 [dput-ng] sftp method silently overwrites files
 #1130552 [dgit-infrastructure] want tag2upload to use dput-ng in sftp mode

tag2upload would like a way of calling dput that will DTRT if the
network craps out halway through and the same command is run again.
I think probably many humans would like that too :-).

I'm not sure precisely what scenario is being described as "uploads
were sometimes rejected because their .orig.tar had already been
removed" since the sftp method behaviour doesn't seem to involve
removing anything.

The whole dput (whether ftp or sftp) and queue daemon protocol is
rather janky.  I don't know if it's possible to achieve my "retries
work every time" goal.

Perhaps using sftp mode isn't the right answer for t2u but it's
difficult to see how to do better with anonftp.  It would be a shame
if changes to dput-ng made it harder rather than easier to make
tag2upload more reliable in the face of network trouble.

Sorry that I don't have good answers...

Ian.

The sftp method itself doesn't involve removing anything, but the system
as a whole does.

The target of sftp will typically be some kind of queue.  The thing
processing the queue typically removes the upload when it's finished.
If you have a shared queue directory (as I think most of these things
do, with the exception of Launchpad), then that means that things can go
wrong if you have two uploads in the queue at the same time that both
mention the same .orig.tar file in their .changes.  Having dput fail all
but one of the uploads in that situation seems strictly better to me,
since it allows outer systems to retry cleanly rather than apparently
succeeding and then resulting in emailed error messages later.

Having per-upload queue directories is strictly better for this purpose,
but it's also rather a lot of effort to retrofit, perhaps even
infeasible.  And historically it hasn't been entirely without its
downsides, since it normally means that people need to resume large
uploads from scratch.  (Of course, t2u offers a quite different solution
to that problem.)

I'm certainly not trying to make t2u's life harder.

With my proposal, would it be sufficient for t2u to just use `dput
--force`?  At the moment I believe that just has the effect of ignoring 
a pre-existing .upload file, which shouldn't make any difference for
t2u, but it would communicate the "no really, I want to overwrite the
previous attempt" intent to dput; and if a user needs to communicate the
same intent outside of t2u, then they very likely also have a .upload
file lying around, so piggybacking on the existing --force option seems
like the right thing to do.

Colin Watson writes ("Bug#1130552: Wanted: idempotent dput (but that may be impossible)"):

Ah, I see.

In principle a protocol that would work reliably would be:

 * Consumer doesn't delete .origs when it processes an upload
 * Instead, origs with "old enough" dates are deleted
 * Use some mode of sftp that overwrites files in place without
   truncating so that (i) timestamp is updated due to write() calls
   (ii) but file contents is not changed (since we write the same data)

Right.

Well, t2u doesn't actually *solve* this problem.  It moves the problem
from the user to the tag2upload service.  The t2u service has
hopefully-better connectivity, but when it does fail it is much more
annoying since you have to burn the version number.

One other possible option which would solve the problem just for t2u
would be for the tag2upload service to have a pet instance of the
queue daemon.  But really I think we ought to be able to get to a
situation where entities that need to do uploads can do so reliably
despite the fact that the public internet is not reliable.

For me the question with this is, what is special about tag2upload
that this option is correct ?

I think what we have here is a difference of likelihood, but not a
difference of fundamental principles.

Presumably debusine is troubled by #1133774 because it's normal with
debusine to make many similar uploads, perhaps even simultaneously.
Presumably they have different versions, but they can of course have
the same upstream version so the same origs.

Whereas tag2upload is troubled by the inability to retry the dput
because redoing the whole t2u processing is annoying and
labour-intensive, and the official archive isn't likely to see
concurrent or nearly-concurrent uploads of a package with the same
.orig.

(This orig stuff causes other kinds of races with tag2upload, too -
archive processing delays and the general nontransactional nature of
the archive can mean that the origs that the user intended is hard for
the t2u service to even find.  We have largely worked around these.)

If this analysis is right then maybe whether overwriting is a good
idea is a property of the upload target.  (If we can't do a *proper*
job without unreasonable effort, which I am quite prepared to
believe.)

Ian.