Hi, I can't use rsync with curlftpfs. The problem is exactly as the following posts: rsync over ftp problem http://www.linuxforums.org/forum/ubuntu-help/140062-rsync-over-ftp-problem.html error using rsync with curlftpfs http://www.murga-linux.com/puppy/viewtopic.php?t=16902&sid=308d52fa0914e789e1620fc3c95f89e2 http://www.linuxmint-art.org/content/show.php?action=knowledgebase&content=94391&kbid=162&PHPSESSID=a4367411c755bfad272e2cd1f40c7f75 I.e., rsync: mkstemp "/dir/.file.pdf.0pkvjU" failed: Operation not supported (95) Then I tried the simplest one cp: seq 100 > /tmp/test.txt $ wc /tmp/test.txt 100 100 292 /tmp/test.txt curlftpfs -v -d ftp.host.net /mnt/share Then in another xterm: $ cd /mnt/share/test/ $ cp /tmp/test.txt . cp: closing `./test.txt': Input/output error I.e., the simplest method, cp, fails with curlftpfs as well. Detail log as follows: ------------------------------------------------------------ unique: 5, opcode: GETATTR (3), nodeid: 2, insize: 56 unique: 5, error: 0 (Success), outsize: 112 unique: 6, opcode: LOOKUP (1), nodeid: 2, insize: 49 LOOKUP /test/test.txt * Re-using existing connection! (#0) with host ftp.host.net * Connected to ftp.host.net (19x.16x.x.x) port 21 (#0) < 250 Directory successfully changed. < 250 Directory successfully changed. * Connect data stream passively < 227 Entering Passive Mode (19x,16x,x,x,41,121) * Trying 19x.16x.x.x... * connected * Connecting to 19x.16x.x.x (19x.16x.x.x) port 10617 < 150 Here comes the directory listing. * Maxdownload = -1 * Remembering we are in dir "test/" < 226 Directory send OK. * Connection #0 to host ftp.host.net left intact NODEID: 3 unique: 6, error: 0 (Success), outsize: 136 unique: 7, opcode: SETATTR (4), nodeid: 3, insize: 128 * Re-using existing connection! (#0) with host ftp.host.net * Connected to ftp.host.net (19x.16x.x.x) port 21 (#0) * Request has same path as previous transfer * Connect data stream passively < 227 Entering Passive Mode (19x,16x,x,x,167,69) * Trying 19x.16x.x.x... * connected * Connecting to 19x.16x.x.x (19x.16x.x.x) port 42821 < 200 Switching to Binary mode. < 150 Ok to send data. * Remembering we are in dir "test/" < 226 File receive OK. * Connection #0 to host ftp.host.net left intact * Re-using existing connection! (#0) with host ftp.host.net * Connected to ftp.host.net (19x.16x.x.x) port 21 (#0) * Request has same path as previous transfer * Connect data stream passively < 227 Entering Passive Mode (19x,16x,x,x,147,33) * Trying 19x.16x.x.x... * connected * Connecting to 19x.16x.x.x (19x.16x.x.x) port 37665 < 200 Switching to ASCII mode. < 150 Here comes the directory listing. * Maxdownload = -1 * Remembering we are in dir "test/" < 226 Directory send OK. * Connection #0 to host ftp.host.net left intact unique: 7, error: 0 (Success), outsize: 112 unique: 8, opcode: OPEN (14), nodeid: 3, insize: 48 unique: 8, error: 0 (Success), outsize: 32 OPEN[27768224] flags: 0x8001 /test/test.txt unique: 9, opcode: GETXATTR (22), nodeid: 3, insize: 68 unique: 9, error: -38 (Function not implemented), outsize: 16 unique: 10, opcode: WRITE (16), nodeid: 3, insize: 356 WRITE[27768224] 292 bytes to 0 * Re-using existing connection! (#0) with host ftp.host.net * Connected to ftp.host.net (19x.16x.x.x) port 21 (#0) * Request has same path as previous transfer * Connect data stream passively < 227 Entering Passive Mode (19x,16x,x,x,102,201) * Trying 19x.16x.x.x... * connected * Connecting to 19x.16x.x.x (19x.16x.x.x) port 26313 < 150 Here comes the directory listing. * Maxdownload = -1 * Remembering we are in dir "test/" < 226 Directory send OK. * Connection #0 to host ftp.host.net left intact * About to connect() to ftp.host.net port 21 (#0) * Trying 19x.16x.x.x... * connected * Connected to ftp.host.net (19x.16x.x.x) port 21 (#0) < 220 (vsFTPd 2.0.5) < 331 Please specify the password. < 230 Login successful. < 257 "/" * Entry path is '/' < 250 Directory successfully changed. * Connect data stream passively < 227 Entering Passive Mode (19x,16x,x,x,54,231) * Trying 19x.16x.x.x... * connected * Connecting to 19x.16x.x.x (19x.16x.x.x) port 14055 < 200 Switching to Binary mode. < 150 Ok to send data. WRITE[27768224] 292 bytes unique: 10, error: 0 (Success), outsize: 24 unique: 11, opcode: FLUSH (25), nodeid: 3, insize: 64 FLUSH[27768224] * Remembering we are in dir "test/" < 226 File receive OK. * Uploaded unaligned file size (292 out of 4294967295 bytes) * Connection #0 to host ftp.host.net left intact ------------------------------------------------------------ Thought cp fails, the file uploaded fine (292B fully uploaded). Please investigate. Thanks
clone 556012 -1
tags 556012 + patch
retitle 556012 Incorrect use of CURLOPT_INFILESIZE
reassign -1 libcurl3
retitle -1 CURLOPT_INFILESIZE does not handle correctly -1
thanks
Hi!
I get the same problem here. Enabling debug (with -o ftpfs_debug=3), I
get some additional information :
1260707694 ftpfs.c:534 write problem: 18(Transferred a partial file)
text=Uploaded unaligned file size (1072275 out of 4294967295 bytes)
1260707694 ftpfs.c:540 leaving streaming write thread #1 curl_res=18
1260707694 ftpfs.c:591 finish_write_thread after
pthread_join. write_fail_cause=18
< 221 Goodbye.
* Closing connection #0
ftpfs: operation ftpfs_flush failed because Input/output error
unique: 639, error: -5 (Input/output error), outsize: 16
I don't quite understand the "Uploaded unaligned file size". Moreover, I
don't understand why the size is 4294967295 bytes.
Here is the corresponding part in libcurl:
if(result || premature)
/* the response code from the transfer showed an error already so no
use checking further */
;
else if(data->set.upload) {
if((-1 != data->set.infilesize) &&
(data->set.infilesize != *ftp->bytecountp) &&
!data->set.crlf &&
(ftp->transfer == FTPTRANSFER_BODY)) {
failf(data, "Uploaded unaligned file size (%" FORMAT_OFF_T
" out of %" FORMAT_OFF_T " bytes)",
*ftp->bytecountp, data->set.infilesize);
result = CURLE_PARTIAL_FILE;
}
It seems that data->set.infilesize is incorrectly set 4294967295 but the
-1 != data->set.infilesize does not hold because data->set.infilesize is
of type curl_off_t which is long long which is 64bit. I suppose that
somewhere in curl, there is a unsigned long converted to long long.
Looking at the source code, this seems because we have this in ftpfs.c :
curl_easy_setopt_or_die(fh->write_conn, CURLOPT_INFILESIZE, -1);
While in curl, we have :
data->set.infilesize = va_arg(param, long);
Therefore, I think the bug is present in both curlftpfs and in libcurl.
I am not sure how to fix this correctly in libcurl, but I suppose that
we should read "va_arg(param, curl_off_t)" like for INFILESIZE_LARGE or
accurately handle -1 case. In curlftpfs, the best way is just to not set
CURLOPT_INFILESIZE to -1 since this seems not needed or use
CURLOPT_INFILESIZE_LARGE. Removing this line works fine for me.
--- curlftpfs-0.9.2/ftpfs.c.old 2009-12-13 14:12:25.000000000 +0100
+++ curlftpfs-0.9.2/ftpfs.c 2009-12-13 14:12:32.000000000 +0100
@@ -503,7 +503,6 @@
curl_easy_setopt_or_die(fh->write_conn, CURLOPT_URL, fh->full_path);
curl_easy_setopt_or_die(fh->write_conn, CURLOPT_UPLOAD, 1);
- curl_easy_setopt_or_die(fh->write_conn, CURLOPT_INFILESIZE, -1);
curl_easy_setopt_or_die(fh->write_conn, CURLOPT_READFUNCTION, write_data_bg);
curl_easy_setopt_or_die(fh->write_conn, CURLOPT_READDATA, fh);
curl_easy_setopt_or_die(fh->write_conn, CURLOPT_LOW_SPEED_LIMIT, 1);
severity 556012 important thanks OoO En ce début d'après-midi nuageux du dimanche 13 décembre 2009, vers 14:13, je disais: [...] Getting an error for any operation on 64bit arch seems an important bug to me. Therefore, I raise the severity accordingly. I also plan an delayed NMU to fix this bug in a week or two.
Dear maintainer, I've prepared an NMU for curlftpfs (versioned as 0.9.2-1.1) and uploaded it to DELAYED/10. Please feel free to tell me if I should delay it longer. Regards. diff -u curlftpfs-0.9.2/debian/rules curlftpfs-0.9.2/debian/rules --- curlftpfs-0.9.2/debian/rules +++ curlftpfs-0.9.2/debian/rules @@ -7,0 +8,2 @@ +include /usr/share/cdbs/1/rules/simple-patchsys.mk + diff -u curlftpfs-0.9.2/debian/changelog curlftpfs-0.9.2/debian/changelog --- curlftpfs-0.9.2/debian/changelog +++ curlftpfs-0.9.2/debian/changelog @@ -1,3 +1,12 @@ +curlftpfs (0.9.2-1.1) unstable; urgency=low + + * Non-maintainer upload. + * Use simple-patchsys for the patch below. + * Fix an incorrect use of CURLOPT_INFILESIZE option causing failures for + any file operation on amd64. Closes: #556012, #556013. + + -- Vincent Bernat <bernat@debian.org> Sun, 14 Feb 2010 15:26:09 +0100 + curlftpfs (0.9.2-1) unstable; urgency=low * New upstream release. (closes: #480320,#449207,#461455) only in patch2: unchanged: --- curlftpfs-0.9.2.orig/debian/patches/fix-CURLOPT_INFILESIZE.patch +++ curlftpfs-0.9.2/debian/patches/fix-CURLOPT_INFILESIZE.patch @@ -0,0 +1,11 @@ +CURLOPT_INFILESIZE does not support -1 arg. This fix bug #556012. +--- curlftpfs-0.9.2/ftpfs.c.old 2009-12-13 14:12:25.000000000 +0100 ++++ curlftpfs-0.9.2/ftpfs.c 2009-12-13 14:12:32.000000000 +0100 +@@ -503,7 +503,6 @@ + + curl_easy_setopt_or_die(fh->write_conn, CURLOPT_URL, fh->full_path); + curl_easy_setopt_or_die(fh->write_conn, CURLOPT_UPLOAD, 1); +- curl_easy_setopt_or_die(fh->write_conn, CURLOPT_INFILESIZE, -1); + curl_easy_setopt_or_die(fh->write_conn, CURLOPT_READFUNCTION, write_data_bg); + curl_easy_setopt_or_die(fh->write_conn, CURLOPT_READDATA, fh); + curl_easy_setopt_or_die(fh->write_conn, CURLOPT_LOW_SPEED_LIMIT, 1);
We believe that the bug you reported is fixed in the latest version of
curlftpfs, which is due to be installed in the Debian FTP archive:
curlftpfs_0.9.2-2.debian.tar.gz
to main/c/curlftpfs/curlftpfs_0.9.2-2.debian.tar.gz
curlftpfs_0.9.2-2.dsc
to main/c/curlftpfs/curlftpfs_0.9.2-2.dsc
curlftpfs_0.9.2-2_amd64.deb
to main/c/curlftpfs/curlftpfs_0.9.2-2_amd64.deb
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 556012@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Vincent Bernat <bernat@debian.org> (supplier of updated curlftpfs package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)
Format: 1.8
Date: Sun, 21 Feb 2010 13:05:13 +0100
Source: curlftpfs
Binary: curlftpfs
Architecture: source amd64
Version: 0.9.2-2
Distribution: unstable
Urgency: low
Maintainer: Vincent Bernat <bernat@debian.org>
Changed-By: Vincent Bernat <bernat@debian.org>
Description:
curlftpfs - filesystem to access FTP hosts based on FUSE and cURL
Closes: 556012 556013 570555 570605
Changes:
curlftpfs (0.9.2-2) unstable; urgency=low
.
* Adopt the package. Thanks to Ding Honghui for his previous work.
Closes: #570605.
* Acknowledge (delayed) NMU. Closes: #570555.
* Bump Standards-Version to 3.8.4.
* Switch Vcs-Browser to viewsvn.
* Switch to 3.0 (quilt) format.
* Update debian/copyright to new DEP5 format.
* Fix an incorrect use of CURLOPT_INFILESIZE option causing failures for
any file operation on amd64. Closes: #556012, #556013.
Checksums-Sha1:
aaeabbc638468e795e4fe9c7dcb76c188e19af8d 1291 curlftpfs_0.9.2-2.dsc
b6109a593a0d6079c10330860030628b7bebe667 3010 curlftpfs_0.9.2-2.debian.tar.gz
e38a6dbe0c35fe9e6960acf1ad04d67b1658dc59 34060 curlftpfs_0.9.2-2_amd64.deb
Checksums-Sha256:
6ba8b912cae01170b3e028dc66abbf1c2faba7dfece6e36e8c61d5ce73a5fd45 1291 curlftpfs_0.9.2-2.dsc
d73dfa4a896f78a5a6b7b87f335d8ba3e518d761bacd5aed2c1969c4cd781b01 3010 curlftpfs_0.9.2-2.debian.tar.gz
d314ae6d7a4431bd4e7b4fafca996d9aff9554129e9fa0faad3ef00a911dc1b4 34060 curlftpfs_0.9.2-2_amd64.deb
Files:
a570f17478f1d90defa29fdf057120ae 1291 utils optional curlftpfs_0.9.2-2.dsc
6d33e8683a65a37f4fa64b9fb8442f9b 3010 utils optional curlftpfs_0.9.2-2.debian.tar.gz
6d754381adc4c07cd735006918d01a4f 34060 utils optional curlftpfs_0.9.2-2_amd64.deb
iEYEARECAAYFAkuBIwYACgkQKFvXofIqeU5K5gCdHujdDBjSJyGgH1yIn1lrG1Ke
2r4An0iHiO+9qR6E7JxGVlwiSiBNkC79
=tftm
-----END PGP SIGNATURE-----
We believe that the bug you reported is fixed in the latest version of
curlftpfs, which is due to be installed in the Debian FTP archive:
curlftpfs_0.9.2-2.debian.tar.gz
to main/c/curlftpfs/curlftpfs_0.9.2-2.debian.tar.gz
curlftpfs_0.9.2-2.dsc
to main/c/curlftpfs/curlftpfs_0.9.2-2.dsc
curlftpfs_0.9.2-2_amd64.deb
to main/c/curlftpfs/curlftpfs_0.9.2-2_amd64.deb
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 556012@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Vincent Bernat <bernat@debian.org> (supplier of updated curlftpfs package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)
Format: 1.8
Date: Sun, 21 Feb 2010 13:05:13 +0100
Source: curlftpfs
Binary: curlftpfs
Architecture: source amd64
Version: 0.9.2-2
Distribution: unstable
Urgency: low
Maintainer: Vincent Bernat <bernat@debian.org>
Changed-By: Vincent Bernat <bernat@debian.org>
Description:
curlftpfs - filesystem to access FTP hosts based on FUSE and cURL
Closes: 556012 556013 570555 570605
Changes:
curlftpfs (0.9.2-2) unstable; urgency=low
.
* Adopt the package. Thanks to Ding Honghui for his previous work.
Closes: #570605.
* Acknowledge (delayed) NMU. Closes: #570555.
* Bump Standards-Version to 3.8.4.
* Switch Vcs-Browser to viewsvn.
* Switch to 3.0 (quilt) format.
* Update debian/copyright to new DEP5 format.
* Fix an incorrect use of CURLOPT_INFILESIZE option causing failures for
any file operation on amd64. Closes: #556012, #556013.
Checksums-Sha1:
aaeabbc638468e795e4fe9c7dcb76c188e19af8d 1291 curlftpfs_0.9.2-2.dsc
b6109a593a0d6079c10330860030628b7bebe667 3010 curlftpfs_0.9.2-2.debian.tar.gz
e38a6dbe0c35fe9e6960acf1ad04d67b1658dc59 34060 curlftpfs_0.9.2-2_amd64.deb
Checksums-Sha256:
6ba8b912cae01170b3e028dc66abbf1c2faba7dfece6e36e8c61d5ce73a5fd45 1291 curlftpfs_0.9.2-2.dsc
d73dfa4a896f78a5a6b7b87f335d8ba3e518d761bacd5aed2c1969c4cd781b01 3010 curlftpfs_0.9.2-2.debian.tar.gz
d314ae6d7a4431bd4e7b4fafca996d9aff9554129e9fa0faad3ef00a911dc1b4 34060 curlftpfs_0.9.2-2_amd64.deb
Files:
a570f17478f1d90defa29fdf057120ae 1291 utils optional curlftpfs_0.9.2-2.dsc
6d33e8683a65a37f4fa64b9fb8442f9b 3010 utils optional curlftpfs_0.9.2-2.debian.tar.gz
6d754381adc4c07cd735006918d01a4f 34060 utils optional curlftpfs_0.9.2-2_amd64.deb
iEYEARECAAYFAkuBIwYACgkQKFvXofIqeU5K5gCdHujdDBjSJyGgH1yIn1lrG1Ke
2r4An0iHiO+9qR6E7JxGVlwiSiBNkC79
=tftm
-----END PGP SIGNATURE-----
After upgrading to 0.9.2-2, rsync still stops with a message like rsync: mkstemp "/dir/.file.pdf.0pkvjU" failed: Operation not supported (95) then I tried mktemp: mktemp --tmpdir=test/ mktemp: failed to create file via template `test/tmp.XXXXXXXXXX': Operation not supported
found 556012 0.9.2-2 thanks OoO Lors de la soirée naissante du mardi 16 mars 2010, vers 17:14, Dietrich Clauss <dc2@clauss.dyndns.org> disait : The operation that is not supported is to open a file in read/write mode. In the code, there is a way to enable support for this mode that makes rsync works. However, I think that it is not enabled by default (and not easily enabled) for some reason. I will ask upstream about this.
this bug persists on 0.9.2-3, there's any way to workaround this? greetings, Lluís
OoO En cette matinée pluvieuse du lundi 23 mai 2011, vers 10:16, lluis <lluis@ingent.net> disait : Nothing was done for this bug. I got no answer from upstream.
0.9.1 seems not affected, downgrading worked for me for those who find it useful: wget http://ftp.de.debian.org/debian/pool/main/c/curlftpfs/curlftpfs_0.9.1-3 +b2_amd64.deb dpkg -i curlftpfs_0.9.1-3+b2_amd64.deb echo curlftpfs hold | dpkg --set-selections to upgrade curlftps again: echo curlftpfs install | dpkg --set-selections cheers, Lluís El dl 23 de 05 de 2011 a les 19:11 +0200, en/na Vincent Bernat va escriure:
tags 556012 + wontfix thanks OoO En ce début d'après-midi nuageux du mardi 24 mai 2011, vers 14:09, lluis <lluis@ingent.net> disait : Hi! I have looked again at this problem. In README, there is: ,---- | 1) Several GUI application (gedit, leafpad,...) use open(O_RDWR) + seek mode for saving files | which cannot be supported by the FTP protocol. Therefore saving files | might throw an error. Hopefully future kernels will provide special | errno's to make it easier to deal with less capable file-systems. `---- With such a warning, it seems dangerous to enable the possibility of opening files with O_RDWR. I tag the bug as wontfix. Sorry.
An easy workaround is to use a temp dir outside the ftp mount (with rsync --temp-dir=/tmp)