#264895 mget: does not get "index.html" with glob pattern (dir/*) via http

Package:
lftp
Source:
lftp
Description:
Sophisticated command-line FTP/HTTP/BitTorrent client programs
Submitter:
Jeff Sheinberg
Date:
2013-09-12 17:39:21 UTC
Severity:
normal
#264895#5
Date:
2004-08-10 20:18:31 UTC
From:
To:
Package: lftp
Version: 3.0.6-1
Severity: normal

Hi,

It appears that "lftp" does not download a file named "index.html"
using the "mget" command with a glob pattern of "dir/*", via the
"http" protocol.  Using the "ftp" protocol, such a file *is*
downloaded correctly.

This failure to download a globbed "index.html" file happens with
both Apache/1.3.26 and boa/0.94.13 web servers.  I discovered this
problem trying to download the debian-installer manual, when I
noticed that 67 files had been downloaded, when I had expected to
get 68 files.

Here is a typescript that demonstrates this problem,

    pwd=> jeff.bsrd.net/
    lftp> !mkdir -p pub/manual/en

    pwd=> jeff.bsrd.net/
    lftp> open http://jeff.bsrd.net
    ---- Connecting to jeff.bsrd.net (192.168.1.22) port 80
    cd ok, cwd=/

    pwd=> jeff.bsrd.net/
    lftp> mget -c -d pub/manual/en/*
    ---- Connecting to jeff.bsrd.net (192.168.1.22) port 80
    450693 bytes transferred in 1 second (414.0K/s)
    Total 67 files transferred

    pwd=> jeff.bsrd.net/
    **** Peer closed connection
    lftp> close

    pwd=> jeff.bsrd.net/
    lftp> open ftp://jeff.bsrd.net

    pwd=> jeff.bsrd.net/
    lftp> mget -c -d pub/manual/en/*
    ---- Connecting to jeff.bsrd.net (192.168.1.22) port 21
    <--- 220 Welcome to vsFTPd service.
    <--- 230 Login successful.
    18698 bytes transferred
    Total 68 files transferred
    pwd=> jeff.bsrd.net/
    lftp> close


Thanks,
#264895#16
Date:
2012-07-07 16:18:03 UTC
From:
To:
tags 264895 + upstream confirmed
found 264895 4.3.8-1
forwarded 264895 lftp@uniyar.ac.ru
thanks

Hello,

I can confirm the described problem. Needed some time to find a public
URL for this test:

$ lftp http://ftp.uni-koeln.de
cd ok, cwd=/
lftp ftp.uni-koeln.de:/> mget -c -d fedora/14/Fedora/x86_64/os/repoview/*
1266611 bytes transferred in 81 seconds (15.2K/s)
Total 206 files transferred

$ lftp ftp://ftp.uni-koeln.de
lftp ftp.uni-koeln.de:~> mget -c -d fedora/14/Fedora/x86_64/os/repoview/*
11272788 bytes transferred in 2218 seconds (5.0K/s)
Total 2942 files transferred

The big different of transferred files is clearly based on the content
of the index.html.
The problem is that when using http the index.html is not saved to disk
(its downloaded to get the filenames).
#264895#21
Date:
2013-06-11 11:50:25 UTC
From:
To:
found 264895 4.4.8-1
forwarded 264895 http://univ.uniyar.ac.ru/pipermail/lftp/2012-July/000299.html
forwarded 264895 https://github.com/lavv17/lftp/issues/44
thanks

still reproducible with the same URLs.
reported to lftp github issues.
#264895#32
Date:
2013-08-07 11:08:33 UTC
From:
To:
tags 264895 + wontfix
thanks

Hello,

answer from upstream author (see forwarded link):

"lftp does not "know" that index.html exists if the directory index does
not contain a link to it. I don't think lftp should save a directory
index to index.html, because it can be contained in a different file or
generated dynamically."