#740971 bash: completion fails on file names that contain spaces or other special characters

Package:
bash
Source:
bash
Description:
GNU Bourne Again SHell
Submitter:
Clément Calmels
Date:
2014-08-05 16:21:06 UTC
Severity:
normal
#740971#5
Date:
2014-03-06 19:16:24 UTC
From:
To:
Dear Maintainer,

The completion of files that contain spaces is broken:
$ touch a\ b1 a\ b2
$ ls a\ b

Press TAB: nothing happen.
The bash-completion package verions
ii  bash-completion              1:2.1-2

It's working with bash version 4.3-1.
I check in the completion script and found that compgen (bash-completion
package) does not behave the same depending on bash version.

Version 4.3-1:
$ compgen -f -X '' -- 'a\\\ '
a b2
a b1
$

Version 4.3-2:
$ compgen -f -X '' -- 'a\\\ '
$

I wrote this patch and it seems to correct this issue but I do not know if
this is the right approach.
$ diff -u /usr/share/bash-completion/bash_completion.bak
/usr/share/bash-completion/bash_completion
--- /usr/share/bash-completion/bash_completion.bak    2014-03-06
20:11:58.580659112 +0100
+++ /usr/share/bash-completion/bash_completion    2014-03-06
20:12:36.824761849 +0100
@@ -563,10 +563,9 @@
     _tilde "$cur" || return 0

     local -a toks
-    local quoted x tmp
+    local x tmp

-    _quote_readline_by_ref "$cur" quoted
-    x=$( compgen -d -- "$quoted" ) &&
+    x=$( compgen -d -- "$cur" ) &&
     while read -r tmp; do
         toks+=( "$tmp" )
     done <<< "$x"
@@ -575,7 +574,7 @@
         # Munge xspec to contain uppercase version too
         #
http://thread.gmane.org/gmane.comp.shells.bash.bugs/15294/focus=15306
         xspec=${1:+"!*.@($1|${1^^})"}
-        x=$( compgen -f -X "$xspec" -- $quoted ) &&
+        x=$( compgen -f -X "$xspec" -- "$cur" ) &&
         while read -r tmp; do
             toks+=( "$tmp" )
         done <<< "$x"
@@ -584,7 +583,7 @@
     # If the filter failed to produce anything, try without it if
configured to
     [[ -n ${COMP_FILEDIR_FALLBACK:-} && \
         -n "$1" && "$1" != -d && ${#toks[@]} -lt 1 ]] && \
-        x=$( compgen -f -- $quoted ) &&
+        x=$( compgen -f -- "$cur" ) &&
         while read -r tmp; do
             toks+=( "$tmp" )
         done <<< "$x"

#740971#10
Date:
2014-03-09 01:03:30 UTC
From:
To:
tags 740971 + confirmed
thanks

Thank you Clément.

I'm pretty sure I noticed this bug on February 26th, using bash 4.3~rc2-1. Note that another completion issue affects recent bash versions (see #733480).

To clarify, it's not that completion is completely broken, it just fails when several files share a start which contains at least one space.

#740971#17
Date:
2014-03-13 00:11:09 UTC
From:
To:
retitle 740971 bash: completion fails on file names that contain spaces or other special characters
thanks

Completion already fails on a single file name which contains a
space. Try:

  $ mkdir /tmp/test && cd /tmp/test && touch 'a a'
  $ ls <TAB><TAB>

Nothing happens ..

Completion also fails for file names which contain other special
characters which bash would escape with a backslash on
completion. Try the above example with file names like "a("
or "a,".

You can use single quotes to work around this bug:

  $ ls '<TAB><TAB>

In the example above this works as expected. Strangely it does
not work when you use double quotes instead.

Uwe

#740971#22
Date:
2014-03-13 01:37:33 UTC
From:
To:
A few corrections to my last mail:

bash completion before the first letter of the file name fails at
all, even for "normal" file names:

  $ mkdir /tmp/test && cd /tmp/test && touch a
  $ ls <TAB><TAB>

Also not all characters where the completion after the first
letter fails are characters which bash would escape with a
backslash during the completion as the comma actually does not
get escaped by bash.
Not sure what qualifies "special characters" in this context.
Characters where completion does not work for me (when file
names share the same beginning, as you have mentioned before)
are:

  !"$&'()*,:;<=>?[\]^`{|}

Interesting is also this example:

  $ mkdir /tmp/test && cd /tmp/test && touch aa 'a('
  $ ls a<TAB><TAB>
  a(  aa

So far it's ok. But:

  $ ls a(<TAB>

results in a wrong completion:

  $ ls a(a

Uwe

#740971#27
Date:
2014-03-13 03:44:54 UTC
From:
To:
Hi Uwe,

Chet Ramey has started work on a fix.

Thank you

#740971#32
Date:
2014-03-13 15:43:12 UTC
From:
To:
That message is a reflection of where things currently are.  It contains a
patch that will allow bash-completion to work unchanged.  (In a nutshell,
bash-completion makes assumptions about what compgen will do with quoted
null arguments that changed between bash-4.2 and bash-4.3.)

I think that patch will fix most, if not all, of the bash-completion
problems.  I need people to test it, since the platforms I commonly use
for development (Mac OS X, RHEL) do not install bash-completion.

For instance, I believe that the problem with these unquoted special
characters is that they break words for readline, and readline passes
an empty argument to the completion function as the word to be completed.
I'm not sure what bash-completion does with that to make it include the
previous word.

Chet

#740971#37
Date:
2014-03-18 05:56:26 UTC
From:
To:
merge 741920 740971
thanks

This is certainly a duplicate of #740971.

#740971#42
Date:
2014-03-20 20:19:21 UTC
From:
To:
I've updated the patch, and attached it.  This seems to fix all of the
reported problems from Debian's version of bash-completion.  One thing
I have yet to do is to verify that other systems that ship bash-completion
have the same issues.  If I cannot replicate the Debian-reported issues
on those systems, I will most likely not release this as an official
patch.

Chet

#740971#49
Date:
2014-03-27 20:57:31 UTC
From:
To:
It looks like this patch is included in Debian bash version
4.3-4. Most of my examples for bash completion which started to
fail with bash 4.3 are working again now. But one example still
fails, though I don't know if this is a problem of bash or
bash-completion:

  $ mkdir /tmp/test && cd /tmp/test && touch aa 'a('
  $ ls a<TAB><TAB>
  a(  aa

So far it's ok. But:

  $ ls a(<TAB>

results in a wrong completion:

  $ ls a(a

Uwe

#740971#54
Date:
2014-03-27 22:42:10 UTC
From:
To:
This is something different.  I addressed it in my first reply in this
thread:

"For instance, I believe that the problem with these unquoted special
characters is that they break words for readline, and readline passes
an empty argument to the completion function as the word to be completed."

The `(' is a character that breaks words for readline.  Readline passes
an empty string to the bash completion function, which completes
it to the longest common prefix of the possible completions.

There is a difference between bash-4.2 and bash-4.3 in how it treats
this word.  bash-4.3 does not interpret it the `(' as a command
delimiter (as it did in bash-4.2) because it's preceded by a character
that is not a command delimiter.  bash-4.2 incorrectly treated that
construct as valid and attempted to perform command completion; since
it's a syntax error, bash-4.3 doesn't.  There aren't any valid completions
for the empty string, so bash doesn't do anything and readline attempts
its default filename completion.  The longest common prefix of the
possible filename completions is "a", which readline inserts into the
line buffer.

Since that line, as you entered it above, is a syntax error, it's not
clear what bash should do with it (probably nothing).  What do you
expect to happen?

Chet

#740971#59
Date:
2014-03-27 23:11:34 UTC
From:
To:
The same as if you would try to complete any non-existing file
name, nothing? The completion should not append an "a" to the
typed in "a(". Instead the cursor should stay in the same
position (after the parenthesis) and bash should beep.

When you try to complete "ls ab" with <TAB> in the same example
with the two files "aa" and "a(" in a directory bash-completion
handles it correctly. It only fails because of the parenthesis
(and maybe also with other special characters).

Uwe

#740971#64
Date:
2014-03-28 00:11:04 UTC
From:
To:
Yes, if bash can detect a syntax error, that's probably the best
thing.

There is virtually nothing common between the two cases other than
the fact that they are two-character strings beginning with `a'.
This has nothing to do with bash-completion (maybe I should remove
the debian bug address from the recipients).

Chet

#740971#69
Date:
2014-08-05 16:19:03 UTC
From:
To:
Dear Maintainer, Chet,

as you explained (I am just rewording for my own sake) the issue about
"ls a(<TAB>" is somehow related to the fact that "(" means that the
following command is expected to be run in a _subshell_, and in fact
"(<TAB>" gives the same behavior. This and the fact that _all_ the files
in the current dir have the same prefix.

The user should do "ls a\(<TAB>" to tell bash that he is referring to
the "(" in the filename and not to the start-a-subshell character.

So this could even be considered the "correct" behavior, although not
the most straightforward one.

However I found another case when bash completion chokes, this is when
multiple special characters are one after another in a particular
sequence in the name of a directory containing files with a common
prefix.

In the case I found, the character pattern is for example:
one normal, two special, one normal, three special, one normal.

  $ mkdir /tmp/bash-test-ao2
  $ cd /tmp/bash-test-ao2
  $ mkdir a\ \ b\ \ \ c
  $ touch a\ \ b\ \ \ c/file{1,2}
  $ ls a<TAB><TAB>

The second <TAB> should complete up to:

  $ ls a\ \ b\ \ \ c/file

but it does not, it stops at:

  $ ls a\ \ b\ \ \ c/


As a counter example, completion works if the directory is named:
a\ \ b\ \ c
i.e.: one normal, two special, one normal, two special, one normal

BTW I did my tests invoking bash with "env -i bash --noprofile --norc"
in order to be sure not to add more variables (literally...) to the
problem.

Thanks,
   Antonio