Suppose we have an ISO-8859 file, lets call it iso-8859.txt, containing those three lines below: first line something here á something else last line Notice that there is an accented letter (á, "a acute") in the second line. The command % tre-agrep first iso-8859.txt returns 'first line' (exit code = 0), which is correct. But both of the following commands % tre-agrep something iso-8859.txt % tre-agrep last iso-8859.txt return nothing (exit code = 1). For the sake of comparison, the standard 'grep' matches correctly all of those three commands above. The file iso-8859.txt is just a minimal working example to demonstrate this wrong behaviour of tre-agrep. The reported issue was found when I was running something like 'find . | tre-agrep "string"' on a directory (ext4 filesystem) containing file names with accented characters and I realized that something was wrong, i.e, some files having "string" in their names were never shown through tre-agrep. Related packages information: ii grep 2.6.3-3 GNU grep, egrep and fgrep ii findutils 4.4.2-1 utilities for finding files--find, xargs Shell: ii zsh 4.3.10-15 A shell with lots of features System information: Debian Release: squeeze/sid APT prefers unstable APT policy: (500, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Kernel: Linux 2.6.35-trunk-amd64 (SMP w/6 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Versions of packages tre-agrep depends on: ii libc6 2.11.2-6 Embedded GNU C Library: Shared lib ii libtre5 0.8.0-2 regexp matching library with appro
I forgot to say that tre-agrep works as expected if the presented 3-line example file is encoded in utf-8.
Hello. I received this from the Debian bug system: Package: tre-agrep Version: 0.8.0-2 Suppose we have an ISO-8859 file, lets call it iso-8859.txt, containing those three lines below: first line something here á something else last line Notice that there is an accented letter (á, "a acute") in the second line. The command % tre-agrep first iso-8859.txt returns 'first line' (exit code = 0), which is correct. But both of the following commands % tre-agrep something iso-8859.txt % tre-agrep last iso-8859.txt return nothing (exit code = 1). For the sake of comparison, the standard 'grep' matches correctly all of those three commands above. The file iso-8859.txt is just a minimal working example to demonstrate this wrong behaviour of tre-agrep. The reported issue was found when I was running something like 'find . | tre-agrep "string"' on a directory (ext4 filesystem) containing file names with accented characters and I realized that something was wrong, i.e, some files having "string" in their names were never shown through tre-agrep. --------------------------------------------------------------------------- Later, the submitter adds: --------------------------------------------------------------------------- I forgot to say that tre-agrep works as expected if the presented 3-line example file is encoded in utf-8. --------------------------------------------------------------------------- So: Is this a bug, or is the file supposed to be always in UTF-8? (Is this documented?) Thanks.
Santiago Vila <sanvila@unex.es> wrote: Hi, I think it is a bug because: 1) Standard 'grep' works just fine on the described examples 2) The behaviour also breaks the integration with other utilities, such as 'find' and 'ls', even in the UTF-8 environment. For instance, I've the following file names in a directory (attached you find the corresponding .tar archive containing those files): % ls (01) [Liszt] Annees de pelerinage Premiere annee Suisse 6 Vallee d'Obermann.flac (08) [Debussy] Images, Book2 1 Cloches ? travers les feuilles.flac (11) [Mozart] Fantasia in C minor, KV475.flac where the second file name contains a weird character (here shown as '?'). Then, the command % ls | tre-agrep flac returns just the first file name (the problematic one is the second file): (01) [Liszt] Annees de pelerinage Premiere annee Suisse 6 Vallee d'Obermann.flac while % ls | grep flac correctly returns all of them: (01) [Liszt] Annees de pelerinage Premiere annee Suisse 6 Vallee d'Obermann.flac (08) [Debussy] Images, Book2 1 Cloches � travers les feuilles.flac (11) [Mozart] Fantasia in C minor, KV475.flac The command % ls | tre-agrep feuilles returns nothing; neither does this one: % ls | tre-agrep KV475
Hi, One goal of tre-agrep is to be somewhat similar to GNU grep, so in that sense this a (significant) deviation can be considered a bug. What is actually happening is that you probably have LANG or LC_CTYPE or such set to UTF-8, and non-UTF-8 input is causing some code somewhere to error out. Invalid characters in the input should be skipped one by one, instead of skipping the rest of the input. I have no fix handy, and don't really have the time to look into this now. Sorry.