#566645 /usr/bin/uniq: uniq tells 2 lines with different invalid utf-8 characters are duplicate

Package:
coreutils
Source:
coreutils
Description:
GNU core utilities
Submitter:
Stephane Chazelas
Date:
2010-01-24 11:42:05 UTC
Severity:
normal
#566645#5
Date:
2010-01-24 11:31:40 UTC
From:
To:
~$ locale charmap
UTF-8
~$ locale collate-codeset
UTF-8
~$ sort .zsh-history|uniq -D|sed -n l
cd Pyr\202n\202es$
cd Pyr\351n\351es$


Both lines are identical except for the invalid UTF-8
characters, uniq reports them as identical.

"sort -u" and "comm" also treat them as identical:
~$ echo '\0300\n\0301' | sort -u | sed -n l
\300$
~$ sed -n l a
cd Pyr\202n\202es$
~$ sed -n l b
cd Pyr\351n\351es$
~$ comm -12 a b | sed -n l
cd Pyr\351n\351es$

If that's an expected behavior, I think it should be better
documented as I think "Comparisons honor the rules specified
by the `LC_COLLATE' locale category." is not enough to cover
that rather unintuitive behavior.
#566645#10
Date:
2010-01-24 11:39:19 UTC
From:
To:
Sorry,

the email address I used to submit the bug is incorrect. It
should have been stephane_chazelas@yahoo.fr.