The sort behaviour depends deeply on the LC_COLLATE settings.
See e.g. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=884943

This behaviour is not documented in the primary documentation
provided by `sort --help` or `man sort`.

This sometimes weird behaviour is documented only in a footnote of
https://www.gnu.org/software/coreutils/manual/html_node/sort-invocation
.html and in a FAQ page https://www.gnu.org/software/coreutils/faq/core
utils-faq.html#Sort-does-not-sort-in-normal-order_0021.


It would be an improvement if a „clean“ „simple byte comparison“ would
be provided via a new option `--machine-sort`.

This would provide a much clearer „API“ and intuitive documentation.

H.-Dirk Schmitt wrote:

But it is mentioned in all three places.

  $ sort --help
  *** WARNING ***
  The locale specified by the environment affects sort order.
  Set LC_ALL=C to get the traditional sort order that uses
  native byte values.

  $ man sort

   *** WARNING *** The locale specified by the environment affects sort
   order.  Set LC_ALL=C to get the traditional sort order that uses native
   byte values.

And also as you mention in the info pages (also on the web) and FAQ:

https://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html

  If you use a non-POSIX locale (e.g., by setting LC_ALL to ‘en_US’),
  then sort may produce output that is sorted differently than you’re
  accustomed to. In that case, set the LC_ALL environment variable to
  ‘C’. Note that setting only LC_COLLATE has two problems. First, it is
  ineffective if LC_ALL is also set. Second, it has undefined behavior
  if LC_CTYPE (or LANG, if LC_CTYPE is unset) is set to an incompatible
  value. For example, you get undefined behavior if LC_CTYPE is
  ja_JP.PCK but LC_COLLATE is en_US.UTF-8.

https://www.gnu.org/software/coreutils/faq/#Sort-does-not-sort-in-normal-order_0021

I know you just burned a lot of time getting to the point where you
now know what is happening.  I can only say that I have been there
myself.  We have all been there.

But the problem with Unix documentation is that there are many details
that one needs to know and those all need to be known everywhere.
This makes it IMPOSSIBLE to document in any concise way.  I know you
are thinking, but this one thing, this thing, this one thing needs to
be written in bold letters at the very top.  An arm needs to reach out
of the program and grab you and shake you and make sure you know that
you need to set LC_ALL=C to get a standard sort order.

Well, that is fine.  But it is only one little detail out of the
zillion details.  For sort people often get confused when working with
keys (-k).  For them details there are THE MOST IMPORTANT and need to
have an arm reach out of the program and grab them and tell them that
they need to know to use both starting and ending key fields.  And so
on.

What would result is documentation that is completely unreadable
because every utility would have 2,000 pages of documentation and then
people would be filing bugs that it is too long, the details are lost
in the noise.  Really!

Really in the end all that can be done is to methodically and
concisely do the best that can be done.  At this time the locale
problem is mentioned as a *** WARNING *** in all of the documentation
and the online help.

Really what you and I would like is to remove UTF-8 and return to the
days of US-ASCII so that this character collation problem would no
longer be a problem.

But wait...  I see that in your mail you are using many UTF-8
characters and would probably not be happy using only a US-ASCII
character set.  So we need UTF-8 and at that point we are stuck with
this character collation problem.  :-)

Bob

P.S. Personally I have the following in my ~/.profile file.  YMMV.
And regardless I always set LC_ALL=C in any script where I want
standard behavior.

LANG=en_US.UTF-8
LC_COLLATE=C
export LANG LC_COLLATE