bug#6791: Problem(bug?) with basic sort command in Linux

[Bob Proulx]

forcemerge 6790 6791
tags 6790 + moreinfo
retitle 6790 locale sort ordering confusion
thanks

George Thomas Irimben (georgeti) wrote:
> Setting LC_ALL to C fixed the issue. 

Yes.  This is a well known behavior.  Thank you for the report anyway.
However what you are seeing is intended behavior.  It isn't something
sort has control over.  The character collation sequence is chosen by
your specified locale.  You can see what locale you have configured
with the 'locale' command.

  $ locale

> But before setting this to C, with same .cshrc file, Unix didn't give me
> a problem.

Your system probably didn't set the locale before and now it does.  Or
you were using a different system.  Or some such.  Definitely this is
a locale change.  Now that you know what to look for I am sure you
will locate the specific thing that changed.

> Does it mean my shell script which has this sort command will not work
> for others unless they set their LC_ALL?
> Should I set this to C in my script itself? 

Correct.  If your script requires a standard sort order then you will
need to ensure it yourself.  Because the environment it runs in may
default to a different locale sort ordering otherwise.

<rant> You don't like it and I don't like it but the-powers-that-be
have confused working with data on a computer with talking about
working with data on a computer.  They have decided that the collation
ordering (sort ordering) for data should be dictionary ordering.  In
dictionary ordering case is folded together and punctuation is
ignored.  For example by having LANG set to any of the "en_*" locales
the system is instructed to use dictionary sort ordering.  This
affects almost everything on the system that sorts.  This includes
commands such as 'ls' and also commands built into your shell
(e.g. 'echo *') too. </rant>

> Any other suggestion?

Your sort order depends upon your locale.  You didn't say what your
locale was and therefore I assume that you were not aware that it
had an effect.

The documentation says:

     Unless otherwise specified, all comparisons use the character
  collating sequence specified by the `LC_COLLATE' locale.(1)
  ...
     (1) If you use a non-POSIX locale (e.g., by setting `LC_ALL' to
  `en_US'), then `sort' may produce output that is sorted differently
  than you're accustomed to.  In that case, set the `LC_ALL'
  environment variable to `C'.  Note that setting only `LC_COLLATE'
  has two problems.  First, it is ineffective if `LC_ALL' is also set.
  Second, it has undefined behavior if `LC_CTYPE' (or `LANG', if
  `LC_CTYPE' is unset) is set to an incompatible value.  For example,
  you get undefined behavior if `LC_CTYPE' is `ja_JP.PCK' but
  `LC_COLLATE' is `en_US.UTF-8'.

As far as I know, which isn't as much as I would like especially in
this case, it is implemented in libc.  Therefore it would need to be
addressed with libc folks.

  http://www.gnu.org/software/libc/

But very likely the chain continues well beyond that point.

Personally I have the following in my $HOME/.bashrc file.

  export LANG=en_US.UTF-8
  export LC_COLLATE=C

That sets most of my locale to a UTF-8 one but forces sorting to be
standard C/POSIX.  This probably won't work in the general case since
I have no idea how that would interact with all character sets.

You may want to look at the FAQ.

  
http://www.gnu.org/software/coreutils/faq/#Sort-does-not-sort-in-normal-order_0021

Bob