In utf-8 collation, why 11- is less then 1-?

Question

I found that the sort result in ASCII:

Source file test:

1-
11-
1-a
11-a

Sort using ASCII:

$ LANG=en_US.ascii sort test
1-
1-a
11-
11-a

And using UTF-8:

$ LANG=en_US.utf8 sort test
1-
11-
11-a
1-a

I feel it's so counter-intuitive, and it's not dictionary order.

Isn't the character '-' (002d) is always less then [0-9] (0030-0039)? What's the general rule in UTF-8 collation?

And how to bypass it, just make - be less then [0-9] while keep other characters unchanged for UTF-8, in Linux? (So it can affects the result of ls --sort, sort, etc. )

Answer 1

The minus sign is ignored in the first pass. So the first pass sorts 1, 11, 1a, 11a. Since 1 < a, you get 11a < 1a and thus 11-a < 1-a.

- is a variable collation element, meaning that you/the implementor can choose to ignore it. The glibc implementation apparently does so. In practice, most punctuation is affected by this behavior.

You can read up on the gory details in the Unicode Collation Algorithm, modulo how glibc implements it.

Answer 2

As explained by Peter Eisentraut, this is because the sorting algorithm for Unicode ignores - when sorting.

The only way around this is to define your own locale, with different collation (sorting rules). This is however rather non-trivial. Also, it would give you a system with unusual sorting rules, which may cause problems with other software.

So realistically, you'll either have to switch your locale to ASCII (if you don't need Unicode character), or sort using a program where you can configure the sorting rules directly.