How can I get zsh to display international characters properly?

Question

I just started using zsh, and love it. However, I've stumbled upon an annoyance when it comes to international characters:

➜  ~  touch åäö.txt               
➜  ~  ls                          
Desktop       Dropbox       Music         Sites
Documents     Library       Pictures      a??a??o??.txt
Downloads     Movies        Public
➜  ~  rm -v a<030a>a<0308>o<0308>.txt
åäö.txt
➜  ~  

With bash it looks like this (the filename in rm -v is auto completed by pressing TAB in both cases).

johan@retina ~ $ touch åäö.txt
johan@retina ~ $ ls
Desktop       Dropbox       Music         Sites
Documents     Library       Pictures      åäö.txt
Downloads     Movies        Public
johan@retina ~ $ rm -v åäö.txt 
åäö.txt
johan@retina ~ $ 

How can I fix this with zsh?

EDIT:

Setting export LANG=en_US:UTF-8 fixes the output of e.g. ls and also shows it properly on the line below current input when there are multiple matches on TAB-completion. However, selecting the file from TAB-completion it shows the wrong way on the input line, the same goes for when there is only one match.

The above example now looks like this with zsh:

➜  ~  touch åäö.txt               
➜  ~  ls                          
åäö.txy
➜  ~  rm -v a<030a>a<0308>o<0308>.txt
åäö.txt
➜  ~  

If I have two files matching on TAB-completion it looks like this:

➜  ~  touch åäö.txt               
➜  ~  touch öäå.txt                          
➜  ~  rm 
öäå.txt    åäö.txt

Selecting one of the above by pressing TAB again and using arrow keys, or pressing either a or o to only make one match before completion generates this:

➜  ~  rm o<0308>a<0308>a<030a>.txt
➜  ~  rm a<030a>a<0308>o<0308>.txt

Any suggestions on what's wrong?

Answer 1

Thanks @mpy for solving the LANG problem. The answer is to use:

export LANG=en_US.UTF-8

in your .zshrc.

The remaining problem is caused by the completion system. Unfortunately completion is a monster feature. It involves shell functions or perhaps even scripts being called and somewhere in that process possibly LANG is again set to a wrong value. If you have root privileges you can debug this shell script code). Good luck with the completion guide.

Answer 2

Try

1. Having a Powerline compatible font installed https://github.com/powerline/fonts
2. Setting these ENV vars in .zshrc:

LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"

Answer 3

The problem is still the same on the last versions of zsh which come with Mac OS X 10.8 (aka Mountain Lion), 10.9 (aka Mavericks) & 10.10 (aka Yosemite) (I am still beta testing 10.11 and can't disclose information about it). The completion of zsh is failing.

The port version is working correctly at least with version 5.1.1:

/usr/bin/sudo port install zsh

Test:

/opt/local/bin/zsh

% touch hølé
% ls -l htab
→ ls -l hølé
% -rw-r--r-- 1 bob wheel 0 Apr 2 18:49 hølé
%

Answer 4

I had the same problem in Arch Linux using zsh.

Using bash everything works just fine, but when I switch to zsh some characters were displayed wrong (e.g. ñ,°).

I've added export LANG="en_US.UTF-8" to my .zshrc and nothing happened.

I did everything to set LANG inside zsh and nothing fixes.

Then I changed my shell back to bash with chsh -s /bin/bash and I noticed my env var LANG was wrong with printenv LANG it showed me LANG=C.

This is a new installation so I forget to create /etc/locale.conf file and set my LANG="en_US.UTF-8" and after restarting everything work perfect.

Hope this helps.