Adding/displaying special characters in Vim

Question

I currently work on a project where I need to write the Unicode characters ɛ and ħ, which I just added to my keyboard.

It works great almost everywhere, even in the command prompt, except for Vim who displays a question mark ? instead of ɛ and h instead of ħ.

The encoding is set to utf-8 and using the :digraphs isn't much of a help since even there, a lot of characters are shown simply as ?.

What can I do to fix this ?

EDIT
I use the Consolas font which has both the aforementioned characters.

Answer 1

I work in debian and had the issue you describe. To fix it you need to be sure you have the locale you require installed on your operating system in debian bash this is how to fix it:

1. check your locale

   locale -a
   

   in bash this reported: locale: Cannot set LC_VARIABLE to default locale: No such file or directory

2. generate the required locale so step 1 no longer complains.

   you can do that with the locale-gen command if you know what locale you require

   sudo locale-gen "en_US.UTF-8"
   

   or just reconfigure locale by issuing the following command

   sudo dpkg-reconfigure locales
   

If done correctly : digraphs should show the proper characters now in vim

https://wiki.debian.org/Locale
http://vim.wikia.com/wiki/Entering_special_characters

Answer 2

From the comment questions, even though the question states "encoding is set to utf-8", it appears Vim's encoding is NOT set to utf-8. It is still latin1 (ISO-8859-1). This means Vim does not know how to represent those special characters in memory, nor save them to a file.

For Vim to correctly write "special" characters, it needs 2 things:

1. An 'encoding' setting that can represent that character in memory, usually utf-8
2. An empty value, or a setting that can represent that character on-disk for 'fileencoding'. Usually this is also utf-8, or empty.

To properly read a file containing these characters, Vim again needs these two things. Vim can automatically detect the proper value for 'fileencoding', if your 'fileencodings' option is set properly.

'fileencodings' is a comma-separated list that tells Vim (in order) which encodings to try. "ucs-bom" as in the comments on your question will detect unicode, but only if there is a BOM (byte-order mark). Additionally since your Vim's 'encoding' is still "latin1" it wouldn't do you any good anyway if Vim DID detect unicode.

Note both 'encoding' and 'fileencodings' should be very close to the top of your .vimrc, before ANY mappings, menus, string options, or register settings. Otherwise existing text in those areas will be misinterpreted by Vim.

See the help for each of these options, and the wiki page on configuring Vim for Unicode, for details.

If Vim does't detect encoding properly on some files even with the correct settings, you can either:

1. Set it manually, like :e ++enc=utf-8 myfile.txt
2. Install a Vim plugin that will automatically detect file encoding