Difference in utf-8 conversion for multibyte characters

Question

I have a text file containing some chinese charactes like 打印机驱动程序安装磁盘. By default the file is in ANSI code set. IF i save it as utf-8 in Textpad editor it is working fine, the binary values are proper, if i open the saved utf-8 file it is fine. But if i convert the original file to utf-8 using iconv, then the binary values are not same as the one which was saved in textpad, if i open the converted file then warning occurs that the characters does not exist in code page 936... this will be converted to system default character....

Why this difference between saving a text file in utf-8 format in textpad and converting the file using iconv?

Answer 1

ANSI

The ANSI character set ought to mean the character set defined by the American National Standards Institute (ANSI). However ANSI have defined many different character sets.

Microsoft and others sometimes incorrectly use the name "ANSI" to mean Code Page 1252 (CP-1252) also called Windows-1252 or Windows-Latin-1. This character set was not one of those defined by ANSI. This character set is similar to ISO-8859-1 but has many differences. Most importantly for this question This character set does not include any Chinese characters.

CP936

"Code page 936 is Microsoft's character encoding for simplified Chinese, one of the four DBCSs for East Asian languages. Originally it was identical to GB 2312, and expanded to cover most part of GBK with the release of Windows 95; now superseded by Code page 54936 (GB 18030)." -- Wikipedia

Iconv

If you ask iconv to convert from either MS-ANSI or ISO-8859-1 to UTF-8 it will not be able to interpret any of the data as Chinese characters as no such characters exist in MS-ANSI or ISO-8859-1.

You have to tell iconv the true encoding of your text file. If your text file is truly encoded in CP936 and iconv is told this, I would expect it to work.

TextPAD

See criticism of textpad

Microsoft

Microsoft's continuing misuse of ANSI's name is shameful and continues to cause it's customers a great deal of confusion and wasted time and money. As this question probably demonstrates.

Microsoft do say "The term "ANSI" as used to signify Windows code pages is a historical reference and a misnomer that persists in the Windows community. The source of this misnomer stems from the fact that the Windows code page 1252 was originally based on an ANSI draft, which became International Organization for Standardization (ISO) Standard 8859-1 [ISO/IEC-8859-1]. In Windows, the ANSI character set can be any of the following code pages: 1252, 1250, 1251, 1253, 1254, 1255, 1256, 1257, 1258, 874, 932, 936, 949, or 950."

Note the inclusion of CP-936 in that list.

Unfortunately, a lot of other web-pages at microsoft.com use the term ANSI incorrectly.