Byte order mark

The byte order mark (BOM) is a particular usage of the special Unicode character, U+FEFF BYTE ORDER MARK, whose appearance as a magic number at the start of a text stream can signal several things to a program reading the text:[1]

  • The byte order, or endianness, of the text stream in the cases of 16-bit and 32-bit encodings;
  • The fact that the text stream's encoding is Unicode, to a high level of confidence;
  • Which Unicode character encoding is used.
"FEFF" redirects here. For the airport in Central African Republic with the airport code FEFF, see Bangui M'Poko International Airport. For the program used in X-ray absorption spectroscopy, see FEFF (software). For the name of U+FEFF in Unicode and the alternative usage as a zero-width non-breaking space, see Word joiner.

BOM use is optional. Its presence interferes with the use of UTF-8 by software that does not expect non-ASCII bytes at the start of a file but that could otherwise handle the text stream.

Unicode can be encoded in units of 8-bit, 16-bit, or 32-bit integers. For the 16- and 32-bit representations, a computer receiving text from arbitrary sources needs to know which byte order the integers are encoded in. The BOM is encoded in the same scheme as the rest of the document and becomes a non-character Unicode code point if its bytes are swapped. Hence, the process accessing the text can examine these first few bytes to determine the endianness, without requiring some contract or metadata outside of the text stream itself. Generally the receiving computer will swap the bytes to its own endianness, if necessary, and would no longer need the BOM for processing.

The byte sequence of the BOM differs per Unicode encoding (including ones outside the Unicode standard such as UTF-7, see table below), and none of the sequences is likely to appear at the start of text streams stored in other encodings. Therefore, placing an encoded BOM at the start of a text stream can indicate that the text is Unicode and identify the encoding scheme used. This use of the BOM character is called a "Unicode signature".[2]

Usage

If the BOM character appears in the middle of a data stream, Unicode says it should be interpreted as a "zero-width non-breaking space" (inhibits line-breaking between word-glyphs). In Unicode 3.2, this usage is deprecated in favor of the "Word Joiner" character, U+2060.[1] This allows U+FEFF to be used only as a BOM.

UTF-8

The UTF-8 representation of the BOM is the (hexadecimal) byte sequence 0xEF,0xBB,0xBF.

The Unicode Standard permits the BOM in UTF-8,[3] but does not require or recommend its use.[4] Byte order has no meaning in UTF-8,[5] so its only use in UTF-8 is to signal at the start that the text stream is encoded in UTF-8, or that it was converted to UTF-8 from a stream that contained an optional BOM. The standard also does not recommend removing a BOM when it is there, so that round-tripping between encodings does not lose information, and so that code that relies on it continues to work.[6][7] The IETF recommends that if a protocol either (a) always uses UTF-8, or (b) has some other way to indicate what encoding is being used, then it "SHOULD forbid use of U+FEFF as a signature."[8]

Not using a BOM allows text to be backwards-compatible with some software that is not Unicode-aware. Examples include programming languages that permit non-ASCII bytes in string literals but not at the start of the file.

UTF-8 is a sparse encoding in the sense that a large fraction of possible byte combinations do not result in valid UTF-8 text. Binary data and text in any other encoding are likely to contain byte sequences that are invalid as UTF-8. Practically the only exceptions to that are when the text consists purely of ASCII-range bytes. Because all modern encodings use ASCII-range bytes to represent ASCII characters, ASCII-only text can be safely interpreted as UTF-8 regardless of what encoding was intended by the system that emitted the bytes. Because of these considerations, heuristic analysis can detect with high confidence whether UTF-8 is in use, without requiring a BOM.

Microsoft compilers[9] and interpreters, and many pieces of software on Microsoft Windows such as Notepad treat the BOM as a required magic number rather than use heuristics. These tools add a BOM when saving text as UTF-8, and cannot interpret UTF-8 unless the BOM is present or the file contains only ASCII. Windows PowerShell (up to 5.1) will add a BOM when it saves UTF-8 XML documents. However, PowerShell Core 6 has added a -Encoding switch on some cmdlets called utf8NoBOM so that document can be saved without BOM. Google Docs also adds a BOM when converting a document to a plain text file for download.

UTF-16

In UTF-16, a BOM (U+FEFF) may be placed as the first character of a file or character stream to indicate the endianness (byte order) of all the 16-bit code units of the file or stream. If an attempt is made to read this stream with the wrong endianness, the bytes will be swapped, thus delivering the character U+FFFE, which is defined by Unicode as a "non character" that should never appear in the text.

  • If the 16-bit units are represented in big-endian byte order, the BOM will appear in the sequence of bytes as 0xFE 0xFF
  • If the 16-bit units use little-endian order, the BOM will appear in the sequence of bytes as 0xFF 0xFE

Neither of these sequences is valid UTF-8, so their presence indicates that the file is not encoded in UTF-8.

For the IANA registered charsets UTF-16BE and UTF-16LE, a byte order mark should not be used because the names of these character sets already determine the byte order. If encountered anywhere in such a text stream, U+FEFF is to be interpreted as a "zero width no-break space".

If there is no BOM, it is possible to guess whether the text is UTF-16 and its byte order by searching for ASCII characters (i.e. a 0 byte adjacent to a byte in the 0x20-0x7E range, also 0x0A and 0x0D for CR and LF). A large number (i.e. far higher than random chance) in the same order is a very good indication of UTF-16 and whether the 0 is in the even or odd bytes indicates the byte order. However, this can result in both false positives and false negatives.

Clause D98 of conformance (section 3.10) of the Unicode standard states, "The UTF-16 encoding scheme may or may not begin with a BOM. However, when there is no BOM, and in the absence of a higher-level protocol, the byte order of the UTF-16 encoding scheme is big-endian." Whether or not a higher-level protocol is in force is open to interpretation. Files local to a computer for which the native byte ordering is little-endian, for example, might be argued to be encoded as UTF-16LE implicitly. Therefore, the presumption of big-endian is widely ignored. The W3C/WHATWG encoding standard used in HTML5 specifies that content labelled either "utf-16" or "utf-16le" are to be interpreted as little-endian "to deal with deployed content".[10] However, if a byte-order mark is present, then that BOM is to be treated as "more authoritative than anything else".[11]

Programs that interpret UTF-16 as a byte-based encoding may display a garbled mess of characters, but ASCII characters would be recognizable because the low byte of the UTF-16 representation is the same as the ASCII code and therefore would be displayed the same. The upper byte of 0 may be displayed as nothing, white space, a period, or some other unvarying glyph.

UTF-32

Although a BOM could be used with UTF-32, this encoding is rarely used for transmission. Otherwise the same rules as for UTF-16 are applicable.

The BOM for little-endian UTF-32 is the same pattern as a little-endian UTF-16 BOM followed by a NUL character, an unusual example of the BOM being the same pattern in two different encodings. Programmers using the BOM to identify the encoding will have to decide whether UTF-32 or a NUL first character is more likely.

Byte order marks by encoding

This table illustrates how the BOM character is represented as a byte sequence in various encodings and how those sequences might appear in a text editor that is interpreting each byte as a legacy encoding (CP1252 and caret notation for the C0 controls):

Encoding Representation (hexadecimal) Representation (decimal) Bytes as CP1252 characters
UTF-8[lower-alpha 1] EF BB BF 239 187 191 
UTF-16 (BE) FE FF 254 255 þÿ
UTF-16 (LE) FF FE 255 254 ÿþ
UTF-32 (BE) 00 00 FE FF 0 0 254 255 ^@^@þÿ (^@ is the null character)
UTF-32 (LE) FF FE 00 00 255 254 0 0 ÿþ^@^@ (^@ is the null character)
UTF-7[lower-alpha 1] 2B 2F 76 38
2B 2F 76 39
2B 2F 76 2B
2B 2F 76 2F[lower-alpha 2]
2B 2F 76 38 2D[lower-alpha 3]		 43 47 118 56
43 47 118 57
43 47 118 43
43 47 118 47
43 47 118 56 45		 +/v8
+/v9
+/v+
+/v/
+/v8-
UTF-1[lower-alpha 1] F7 64 4C 247 100 76 ÷dL
UTF-EBCDIC[lower-alpha 1] DD 73 66 73 221 115 102 115 Ýsfs
SCSU[lower-alpha 1] 0E FE FF[lower-alpha 4] 14 254 255 ^Nþÿ (^N is the "shift out" character)
BOCU-1[lower-alpha 1] FB EE 28 251 238 40 ûî(
GB-18030[lower-alpha 1] 84 31 95 33 132 49 149 51 „1•3
  1. This is not literally a "byte order" mark, since a code unit in these encodings is one byte and therefore cannot have bytes in a "wrong" order. Nevertheless, the BOM can be used to indicate the encoding of the text that follows it.[5][12]
  2. In UTF-7, the fourth byte of the BOM, before encoding as base64, is 001111xx in binary. The final two bits, xx, are not specifically part of the BOM, but contain the first two bits of the first encoded character following the BOM. All four possible byte combinations are shown in the table, as well as a fifth which is used for an empty string.
  3. If no following character is encoded, 38 is used for the fourth byte and the following byte is 2D.
  4. SCSU allows other encodings of U+FEFF, the shown form is the signature recommended in UTR #6.[13]
gollark: He doesn't ACTUALLY have those points, by the glorious RULE™.
gollark: bees™
gollark: Technically, it is only made to assist the glory of the RULES™, it is not the source of truth™.
gollark: I fear that we may need to use Procedure 206-Potato.
gollark: If you want chaos to be harbinged, you will need to make a proposal like everybody else.

See also

References
  1. "FAQ - UTF-8, UTF-16, UTF-32 & BOM". Unicode.org. Retrieved 2017-01-28.
  2. "The Unicode® Standard Version 9.0" (PDF). The Unicode Consortium.
  3. "The Unicode Standard 5.0, Chapter 2:General Structure" (PDF). p. 36. Retrieved 2009-03-29. Table 2-4. The Seven Unicode Encoding Schemes
  4. "The Unicode Standard 5.0, Chapter 2:General Structure" (PDF). p. 36. Retrieved 2008-11-30. Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature
  5. "FAQ - UTF-8, UTF-16, UTF-32 & BOM: Can a UTF-8 data stream contain the BOM character (in UTF-8 form)? If yes, then can I still assume the remaining UTF-8 bytes are in big-endian order?". Unicode.org. Retrieved 2009-01-04.
  6. "Re: pre-HTML5 and the BOM from Asmus Freytag on 2012-07-13 (Unicode Mail List Archive)". Unicode.org. Retrieved 2012-07-14.
  7. "Bug ID: JDK-6378911 UTF-8 decoder handling of byte-order mark has changed". Bugs.sun.com. Retrieved 2017-01-28.
  8. Yergeau, Francois (November 2003). UTF-8, a transformation format of ISO 10646. IETF. doi:10.17487/RFC3629. RFC 3629. Retrieved May 15, 2014.
  9. Alf P. Steinbach (2011). "Unicode part 1: Windows console i/o approaches". Retrieved 24 March 2012. However, since the C++ source code was encoded as UTF-8 without BOM (as is usual in Linux), the Visual C++ compiler erroneously assumed that the source code was encoded as Windows ANSI.
  10. "UTF-16LE". Encoding Standard. WHATWG.
  11. "Decode". Encoding Standard. WHATWG.
  12. "RFC 3629 - UTF-8, a transformation format of ISO 10646". Tools.ietf.org. 2003-11-08. Retrieved 2017-01-28.
  13. Markus Scherer. "UTS #6: Compression Scheme for Unicode". Unicode.org. Retrieved 2017-01-28.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.