A byte order mark (BOM) is the character U+FEFF placed at the beginning of a Unicode text stream to indicate its encoding and, where relevant, the order of its bytes. Its encoded form provides a recognizable signature for software reading a file or receiving text. In UTF-16 and UTF-32, it can distinguish big-endian from little-endian representations. In UTF-8, which has no byte-order variants, it serves only as an encoding signature. Whether a BOM is required, permitted, or prohibited depends on the surrounding format or protocol.
Purpose and underlying mechanism
A character encoding specifies how characters are represented as numeric units and ultimately as bytes. Some Unicode encodings use code units wider than one byte. When those units are serialized, their bytes can be written in different orders:
- Big-endian: the most significant byte comes first.
- Little-endian: the least significant byte comes first.
This distinction is called endianness. It concerns the serialized representation of each code unit, not the direction in which a language is written or the order of characters in a string.
The numeric value U+FEFF was chosen so that its byte sequence reveals this ordering. In UTF-16, it becomes FE FF in big-endian order and FF FE in little-endian order. Interpreting the bytes in the wrong order yields U+FFFE, a permanently reserved Unicode noncharacter rather than an ordinary text character.
A BOM is an in-band signature: it travels in the same byte stream as the text, rather than being supplied separately through a filename, protocol header, or other metadata. When recognized as a signature, it is not part of the document’s textual content.
Byte sequences
The principal BOM representations are:
| Encoding | Initial bytes in hexadecimal | Function |
|---|---|---|
| UTF-8 | EF BB BF |
Identifies UTF-8; does not indicate byte order |
| UTF-16, big-endian | FE FF |
Indicates big-endian UTF-16 |
| UTF-16, little-endian | FF FE |
Indicates little-endian UTF-16 |
| UTF-32, big-endian | 00 00 FE FF |
Indicates big-endian UTF-32 |
| UTF-32, little-endian | FF FE 00 00 |
Indicates little-endian UTF-32 |
For example, a UTF-16 file containing the single letter A can have either of these representations:
FE FF 00 41 Big-endian UTF-16 with BOM
FF FE 41 00 Little-endian UTF-16 with BOM
The letter has the same Unicode value in both cases; only its serialization differs.
A detector supporting several encodings must account for overlapping signatures. In particular, the UTF-16 little-endian signature is also the beginning of the UTF-32 little-endian signature. Checking only the first two bytes is therefore insufficient to distinguish them. More generally, signature recognition operates within the expectations of a file format or protocol; arbitrary binary data can begin with the same bytes.
Use in different Unicode encodings
UTF-16
UTF-16 represents text using 16-bit code units. Characters outside the Basic Multilingual Plane require pairs of code units, but byte ordering applies to each individual unit.
A BOM allows a reader to determine byte order when that order has not already been specified. Its absence does not make all UTF-16 text invalid: an enclosing format, protocol, or explicitly selected decoder may establish the order instead.
The encoding labels UTF-16, UTF-16BE, and UTF-16LE are not interchangeable. The explicitly endian labels establish byte order without requiring detection. The applicable specification determines whether an initial U+FEFF is permitted and whether it is treated as a signature or content. Rules for generic UTF-16 must not automatically be applied to every explicitly endian-labeled stream.
UTF-32
UTF-32 uses one 32-bit code unit for each Unicode scalar value. Its BOM performs the same byte-order function as in UTF-16, but occupies four bytes.
UTF-32 is less common as an interchange encoding than UTF-8. Nevertheless, the distinction between an unspecified byte order and an explicitly endian representation remains relevant.
UTF-8
UTF-8 is defined as an ordered sequence of bytes, so there is no big-endian or little-endian UTF-8. Its optional BOM, EF BB BF, indicates only that the stream is intended to be read as UTF-8.
This signature can help software distinguish UTF-8 from legacy encodings when no other information is available. However, many applications identify UTF-8 through a format requirement, an external declaration, or validation of the byte stream, making the BOM unnecessary.
UTF-8 preserves ASCII byte values for ASCII characters. A BOM adds three non-ASCII bytes before otherwise ASCII-only content, which can affect software that expects a particular character sequence at byte offset zero.
U+FEFF as a character
U+FEFF has the Unicode character name ZERO WIDTH NO-BREAK SPACE. Historically, it served both as an encoding signature at the beginning of a stream and as an invisible character preventing a line break within text.
Unicode deprecated its use for the latter purpose in favor of U+2060, WORD JOINER, introduced in Unicode 3.2. U+FEFF continues to serve as the byte order mark.
Position and context matter:
- At the beginning of an encoded stream, U+FEFF may function as a signature.
- Within text, it is not a byte-order instruction.
- An initial U+FEFF is not automatically removable in every context; the governing format determines whether it represents a signature or actual content.
Consequently, removing every U+FEFF from a document is not equivalent to removing its BOM.
Formats and protocols
BOM handling is part of a format’s encoding rules, not a universal property of text files.
In HTML, encoding detection recognizes Unicode BOMs, and a recognized BOM can take precedence over other encoding declarations. XML likewise defines encoding-detection rules involving initial byte patterns, BOMs, and encoding declarations.
For JSON exchanged outside a closed ecosystem, the standard requires UTF-8. Generators must not add a BOM to network-transmitted JSON, although parsers may ignore one rather than reject the input.
A web browser, text editor, compiler, and command-line utility may therefore handle the same initial bytes differently because they implement different formats and decoding rules. A BOM-aware text editor may hide the signature entirely, while a byte-oriented utility sees ordinary leading bytes.
Interoperability problems
Unexpected leading content
If software decodes a BOM as text instead of consuming it as a signature, the resulting string begins with an invisible U+FEFF. This can cause a field name, identifier, or header to differ from its visible spelling.
For example, a table’s first column name may be decoded as \uFEFFname rather than name. A lookup for name then fails even though both strings appear identical on screen.
Misdecoded signatures
When a UTF-8 BOM is interpreted using Windows-1252, its bytes appear as . This is an instance of mojibake: text displayed through an inappropriate decoding. The visible characters are not the BOM’s intended representation.
Position-sensitive syntax
Some tools require a particular byte sequence at the very beginning of a file. On many Unix-like systems, executable scripts use an initial #! sequence to identify an interpreter. A preceding UTF-8 BOM can prevent recognition of that sequence.
Similar problems arise in parsers and file-type detectors that do not explicitly allow a BOM before their expected opening syntax.
Concatenation and byte-level identity
Concatenating independently BOM-prefixed files places additional signatures inside the combined stream. These internal sequences do not establish a new byte order for the surrounding text and may appear as embedded U+FEFF characters.
Adding or removing a BOM also changes the file’s bytes even when its displayed text remains unchanged. It therefore changes byte offsets and inputs to a cryptographic hash function, and can invalidate a digital signature over the original bytes. Textual equivalence and byte-for-byte identity are separate properties.
Detection and limitations
A BOM provides a strong indication of an encoding when it appears at the expected stream boundary and the applicable format recognizes it. It does not, by itself, prove that the remainder of the stream is well-formed Unicode, identify the language of the text, or establish the integrity of the content.
Conversely, the absence of a BOM does not imply that text is non-Unicode. UTF-8 commonly appears without one, while UTF-16 and UTF-32 can rely on externally specified byte order.
BOM detection and decoding are therefore separate operations: the signature helps select or confirm an encoding, after which the remaining bytes must still be interpreted and validated under that encoding’s rules.