{
  "id": "byte-order-mark",
  "title": "Byte Order Mark",
  "lang": "en",
  "source_lang": "en",
  "summary": "A byte order mark is an initial Unicode signature that can identify a text encoding and, for UTF-16 and UTF-32, its byte order.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "concept",
  "aliases": [
    "BOM",
    "Unicode signature",
    "encoding signature"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/byte-order-mark/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/byte-order-mark/",
      "json": "https://aiwiki.page/zh/byte-order-mark.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A byte order mark (BOM) is the character U+FEFF placed at the beginning of a Unicode text stream to indicate its encoding and, where relevant, the order of its bytes. Its encoded form provides a recognizable signature for software reading a file or receiving text. In UTF-16 and UTF-32, it can distinguish big-endian from little-endian representations. In UTF-8, which has no byte-order variants, it serves only as an encoding signature. Whether a BOM is required, permitted, or prohibited depends on the surrounding format or protocol."
    },
    {
      "heading": "Purpose and underlying mechanism",
      "text": "A character encoding specifies how characters are represented as numeric units and ultimately as bytes. Some Unicode encodings use code units wider than one byte. When those units are serialized, their bytes can be written in different orders:\n\n- Big-endian: the most significant byte comes first.\n- Little-endian: the least significant byte comes first.\n\nThis distinction is called endianness. It concerns the serialized representation of each code unit, not the direction in which a language is written or the order of characters in a string.\n\nThe numeric value U+FEFF was chosen so that its byte sequence reveals this ordering. In UTF-16, it becomes FE FF in big-endian order and FF FE in little-endian order. Interpreting the bytes in the wrong order yields U+FFFE, a permanently reserved Unicode noncharacter rather than an ordinary text character.\n\nA BOM is an in-band signature: it travels in the same byte stream as the text, rather than being supplied separately through a filename, protocol header, or other metadata. When recognized as a signature, it is not part of the document’s textual content."
    },
    {
      "heading": "Byte sequences",
      "text": "The principal BOM representations are:\n\nEncoding | Initial bytes in hexadecimal | Function\nUTF-8 | EF BB BF | Identifies UTF-8; does not indicate byte order\nUTF-16, big-endian | FE FF | Indicates big-endian UTF-16\nUTF-16, little-endian | FF FE | Indicates little-endian UTF-16\nUTF-32, big-endian | 00 00 FE FF | Indicates big-endian UTF-32\nUTF-32, little-endian | FF FE 00 00 | Indicates little-endian UTF-32\n\nFor example, a UTF-16 file containing the single letter A can have either of these representations:\n\nFE FF 00 41    Big-endian UTF-16 with BOM\nFF FE 41 00    Little-endian UTF-16 with BOM\n\nThe letter has the same Unicode value in both cases; only its serialization differs.\n\nA detector supporting several encodings must account for overlapping signatures. In particular, the UTF-16 little-endian signature is also the beginning of the UTF-32 little-endian signature. Checking only the first two bytes is therefore insufficient to distinguish them. More generally, signature recognition operates within the expectations of a file format or protocol; arbitrary binary data can begin with the same bytes."
    },
    {
      "heading": "Use in different Unicode encodings",
      "text": "UTF-16\n\nUTF-16 represents text using 16-bit code units. Characters outside the Basic Multilingual Plane require pairs of code units, but byte ordering applies to each individual unit.\n\nA BOM allows a reader to determine byte order when that order has not already been specified. Its absence does not make all UTF-16 text invalid: an enclosing format, protocol, or explicitly selected decoder may establish the order instead.\n\nThe encoding labels UTF-16, UTF-16BE, and UTF-16LE are not interchangeable. The explicitly endian labels establish byte order without requiring detection. The applicable specification determines whether an initial U+FEFF is permitted and whether it is treated as a signature or content. Rules for generic UTF-16 must not automatically be applied to every explicitly endian-labeled stream.\n\nUTF-32\n\nUTF-32 uses one 32-bit code unit for each Unicode scalar value. Its BOM performs the same byte-order function as in UTF-16, but occupies four bytes.\n\nUTF-32 is less common as an interchange encoding than UTF-8. Nevertheless, the distinction between an unspecified byte order and an explicitly endian representation remains relevant.\n\nUTF-8\n\nUTF-8 is defined as an ordered sequence of bytes, so there is no big-endian or little-endian UTF-8. Its optional BOM, EF BB BF, indicates only that the stream is intended to be read as UTF-8.\n\nThis signature can help software distinguish UTF-8 from legacy encodings when no other information is available. However, many applications identify UTF-8 through a format requirement, an external declaration, or validation of the byte stream, making the BOM unnecessary.\n\nUTF-8 preserves ASCII byte values for ASCII characters. A BOM adds three non-ASCII bytes before otherwise ASCII-only content, which can affect software that expects a particular character sequence at byte offset zero."
    },
    {
      "heading": "U+FEFF as a character",
      "text": "U+FEFF has the Unicode character name ZERO WIDTH NO-BREAK SPACE. Historically, it served both as an encoding signature at the beginning of a stream and as an invisible character preventing a line break within text.\n\nUnicode deprecated its use for the latter purpose in favor of U+2060, WORD JOINER, introduced in Unicode 3.2. U+FEFF continues to serve as the byte order mark.\n\nPosition and context matter:\n\n- At the beginning of an encoded stream, U+FEFF may function as a signature.\n- Within text, it is not a byte-order instruction.\n- An initial U+FEFF is not automatically removable in every context; the governing format determines whether it represents a signature or actual content.\n\nConsequently, removing every U+FEFF from a document is not equivalent to removing its BOM."
    },
    {
      "heading": "Formats and protocols",
      "text": "BOM handling is part of a format’s encoding rules, not a universal property of text files.\n\nIn HTML, encoding detection recognizes Unicode BOMs, and a recognized BOM can take precedence over other encoding declarations. XML likewise defines encoding-detection rules involving initial byte patterns, BOMs, and encoding declarations.\n\nFor JSON exchanged outside a closed ecosystem, the standard requires UTF-8. Generators must not add a BOM to network-transmitted JSON, although parsers may ignore one rather than reject the input.\n\nA web browser, text editor, compiler, and command-line utility may therefore handle the same initial bytes differently because they implement different formats and decoding rules. A BOM-aware text editor may hide the signature entirely, while a byte-oriented utility sees ordinary leading bytes."
    },
    {
      "heading": "Interoperability problems",
      "text": "Unexpected leading content\n\nIf software decodes a BOM as text instead of consuming it as a signature, the resulting string begins with an invisible U+FEFF. This can cause a field name, identifier, or header to differ from its visible spelling.\n\nFor example, a table’s first column name may be decoded as \\uFEFFname rather than name. A lookup for name then fails even though both strings appear identical on screen.\n\nMisdecoded signatures\n\nWhen a UTF-8 BOM is interpreted using Windows-1252, its bytes appear as ï»¿. This is an instance of mojibake: text displayed through an inappropriate decoding. The visible characters are not the BOM’s intended representation.\n\nPosition-sensitive syntax\n\nSome tools require a particular byte sequence at the very beginning of a file. On many Unix-like systems, executable scripts use an initial #! sequence to identify an interpreter. A preceding UTF-8 BOM can prevent recognition of that sequence.\n\nSimilar problems arise in parsers and file-type detectors that do not explicitly allow a BOM before their expected opening syntax.\n\nConcatenation and byte-level identity\n\nConcatenating independently BOM-prefixed files places additional signatures inside the combined stream. These internal sequences do not establish a new byte order for the surrounding text and may appear as embedded U+FEFF characters.\n\nAdding or removing a BOM also changes the file’s bytes even when its displayed text remains unchanged. It therefore changes byte offsets and inputs to a cryptographic hash function, and can invalidate a digital signature over the original bytes. Textual equivalence and byte-for-byte identity are separate properties."
    },
    {
      "heading": "Detection and limitations",
      "text": "A BOM provides a strong indication of an encoding when it appears at the expected stream boundary and the applicable format recognizes it. It does not, by itself, prove that the remainder of the stream is well-formed Unicode, identify the language of the text, or establish the integrity of the content.\n\nConversely, the absence of a BOM does not imply that text is non-Unicode. UTF-8 commonly appears without one, while UTF-16 and UTF-32 can rely on externally specified byte order.\n\nBOM detection and decoding are therefore separate operations: the signature helps select or confirm an encoding, after which the remaining bytes must still be interpreted and validated under that encoding’s rules."
    }
  ],
  "links": [
    "unicode",
    "utf-sixteen",
    "utf-eight",
    "character-encoding",
    "byte",
    "metadata",
    "ascii",
    "html",
    "web-browser",
    "compiler",
    "unix",
    "cryptographic-hash-function",
    "digital-signature"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}