UTF-8 is a character encoding that represents Unicode text as sequences of one to four bytes. It preserves the single-byte representation of every ASCII character while supporting the full range of Unicode scalar values, including characters from contemporary and historical writing systems, mathematical symbols, and emoji. The name refers to a Unicode Transformation Format whose code units are eight bits wide.
UTF-8 is the predominant encoding for text exchanged on the World Wide Web and is widely used in software, structured data formats, network protocols, and text files. Its importance rests on a combination of broad character coverage and practical engineering properties: ASCII compatibility, independence from byte order, recognizable character boundaries, and a unique valid encoding for each Unicode scalar value. These properties simplify interoperability, but they do not eliminate the need to distinguish bytes from characters, validate incoming data, or account for the linguistic structure of text.
Unicode characters, code points, and code units
UTF-8 is an encoding of Unicode, not an alternative character repertoire. Unicode assigns numerical positions called code points to characters and to several kinds of special-purpose values. Code points are conventionally written with the prefix U+, as in U+0041 for the Latin capital letter A, U+03B1 for the Greek small letter alpha, and U+1F600 for the grinning face emoji. UTF-8 specifies how appropriate values from this numerical space become bytes.
Unicode code points range from U+0000 through U+10FFFF, giving 1,114,112 possible positions. The range U+D800–U+DFFF is reserved for surrogate code points, which have a special role in UTF-16. A Unicode scalar value is any Unicode code point outside that surrogate range. There are therefore 1,112,064 possible scalar values, although many are unassigned to abstract characters. Standard UTF-8 encodes scalar values, not isolated surrogate code points.
A code unit is the basic storage unit of an encoding. In UTF-8, a code unit is an eight-bit byte, often called an octet in protocol specifications. One Unicode scalar value occupies between one and four such units. This distinguishes UTF-8 from UTF-16, whose code units are 16 bits wide, and UTF-32, whose code units are 32 bits wide. The number in each encoding’s name describes its code-unit size, not the number of bits required for every character.
Unicode also distinguishes the numerical identity of a character from its appearance. A font and a text-rendering system determine the visible glyphs, while language, context, and layout rules may affect how those glyphs are selected and positioned. UTF-8 does not specify fonts, handwriting styles, line breaking, or the direction in which text is displayed. It encodes the values that higher-level text systems interpret.
The relationship between code points and reader-perceived characters is likewise not necessarily one to one. A letter followed by a combining accent consists of multiple code points, and an emoji sequence may contain several independently encoded components. Conversely, a typographic ligature may display several letters using one glyph. Understanding UTF-8 consequently requires keeping four levels distinct: encoded bytes, Unicode scalar values, textual units perceived as characters, and rendered shapes.
Origins and standardization
UTF-8 emerged from the problem of introducing a universal character repertoire into software built around eight-bit bytes and ASCII. Earlier national and regional encodings could represent particular writing systems efficiently, but exchanging text between systems often required knowing which encoding had produced the bytes. The same byte value could denote different characters in different encodings, and some multibyte encodings allowed ASCII-valued bytes inside non-ASCII characters.
Ken Thompson and Rob Pike developed UTF-8 in September 1992 in connection with Plan 9, the research operating system developed at Bell Labs. The design drew on earlier work toward a file-system-safe transformation of the character set associated with ISO/IEC 10646. Its central practical objective was to support a much larger character repertoire without making ordinary ASCII text, file names, and common text-processing conventions incompatible with existing software.
Thompson and Pike’s paper “Hello World,” presented at the Winter 1993 USENIX conference, described Plan 9’s approach to international text. Plan 9 demonstrated that an operating system and its tools could use UTF-8 while retaining byte-oriented interfaces where appropriate. ASCII punctuation and path separators remained recognizable, while non-ASCII characters used byte values outside the ASCII range. The system nevertheless required changes to programs whose logic had assumed that one byte always meant one character.
Early UTF-8 specifications allowed sequences of up to six bytes, reflecting a larger numerical coding space contemplated in early work on universal character sets. RFC 2044, published in October 1996, documented that form; RFC 2279 replaced it in January 1998. RFC 3629, published in November 2003 through the Internet Engineering Task Force, restricted UTF-8 to the Unicode range ending at U+10FFFF and therefore to a maximum of four bytes. It also explicitly excluded surrogate code points.
Modern UTF-8 is specified within the Unicode Standard and ISO/IEC 10646, with RFC 3629 providing a central specification for Internet use. The historical restrictions matter when interpreting older descriptions: tables containing five- or six-byte UTF-8 sequences describe an obsolete form, not additional valid representations in the present encoding. Later Unicode versions assign new characters within the existing scalar-value space without changing the basic UTF-8 byte structure.
Byte patterns and encoding construction
UTF-8 partitions scalar values into four numerical ranges. Values in the ASCII range use one byte. Larger values use a leading byte that indicates the sequence length, followed by one, two, or three continuation bytes. Each continuation byte begins with the binary prefix 10; its remaining six bits contribute to the encoded value.
| Unicode scalar-value range | UTF-8 byte pattern | Length |
|---|---|---|
U+0000–U+007F |
0xxxxxxx |
1 byte |
U+0080–U+07FF |
110xxxxx 10xxxxxx |
2 bytes |
U+0800–U+FFFF, excluding surrogates |
1110xxxx 10xxxxxx 10xxxxxx |
3 bytes |
U+10000–U+10FFFF |
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx |
4 bytes |
The x positions carry the scalar value in binary, with the most significant payload bits first. The available payload capacities are 7, 11, 16, and 21 bits, respectively. Not every combination of the 21 payload bits is allowed, because Unicode ends at U+10FFFF; similarly, the three-byte form cannot encode surrogate values. Every scalar value must use the shortest form capable of representing it.
For example, the letter A, code point U+0041, is encoded as the single hexadecimal byte 41. The letter é, U+00E9, is encoded as C3 A9. The euro sign €, U+20AC, becomes E2 82 AC, and the grinning face 😀, U+1F600, becomes F0 9F 98 80. The text A€😀 therefore contains three scalar values but occupies eight bytes in UTF-8.
Encoding can be described using shifts and masks. For a scalar value requiring two bytes, the byte values are
where denotes a right shift, a bitwise AND, and a bitwise OR. Three- and four-byte sequences distribute the value into successive groups of six low-order payload bits, with the remaining high-order bits placed in the leading byte.
The encoded length can be expressed as
with surrogate values excluded from the function’s domain. The output depends only on the scalar value, not on the language of the text, the font, or the machine architecture.
Validity rules and decoding errors
Recognizing the broad leading-byte patterns is necessary but insufficient to validate UTF-8. A conforming decoder must also enforce the minimum value for each sequence length, exclude surrogates, reject values above U+10FFFF, and verify that every required continuation byte is present. These constraints give each scalar value exactly one valid UTF-8 representation.
The complete restrictions can be summarized by the permitted byte ranges below. All values are hexadecimal.
| Sequence length | First byte | Second byte | Remaining bytes |
|---|---|---|---|
| 1 | 00–7F |
— | — |
| 2 | C2–DF |
80–BF |
— |
| 3 | E0 |
A0–BF |
80–BF |
| 3 | E1–EC, EE–EF |
80–BF |
80–BF |
| 3 | ED |
80–9F |
80–BF |
| 4 | F0 |
90–BF |
80–BF twice |
| 4 | F1–F3 |
80–BF |
80–BF twice |
| 4 | F4 |
80–8F |
80–BF twice |
Bytes C0 and C1 cannot begin valid sequences because their possible results would be overlong representations of ASCII values. Bytes F5 through FF are also forbidden. A continuation byte encountered without an appropriate leading byte is invalid, as is an incomplete sequence at the end of a complete input. For example, C0 AF is not an alternative spelling of /, and ED A0 80 is not a valid encoding of a surrogate code point.
The encoding distinguishes malformed data from unusual but well-formed Unicode content. Unassigned scalar values, private-use values, and Unicode noncharacters can all have well-formed UTF-8 encodings. Their acceptability in a particular document or protocol is a separate question. A validator that rejects every unassigned code point is imposing a repertoire policy, not merely checking UTF-8 syntax, and may reject characters assigned by a later Unicode version.
Error handling depends on the surrounding specification. A decoder may report failure, stop at the first error, or substitute the replacement character U+FFFD. The Unicode Standard describes replacement based on maximal subparts of ill-formed sequences, making the treatment of malformed input more precise than a rule such as “replace each bad character.” Different recovery policies can otherwise produce different numbers of replacement characters.
Streaming introduces an additional distinction between invalid and incomplete input. A buffer ending with E2 82 may simply await its next byte; at a confirmed end of stream, it is malformed. A streaming decoder therefore retains a small amount of state across chunks. Following a valid prefix, at most three trailing bytes need to be held while awaiting completion of a four-byte sequence.
ASCII compatibility, synchronization, and bytewise operations
UTF-8 preserves ASCII exactly: code points U+0000–U+007F use bytes 00–7F, and those bytes never occur inside a multibyte sequence. This property allows software to recognize ASCII delimiters without accidentally finding them as components of other encoded characters. A slash byte remains a slash, a newline byte remains a newline, and the quote byte used by many textual syntaxes does not appear inside a non-ASCII character.
This compatibility is narrower than a claim that every ASCII-era program handles Unicode correctly. A program can safely locate an ASCII comma in valid UTF-8 yet still mishandle character counts, case conversion, cursor movement, or truncation. Programs using the C programming language also retain the usual issue that U+0000 is encoded as a zero byte and therefore terminates a conventional null-terminated string. UTF-8 does not remove that interface constraint.
UTF-8 is self-synchronizing because continuation bytes have a pattern distinct from both ASCII bytes and leading bytes. Given a position inside a valid sequence, a program can locate its beginning by moving backward over at most three continuation bytes. Moving forward to a non-continuation byte similarly identifies a candidate boundary. Validity checking remains necessary: encountering a boundary-shaped byte does not establish that the surrounding input is well formed.
A complete UTF-8 character encoding cannot be a proper prefix of another character’s encoding, a property related to prefix codes. Moreover, in well-formed UTF-8, an exact byte search for a well-formed encoded string cannot falsely match starting inside another character. The pattern’s first byte cannot be a continuation byte, and ASCII bytes are likewise excluded from multibyte interiors. These properties make ordinary byte-search algorithms useful for exact matching of encoded text.
Unsigned lexicographic comparison of well-formed UTF-8 preserves the numerical order of scalar values. Within each encoding length, the payload bits are arranged from most to least significant, and the successive length ranges also sort in scalar-value order. For strings, bytewise lexicographic order consequently agrees with lexicographic order over their scalar-value sequences, provided bytes are compared as unsigned values and ordinary prefix ordering is used.
Such ordering is not linguistic collation. Dictionary order may treat accents as secondary distinctions, handle digraphs specially, or order uppercase and lowercase letters together. UTF-8’s bytewise properties are useful for storage engines and exact comparisons, but they do not provide culturally appropriate sorting rules. Likewise, self-synchronization is not an error-correcting code: a corrupted byte may turn one valid character into another without producing any detectable encoding error.
Relationship to UTF-16, UTF-32, and legacy encodings
UTF-8, UTF-16, and UTF-32 represent the same Unicode scalar-value repertoire but distribute storage costs differently. UTF-16 uses one 16-bit code unit for most values in the Basic Multilingual Plane and a pair of code units for supplementary values. UTF-32 uses one 32-bit unit for each scalar value. UTF-8 instead uses one to four eight-bit units, with especially compact representation of ASCII text.
Consequently, ASCII-dominated text generally occupies less space in UTF-8 than in UTF-16 or UTF-32. Many European letters outside ASCII require two UTF-8 bytes, matching the two-byte size of a single UTF-16 code unit. Most commonly used Han characters require three bytes in UTF-8 and two bytes in UTF-16. Supplementary characters, including many emoji and historical-script characters, require four bytes in either UTF-8 or UTF-16.
For a string containing scalar values, uncompressed UTF-8 occupies between and bytes. This bound does not predict the size of a multilingual document without information about its contents. Markup, digits, spaces, punctuation, and programming-language syntax often contribute substantial ASCII content even when the natural-language portions use another script. Data compression further changes comparisons because repeated patterns and byte distributions affect compressed size.
UTF-8 has no little-endian and big-endian variants. Its byte sequence is fully specified, and storing it does not require choosing a byte order for multibyte code units. UTF-16 and UTF-32, when serialized as bytes, require a byte-order convention or an external indication of endianness. This difference concerns serialization; it does not mean that UTF-8 characters are numerically unordered or that processor byte order has no effect on implementation techniques.
Legacy encodings such as ISO-8859-1, Windows-1252, Shift JIS, and GB18030 have different mappings and structural rules. A byte sequence that happens to be valid UTF-8 is not necessarily intended as UTF-8, and ASCII-only text is compatible with many encodings simultaneously. Encoding detection therefore combines declarations, format rules, byte-pattern analysis, and sometimes statistical evidence; it is not generally a uniquely solvable deduction from arbitrary bytes.
Transcoding converts text between encodings by interpreting the source bytes and producing the destination representation. Converting valid Unicode text between UTF-8, UTF-16, and UTF-32 preserves its scalar-value sequence. Conversion into a legacy encoding can fail because the destination repertoire lacks a character. Merely changing an encoding label performs no conversion and is a common cause of mojibake, in which text is decoded under the wrong character mapping.
Grapheme clusters, normalization, and text semantics
The term “character” is ambiguous in software interfaces. A byte count, a Unicode scalar-value count, a UTF-16 code-unit count, and a count of user-perceived characters can all differ for the same text. UTF-8 specifies only the relation between scalar values and bytes. Higher-level operations require additional Unicode algorithms and, in some cases, language-specific rules.
A grapheme cluster approximates a user-perceived character for operations such as cursor movement and deletion. The text é can be represented as the single code point U+00E9, encoded as C3 A9, or as U+0065 followed by the combining acute accent U+0301, encoded as 65 CC 81. These forms commonly render alike but have different byte sequences and different scalar-value counts. Both normally form one extended grapheme cluster.
Emoji provide larger examples. A flag can be represented by a pair of regional-indicator code points, while some family or profession emoji use sequences containing a zero-width joiner. Variation selectors can request a particular presentation, and skin-tone modifiers can alter an emoji’s appearance. The resulting displayed unit may occupy many more than four UTF-8 bytes because the four-byte maximum applies to each scalar value, not to an entire grapheme cluster.
Unicode normalization defines standardized transformations among certain equivalent sequences. Normalization Form C, or NFC, performs canonical decomposition followed by canonical composition where possible. Normalization Form D, or NFD, uses canonical decomposition. NFKC and NFKD additionally apply compatibility mappings, which can remove distinctions such as those between some presentation forms and their ordinary counterparts. UTF-8 encoding by itself performs none of these transformations.
Normalization is relevant to equality, indexing, and interchange because canonically equivalent strings need not compare equal byte for byte. A system that normalizes text before comparison may regard the two representations of é as equivalent, whereas a raw byte comparison does not. Normalization also does not simply “make every character one code point”: many combining sequences have no single precomposed equivalent, and emoji sequences remain sequences.
Case conversion, word segmentation, line breaking, and regular-expression matching introduce further layers. The German letter ß, for example, illustrates why case-related operations need not preserve string length. A regular-expression engine may interpret its matching units as bytes, code points, or grapheme clusters depending on its design and mode. UTF-8 support therefore does not, by itself, establish that a text-processing system implements all relevant Unicode semantics.
Files, network protocols, and encoding declarations
UTF-8’s byte-order independence simplifies interchange, but a receiver still needs to know that the bytes are UTF-8. File formats and network protocols establish this through mandatory encoding rules, metadata, declarations, or signatures. The registered Internet charset label is UTF-8; the label is used in contexts such as media-type parameters to distinguish an encoding from the media type itself.
A UTF-8 file may begin with the three bytes EF BB BF, the encoding of U+FEFF. At the beginning of text, this can function as a byte order mark, commonly abbreviated BOM, although UTF-8 has no byte order to mark. Its practical role is an encoding signature. A BOM is not required for UTF-8 and is forbidden, ignored, or specially interpreted in some formats and protocols.
The BOM can also interfere with software that expects particular bytes at the start of a file. A Unix script may require its interpreter directive to begin with #!, while another format may require a fixed header. Software that fails to recognize a UTF-8 signature may expose it as an unexpected character or as mojibake. Whether to consume the signature is therefore a property of the enclosing format, not an unrestricted transformation to apply to all text.
On the Web, HTML supports encoding declarations such as <meta charset="utf-8">, while HTTP can communicate charset information in a Content-Type field. Browser processing includes detailed rules for reconciling transport metadata, byte signatures, and in-document declarations. UTF-8 is the standard choice for new HTML authoring, although web browsers retain decoders for numerous legacy encodings to support existing content.
Structured formats make different choices. RFC 8259 requires JSON exchanged between systems outside a closed ecosystem to use UTF-8. It also says that generators must not add a BOM to network-transmitted JSON, while parsers may ignore one for interoperability. XML supports encoding declarations and byte-based detection rules, and conforming XML processors must support UTF-8 and UTF-16. The same scalar-value content can therefore be serialized under different format-specific conventions.
Encoding validity and document validity remain separate. A byte stream may be valid UTF-8 but contain invalid JSON syntax, prohibited XML characters, or an unacceptable protocol field. Conversely, an encoding declaration cannot make malformed bytes valid. In email, UTF-8 text may additionally be wrapped in transfer encodings such as Base64 or quoted-printable; those mechanisms transport bytes and do not replace the underlying character encoding.
Programming interfaces, storage, and performance
Programming environments expose UTF-8 through different string models. Some store UTF-8 directly and distinguish byte indexing from character iteration. Others use UTF-16 internally and convert when reading or writing UTF-8. Still others present an abstract Unicode string interface whose internal representation varies. An application programming interface that accepts “a string” is therefore incomplete as a byte-level specification unless its encoding and length conventions are also defined.
Rust’s string types contain valid UTF-8 and expose byte lengths, while restricting slicing to valid character boundaries. Go strings, by contrast, are byte sequences and may contain arbitrary bytes; iteration with range decodes UTF-8 and reports malformed input through replacement-character behavior. JavaScript strings are sequences of UTF-16 code units, so their ordinary length property does not directly report either UTF-8 byte length or the number of Unicode scalar values.
Variable-width encoding affects indexing. Finding the -th scalar value in an unindexed UTF-8 string generally requires scanning the preceding encoding units, whereas accessing a known byte offset is direct. Libraries can add indexes, checkpoints, or other data structures to accelerate repeated positional access. UTF-32 offers direct scalar-position indexing, but even that does not provide constant-time access to the -th grapheme cluster without additional segmentation information.
Validation, transcoding, and scalar-value counting can be performed in linear time in the number of input bytes. Implementations often process long ASCII runs efficiently because every byte below 80 is immediately valid and represents one scalar value. Single instruction, multiple data techniques validate many bytes at once by checking leading-byte classes, continuation positions, and exceptional range restrictions using vector operations. The resulting performance depends on processor features, input composition, memory bandwidth, and error-handling requirements.
Storage systems must distinguish byte limits from character limits. A field allowing 100 bytes cannot necessarily hold 100 non-ASCII scalar values, while a field allowing 100 code points may require up to 400 UTF-8 bytes before other overheads. Index keys, fixed-size records, and network-message limits can therefore behave differently after conversion from a single-byte encoding. In a database, encoding is also separate from collation, which governs comparisons and ordering.
Operating-system interfaces impose another distinction. Linux file names are generally byte sequences subject to restrictions such as the exclusion of zero bytes and the path separator, rather than objects universally guaranteed to contain valid UTF-8. Windows widely uses UTF-16-based system interfaces. Cross-platform software consequently encounters both conversion requirements and paths that do not fit a simple model of universally well-formed Unicode text.
Security boundaries and nonstandard variants
UTF-8 validation has direct implications for cybersecurity because multiple components may interpret the same input differently. If a filter accepts one representation while a later decoder interprets it as a restricted character, the effective meaning can change across a trust boundary. Overlong sequences are a classic example: an obsolete or permissive decoder might interpret C0 AF as /, although modern UTF-8 rules reject it outright.
This problem is not limited to slash characters. Some historical encoders represented the zero value with C0 80, avoiding a literal zero byte. If one component treats those bytes as harmless text while another produces a null character, string termination and input validation can disagree. Modern standard UTF-8 permits only 00 for U+0000; accepting an overlong substitute introduces an interpretation not supported by the encoding.
Replacement decoding can also erase distinctions. Several different malformed byte sequences may become the same replacement-character text, so decoding with substitution is not generally reversible. Logging, authorization checks, digital signatures, and cache keys can diverge if some components operate on raw bytes while others operate on repaired text. Cryptographic signatures authenticate the bytes supplied to the signing algorithm; text normalization or error repair performed elsewhere can produce different bytes despite superficially similar display.
Several named formats resemble UTF-8 without being standard UTF-8. Java’s modified UTF-8, used in particular interfaces and file-format structures, encodes the zero value as C0 80 and represents supplementary characters through separately encoded UTF-16 surrogate code units. A supplementary character can therefore occupy six bytes in that format. The name does not imply that a conforming standard UTF-8 decoder must accept it.
CESU-8 similarly preserves UTF-16 code-unit structure by encoding surrogate pairs as two three-byte sequences. WTF-8 is a related convention designed to preserve potentially ill-formed UTF-16 data, including isolated surrogates, while remaining identical to UTF-8 for ordinary scalar-value text. Such formats address specific compatibility or round-trip requirements, but byte strings containing encoded surrogates are not well-formed standard UTF-8.
Finally, valid UTF-8 can contain deceptive text without any encoding error. Visually similar letters from different scripts, invisible formatting characters, and bidirectional controls can alter how an identifier or message is perceived. These are Unicode interpretation and presentation issues rather than failures of UTF-8 syntax. A byte-level validator can establish that a stream encodes scalar values correctly; it cannot establish that the displayed text is unambiguous, linguistically appropriate, or equivalent to another string.