Character encoding is a system for representing text as numerical values that a computer can store, process, and transmit. It connects abstract characters—such as letters, digits, punctuation marks, and control characters—to representations built from bits and bytes. An encoding determines, for example, whether the letter “A” is represented by the byte 41 in hexadecimal, and how a character such as “é” or “中” is represented. Without an agreed encoding, the same bytes can be interpreted as different text.
Character encoding is foundational to digital communication because text crosses boundaries between devices, applications, languages, and historical computing systems. Early encodings were designed for restricted alphabets and particular equipment; later standards accommodated national writing systems, and Unicode established a common character repertoire for multilingual text. Correct text handling nevertheless requires more than choosing Unicode: it also involves serialization, encoding declarations, conversion, validation, and distinctions between characters, visible symbols, and units of storage.
Characters, code points, and encoded data
A character is an abstract element used in representing written information. It is not necessarily a complete visible symbol, nor is it identical to a particular printed shape. The Latin letter “A” remains the same encoded character across many typefaces, while Latin “A” and the visually similar Cyrillic “А” are different characters. A glyph is a visual form used to display a character or a combination of characters. Rendering systems select and position glyphs after the encoded text has been interpreted.
A character repertoire specifies which abstract characters a system recognizes. A coded character set assigns numerical positions, called code points, to characters in that repertoire. Unicode conventionally writes a code point using U+ followed by hexadecimal digits: Latin capital letter A is U+0041, and the euro sign is U+20AC. A code point is a number within the code space; not every possible code point necessarily has an assigned character. These distinctions matter because adding a character to a standard is different from inventing a new way of storing existing character numbers.
A character encoding form maps characters or code points to sequences of code units. A code unit is the basic numerical unit used by that form: eight bits in UTF-8, sixteen bits in UTF-16, and thirty-two bits in UTF-32. An encoding scheme additionally specifies how code units become bytes, including byte order where necessary. Everyday usage often compresses these layers into the single word “encoding.” Thus “UTF-16” may refer to a code-unit representation inside a program or to serialized data whose byte order must be established separately.
Encoding converts text into an encoded representation; decoding interprets that representation as text. For a supported string , a correctly paired encoder and decoder normally satisfy
This property depends on the repertoire and error policy. Encoding a Chinese character into a Western European single-byte encoding cannot preserve it if that encoding has no corresponding character. Conversely, arbitrary bytes need not be valid input for a particular decoder. Character encoding is therefore neither encryption, which protects information using a cryptographic transformation, nor data compression, which seeks a more compact representation. Its primary purpose is to establish an interoperable representation of text.
Telegraph codes and early computer standards
Digital character representation predates electronic computers. Nineteenth-century telegraph systems assigned signals to letters and operational commands. Morse code used variable-length combinations of short and long signals, while Émile Baudot’s nineteenth-century telegraph code used five-unit combinations. International Telegraph Alphabet No. 2, derived from later developments in this tradition, became widely associated with teleprinters. Five binary positions provide only combinations, so shift signals switched between sets containing letters and figures. A received pattern could consequently have different meanings depending on the current shift state.
Early computing inherited the constraints of telegraphy, punched cards, printers, and available hardware. Character representations often favored uppercase letters, digits, and a limited collection of punctuation. Word sizes and input equipment influenced how characters were packed into memory. Six-bit codes, for example, could represent 64 values and fitted conveniently into several historical machine-word formats. These arrangements were useful within a controlled installation but were poorly suited to exchanging unrestricted text among different manufacturers.
ASCII, first published in 1963 and substantially revised in 1967, established a seven-bit code with 128 positions. Its layout includes control characters, space, punctuation, digits, uppercase letters, and lowercase letters. Decimal 65, hexadecimal 41, represents “A”; decimal 97, hexadecimal 61, represents “a.” Control characters such as carriage return, line feed, and escape reflected the operation of terminals and communications equipment. Their legacy remains visible in line-ending conventions and terminal-control protocols. An eight-bit byte could store an ASCII character while leaving one bit available for purposes such as parity or later extensions.
ASCII did not become the only important computer code. IBM introduced EBCDIC, the Extended Binary Coded Decimal Interchange Code, with the System/360 family in 1964. Its organization differed from ASCII, including the placement of alphabetic characters. EBCDIC subsequently developed into a family of code pages and remains relevant in mainframe environments. Neither ASCII nor EBCDIC was simply a set of font choices: exchanging bytes between them required character conversion.
International standardization also exposed the limits of small code spaces. ISO 646 provided a seven-bit framework with national variants that replaced selected characters with locally needed symbols. A byte interpreted as a punctuation mark in one variant could represent an accented letter in another. Such compromises supported local use but complicated international exchange, helping motivate larger repertoires and more explicit identification of encodings.
Legacy encoding families and regional requirements
An eight-bit encoding can assign at most 256 distinct byte values. Many legacy systems preserved ASCII in positions 00–7F and used the upper half for additional characters. The ISO 8859 family followed this general pattern, with different parts covering different script requirements. ISO 8859-1, commonly called Latin-1, supported many Western European languages but did not include every character later needed in those languages, notably the euro sign. Separate encodings were required for other regions and scripts.
A code page is a particular mapping between encoded values and characters, often identified by a number or registered name. DOS code page 437 included box-drawing characters and symbols useful for text-mode interfaces. Windows-1252 covered Western European text with a different upper-byte mapping. Although often confused with ISO 8859-1, Windows-1252 assigns printable punctuation and symbols to many positions in 80–9F, where ISO 8859-1 has control characters. This distinction explains why quotation marks and dashes in older documents sometimes become controls or unexpected symbols when decoded under the wrong label.
Writing systems containing thousands of characters required multibyte approaches. Shift_JIS and EUC-JP became important for Japanese; Big5 was widely used for traditional Chinese; GB2312, GBK, and GB18030 served successive requirements associated with Chinese text. These families differ in repertoire, byte structure, and compatibility. Shift_JIS, for example, combines single-byte characters with two-byte sequences. GB18030 uses sequences of one, two, or four bytes and provides extensive coverage beyond earlier Chinese encodings. “Chinese encoding” or “Japanese encoding” is therefore not a sufficiently precise technical designation.
Some encodings are stateful rather than merely variable-length. ISO-2022-JP uses escape sequences to switch among designated character sets, allowing Japanese text to pass through environments constrained to seven-bit data. A byte’s interpretation can depend on an earlier escape sequence, so decoding a fragment independently may fail even when that fragment came from valid text. Random access, truncation, and recovery after transmission errors are more complicated than in encodings whose character boundaries are locally recognizable.
Legacy encodings remain important in archives, industrial equipment, mainframes, older software, and established document collections. Conversion tables are not always perfectly reversible: different source characters may map to the same destination character, and vendor variants may assign different meanings to nominally similar codes. User-defined and private character areas create additional uncertainty because their intended appearance may depend on local fonts or organizational conventions rather than a public interchange standard.
Unicode and the universal character repertoire
Unicode was developed to replace fragmented character repertoires with a common framework for text from many writing systems. The first version appeared in 1991. Its repertoire and architecture were subsequently coordinated with ISO/IEC 10646, the Universal Coded Character Set. Unicode is not a single byte encoding: it defines character identities, code points, properties, and text-processing behavior, while UTF-8, UTF-16, and UTF-32 provide principal ways to represent its scalar values in code units.
The Unicode code space extends from U+0000 to U+10FFFF, comprising 1,114,112 code points arranged in 17 planes of 65,536 positions each. Plane 0 is the Basic Multilingual Plane, which contains many commonly used scripts and symbols. Other planes accommodate additional scripts, symbols, and large collections of ideographs. The range U+D800–U+DFFF is reserved for surrogate code points. A Unicode scalar value is any Unicode code point except one in that surrogate range; scalar values are the units validly represented by standard UTF encodings.
Unicode distinguishes characters that have linguistic or textual identity from their presentation. A font may display the same character in different styles, and contextual shaping may cause one character to have different visible forms. Conversely, some visible units require several encoded characters. A letter followed by a combining accent is one example. The standard also assigns nonprinting characters, including directionality controls, joining controls, and variation selectors. These can affect interpretation or presentation even though they do not produce independent visible marks.
Han unification is an important example of the repertoire’s design principles. Many related Chinese, Japanese, and Korean ideographs share encoded identities where the standard treats them as the same abstract character. Regionally appropriate glyph forms are generally handled through fonts and language-sensitive rendering. Distinct characters are not all merged, and variation mechanisms provide ways to represent certain standardized glyph distinctions. This approach separates character identity from typography, although applications requiring exact historical or regional forms may need information beyond a plain code-point sequence.
Unicode includes private-use code points whose meanings are established by agreement among users rather than by the standard. It also reserves noncharacters for internal uses and leaves many code points unassigned for future additions. Its stability policies protect existing text from disruptive reassignment. Unicode does not define a universal font, guarantee display support, or prescribe one linguistic interpretation for every text operation. Character assignment, rendering support, language-specific spelling rules, and application behavior remain separate layers.
UTF-8, UTF-16, and UTF-32
UTF-8 represents Unicode scalar values using one to four bytes. Developed in 1992 by Ken Thompson and Rob Pike, it preserves ASCII byte values unchanged: U+0000–U+007F occupy one byte each. Other scalar values occupy two, three, or four bytes. Leading-bit patterns distinguish initial bytes from continuation bytes, whose binary form begins with 10. An ASCII byte cannot occur inside a multibyte UTF-8 sequence, making ASCII punctuation recognizable without mistaking part of a non-ASCII character for an ASCII delimiter.
For example, “A” is encoded as 41, “é” at U+00E9 as C3 A9, and “中” at U+4E2D as E4 B8 AD. The character “😀” at U+1F600 becomes F0 9F 98 80. UTF-8 is compact for ASCII-heavy text and has no byte-order variation. Its structure also supports recovery of character boundaries near an arbitrary byte position, although this does not eliminate the need to validate input or preserve incomplete sequences across streamed chunks.
UTF-16 uses one 16-bit code unit for most scalar values in the Basic Multilingual Plane and a pair of code units for supplementary characters. For a supplementary code point , let . The high and low surrogates are
Thus U+1F600 becomes the code-unit pair D83D DE00. An isolated surrogate is not a Unicode scalar value and is not well-formed UTF-16. Historical UCS-2, which represented characters with individual 16-bit values, lacks UTF-16’s mechanism for supplementary characters.
UTF-32 represents each Unicode scalar value with one 32-bit code unit. Its fixed code-unit count simplifies some operations on scalar values, but does not make every user-perceived character a single unit: combining sequences and emoji sequences may still contain several scalar values. UTF-32 also requires four bytes for an ASCII character, compared with one in UTF-8. Storage efficiency therefore depends on the text and representation. For common BMP Chinese characters, UTF-8 typically uses three bytes while UTF-16 uses two; supplementary characters require four bytes in both.
The following values illustrate these differences. UTF-16 and UTF-32 entries show code units, not an assumed byte order.
| Character | Code point | UTF-8 bytes | UTF-16 code units | UTF-32 code unit |
|---|---|---|---|---|
| A | U+0041 | 41 |
0041 |
00000041 |
| é | U+00E9 | C3 A9 |
00E9 |
000000E9 |
| 中 | U+4E2D | E4 B8 AD |
4E2D |
00004E2D |
| 😀 | U+1F600 | F0 9F 98 80 |
D83D DE00 |
0001F600 |
Byte order, declarations, and transport protocols
Multibyte code units require a rule for ordering bytes. Endianness distinguishes big-endian storage, which places the most significant byte first, from little-endian storage, which places the least significant byte first. The UTF-16 code unit 0041 is serialized as 00 41 in UTF-16BE and 41 00 in UTF-16LE. UTF-32 has the same issue with four-byte units. UTF-8 operates directly on individual bytes and therefore has no big-endian or little-endian variants.
A byte order mark is the character U+FEFF used at the beginning of a stream as an encoding signature. In UTF-16 it appears as FE FF or FF FE, revealing byte order. In UTF-8 its bytes are EF BB BF; these identify UTF-8 but convey no byte-order information. A leading signature can be useful in some file formats and disruptive in others. Its treatment is format-specific: a program that does not recognize it may treat it as content before the first expected character. Explicit labels such as UTF-16LE and UTF-16BE specify byte order independently of such detection.
Encoding information is commonly carried as metadata rather than encoded in every character. HTTP can declare a text representation with a header such as Content-Type: text/html; charset=utf-8. HTML also supports an in-document declaration such as <meta charset="utf-8">. XML has encoding-detection rules and an optional encoding declaration. A filename extension alone generally does not establish the encoding of a plain-text file. When declarations conflict, the relevant format or protocol specifies their precedence; there is no single rule covering every kind of text.
In email, character encoding and transfer encoding are separate layers. MIME’s charset parameter identifies how text bytes represent characters. Base64 and quoted-printable instead transform those bytes for transport through systems with restrictions on line lengths or byte values. A receiver reverses the transfer encoding before decoding the resulting bytes using the character encoding. Confusing these steps can produce text that is still encoded at one layer or has been decoded twice.
Modern interchange formats often narrow the permitted choices to reduce ambiguity. JSON exchanged between systems outside a closed ecosystem is required by its interoperability specification to use UTF-8. Other protocols retain multiple encodings for historical reasons. Registered charset names and aliases help systems identify them, but a label’s precise interpretation can depend on the consuming standard. Web-compatible decoding, for example, deliberately treats several historical labels according to compatibility rules rather than merely reproducing their original standards.
Text processing beyond encoding
Encoding determines how text is represented, not how many meaningful “characters” a user sees. The word can refer to bytes, code units, Unicode code points, or grapheme clusters. These quantities differ. The visible “é” can be represented as U+00E9 or as U+0065 followed by U+0301, a combining acute accent. The second representation contains two code points but usually forms one extended grapheme cluster. A family emoji may involve several pictographs joined by zero-width joiners and appear as one visual unit in a supporting font.
Unicode normalization gives consistent forms to certain equivalent sequences. NFD performs canonical decomposition; NFC applies canonical decomposition followed by canonical composition where permitted. NFKD and NFKC additionally apply compatibility mappings, which may remove distinctions such as some presentation forms or formatting variants. The precomposed and decomposed representations of “é” become equal after suitable canonical normalization. Compatibility normalization is broader: it can change distinctions significant in specialized notation or document preservation. Normalization therefore has a defined semantic scope rather than being a universally lossless cleanup operation.
String length and indexing depend on a programming environment’s representation. JavaScript strings use sequences of UTF-16 code units, so "😀".length is 2 even though the string contains one Unicode scalar value. Python 3’s ordinary string length counts code points, so the corresponding length is 1. Neither rule by itself measures display width or user-perceived character count. Text editors and user interfaces often need grapheme-aware boundaries for cursor movement and deletion, while terminal layouts may additionally depend on width conventions and rendering behavior.
Ordering and case conversion also extend beyond encoding. Bytewise UTF-8 order preserves the numerical ordering of scalar values, but numerical code-point order is not generally an appropriate dictionary order. Collation uses language-sensitive rules and may treat accents, case, or letter combinations at different comparison levels. Case conversion is not always a one-character substitution: Unicode case folding can map German “ß” to ss, and Turkish casing distinguishes dotted and dotless forms of the letter I. A database may accordingly store UTF-8 text while using an independently selected collation for comparisons and indexes.
Visual display adds further processing. The bidirectional text algorithm handles mixtures of left-to-right and right-to-left writing, while shaping systems select contextual forms and ligatures. Encoded text is normally stored in logical order, not simply in the left-to-right order of glyphs on a screen. Changing the byte encoding does not translate a language, supply a missing font, fix every broken line break, or perform these higher-level layout operations.
Detection, conversion, and corrupted text
When an encoding declaration is missing, programs may attempt to infer the encoding. Some evidence is structural: a byte order mark can identify a Unicode representation, and a sequence can be tested for UTF-8 validity. Other evidence is statistical, such as expected letter frequencies or common byte patterns in a language. Detection cannot generally be certain. A file containing only ASCII letters is valid ASCII and also valid under UTF-8 and numerous ASCII-compatible legacy encodings. Many short byte sequences have plausible interpretations under more than one encoding.
Mojibake is garbled text produced when encoded data is interpreted using the wrong character encoding. The UTF-8 bytes C3 A9, intended as “é,” become “é” when decoded as Windows-1252. If that mistaken result is then saved as UTF-8, the corruption becomes an additional encoding layer. Such errors can sometimes be reversed when the exact transformations are known and no information was discarded. Guessing replacement rules without reconstructing those transformations may alter legitimate text that happens to resemble corruption.
Transcoding normally consists of decoding the source bytes into an internal character representation and then encoding those characters in the destination encoding. Both stages can encounter errors. A decoder may see malformed input; an encoder may encounter a character absent from the destination repertoire. Implementations commonly offer strict failure, replacement, or omission. Unicode decoders often use U+FFFD, the replacement character, for malformed input. The exact replacement behavior—including how many input bytes are consumed—depends on the encoding algorithm and the applicable standard.
Replacement and omission can destroy recoverable information. Once several different invalid sequences have all become U+FFFD, the original bytes cannot be inferred from that character alone. Likewise, replacing an unsupported character with ? does not preserve its identity. Some systems provide specialized mechanisms for retaining undecodable bytes, but these are distinct from ordinary interoperable Unicode text. Archived source bytes can therefore contain information that a decoded and re-saved text version no longer retains.
Streaming creates a separate boundary problem. A network packet or input buffer may end halfway through a multibyte sequence even though the complete stream is valid. An incremental decoder retains pending bytes until more input arrives and reports an incomplete sequence only when required, such as at the actual end of input. Stateful legacy encodings may also require retention of shift state. Processing each arbitrary buffer as an independent text document can introduce replacement characters or false errors that were absent from the source.
Validation, security, and system interfaces
Encoding errors become security issues when components disagree about what bytes mean. A filter may inspect one interpretation while a later parser acts on another. Historical UTF-8 implementations sometimes accepted overlong encodings, in which a character was represented using more bytes than permitted. For example, the ASCII slash has the valid UTF-8 representation 2F; the byte sequence C0 AF is not a valid alternative. If one component rejected literal slashes but another accepted this malformed sequence as a slash, the disagreement could undermine path checks. Conforming UTF-8 validation excludes overlong sequences, surrogate code points, and values above U+10FFFF.
Normalization and comparison rules can create similar mismatches without malformed input. Two strings may appear alike or be canonically equivalent while having different bytes. A system that checks an identifier under one equivalence relation and stores or retrieves it under another can produce collisions or inconsistent authorization decisions. Conversely, treating visually similar text as necessarily identical can merge genuinely different names. Encoding validity, canonical equivalence, linguistic equality, and visual similarity are separate properties.
Unicode also represents characters from different scripts that can look alike. Latin “a” and Cyrillic “а” have different code points despite their similar appearance in many fonts. Such confusables are relevant to phishing, internationalized domain names, and deceptive identifiers. Bidirectional controls can also make the displayed ordering of source text differ from the ordering processed by a compiler. The 2021 “Trojan Source” research drew attention to this class of source-code presentation problem. These issues concern interpretation and display of valid text, not simply malformed encodings.
System interfaces introduce further distinctions. In the C programming language, many string interfaces use a zero byte as a terminator, whereas a Unicode string may legitimately contain U+0000. Passing length-delimited text through a null-terminated interface can therefore truncate it. A byte buffer’s capacity is also not interchangeable with its capacity in characters or UTF-16 code units. Incorrect assumptions during conversion can contribute to truncation and buffer overflow defects. Operating systems differ as well: many Unix interfaces expose pathnames as byte sequences, while Windows commonly provides UTF-16-based wide-character interfaces.
Byte-exact integrity checks impose another requirement. A cryptographic hash function or digital signature operates on specified bytes, so changing an encoding, normalization form, byte order mark, or line ending normally changes the result even when the displayed text appears unchanged. Protocols that sign textual information must consequently define the serialized representation or a precise canonicalization procedure. At these boundaries, preserving meaning and preserving bytes are related but distinct requirements.