{
  "id": "connectionist-temporal-classification",
  "title": "Connectionist Temporal Classification",
  "lang": "en",
  "source_lang": "en",
  "summary": "Connectionist Temporal Classification trains sequence models from unaligned labels by summing the probabilities of all valid monotonic alignments.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "CTC",
    "CTC loss",
    "CTC objective"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/connectionist-temporal-classification/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/connectionist-temporal-classification/",
      "json": "https://aiwiki.page/zh/connectionist-temporal-classification.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Connectionist Temporal Classification (CTC) is a training objective and probabilistic framework for mapping an input sequence to a usually shorter sequence of discrete labels when the alignment between them is unknown. Introduced in 2006, it made it possible to train artificial neural networks directly on examples such as speech recordings paired with transcriptions, without requiring a label for every audio frame. It is widely used in speech recognition, handwriting recognition, and other tasks in which output symbols follow the order of the input but occupy variable amounts of time or space.\n\nCTC adds a special blank symbol to the output alphabet and defines a rule that converts frame-level symbol sequences into label sequences. Its loss function sums the probabilities of all frame-level sequences that produce the desired transcription, rather than selecting one alignment in advance. A dynamic programming algorithm makes this summation tractable and provides gradients for end-to-end learning.\n\nCTC is not a particular neural-network architecture. It can be attached to recurrent, convolutional, or Transformer encoders, provided their outputs form an ordered sequence. Its defining characteristics are monotonic alignment, marginalization over possible alignments, and a factorized distribution over frame-level outputs. These characteristics account both for its computational advantages and for its limitations in language modeling, timing, and tasks involving reordering."
    },
    {
      "heading": "Origins and the unsegmented sequence problem",
      "text": "Many recognition tasks provide supervision at the sequence level rather than at the level of individual observations. A speech recording may contain thousands of acoustic frames but have a transcription containing only a few dozen characters. A handwritten word may span hundreds of image columns, without explicit boundaries between letters. Even when the transcription is known, the location and duration of each symbol remain uncertain.\n\nEarlier recognition systems commonly addressed this problem through segmentation, alignment procedures, or hidden Markov models. A speech recognizer might associate phonetic states with short acoustic intervals and use a pronunciation dictionary to connect those states to words. Neural networks could estimate acoustic state probabilities, but training them often depended on frame-level targets obtained from another model. Such arrangements were effective, yet they separated acoustic learning from parts of the alignment process and required additional modeling assumptions.\n\nCTC was introduced by Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber in the paper “Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks,” presented at the International Conference on Machine Learning in 2006. The method allowed a network to receive an unsegmented sequence and learn from its ordered label sequence alone. The original work used recurrent networks and demonstrated the method on phonetic recognition.\n\nThe word connectionist reflects its origin in neural-network modeling. The word classification does not mean that the entire input receives one class label: CTC performs sequence labeling, producing a variable-length string from an ordered stream of observations. Its principal contribution was to make an unknown alignment a latent variable that could be summed out during training.\n\nThe supervision remains supervised learning. A recording and its correct transcription are still required; what is unnecessary is the detailed correspondence between transcript symbols and input positions. This distinction separates CTC from unsupervised learning and from methods that discover linguistic units without labeled transcripts.\n\nDuring the following decade, CTC became an important component of deep learning systems for speech and handwriting. Deep Speech, described in 2014, and Deep Speech 2, described in 2015, were prominent large-scale speech-recognition systems using the objective. Its adoption also extended beyond recurrent architectures, establishing a separation that remains central to the method: the encoder determines how observations are represented, while CTC determines how uncertain monotonic alignments are scored and learned."
    },
    {
      "heading": "Paths, blanks, and sequence probabilities",
      "text": "Let an input sequence be \\(x=(x_1,\\ldots,x_N)\\). An encoder transforms it into \\(T\\) ordered output positions, where \\(T\\) need not equal \\(N\\), because the encoder may reduce temporal or spatial resolution. At each position \\(t\\), it produces a probability distribution \\(q_t\\) over a label alphabet \\(\\mathcal A\\) augmented with a special blank symbol, written \\(\\varnothing\\). A softmax function commonly converts the encoder’s output scores into this distribution.\n\nA path is a length-\\(T\\) sequence\n\n\\[\\pi=(\\pi_1,\\ldots,\\pi_T), \\qquad \\pi_t\\in\\mathcal A\\cup\\{\\varnothing\\}.\\]\n\nCTC assigns the path probability\n\n\\[P(\\pi\\mid x)=\\prod_{t=1}^{T}q_t(\\pi_t).\\]\n\nThis factorization is a modeling assumption: after conditioning on the input representation, the path symbols at different positions are treated as independent.\n\nA collapse function \\(B\\) converts a path into an output label sequence. It first merges consecutive repetitions of the same symbol and then removes blanks. The order of these operations matters. The path\n\n\\[(a,a,\\varnothing,a,b,b)\\]\n\ncollapses to \\(aab\\), not \\(ab\\): the blank separates the two occurrences of \\(a\\), preventing them from being merged. Blank is therefore not simply an ordinary space character. It represents an alignment position that contributes no output label.\n\nFor a target sequence \\(y=(y_1,\\ldots,y_U)\\), the conditional probability is\n\n\\[P(y\\mid x) = \\sum_{\\pi:B(\\pi)=y} \\prod_{t=1}^{T}q_t(\\pi_t).\\]\n\nThe training loss for one example is\n\n\\[\\mathcal L_{\\mathrm{CTC}}(x,y) = -\\log P(y\\mid x).\\]\n\nDifferent paths can allocate different numbers of positions to a label, place blanks differently, or shift a label’s location while producing the same transcription. Training rewards their combined probability.\n\nThe collapse rule imposes a precise length constraint. If \\(R\\) is the number of adjacent equal-label pairs in the target, a valid alignment requires\n\n\\[T\\ge U+R.\\]\n\nFor example, the target \\(aa\\) requires at least three positions, because its two labels must be separated by a blank. The target \\(ab\\) requires only two. A target that violates this condition has zero probability under the model.\n\nCTC thus supports variable duration and uncertain boundaries, but not arbitrary changes in label order. Its alignment is monotonic in encoder position. An encoder may incorporate information from the entire input, yet the output labels must still be emitted in their target order along that sequence."
    },
    {
      "heading": "Forward–backward computation and gradients",
      "text": "Enumerating all possible paths would require considering \\((|\\mathcal A|+1)^T\\) sequences. CTC avoids this exponential computation by grouping paths according to how far they have progressed through the target. The resulting forward–backward algorithm is closely related to other dynamic programs for probabilistic sequence models.\n\nFirst construct an expanded target by placing blanks before, after, and between labels:\n\n\\[z=(\\varnothing,y_1,\\varnothing,y_2,\\ldots,y_U,\\varnothing).\\]\n\nIts length is \\(S=2U+1\\). Let \\(\\alpha_t(s)\\) denote the total probability of valid partial paths that have consumed \\(t\\) output positions and end at expanded-target position \\(s\\). For a nonempty target, initialization is\n\n\\[\\alpha_1(1)=q_1(\\varnothing),\\qquad \\alpha_1(2)=q_1(y_1),\\]\n\nwith all other values zero.\n\nAt subsequent positions,\n\n\\[\\alpha_t(s)=q_t(z_s) \\left[ \\alpha_{t-1}(s) +\\alpha_{t-1}(s-1) +\\delta_s\\alpha_{t-1}(s-2) \\right],\\]\n\nwhere out-of-range terms are zero and\n\n\\[\\delta_s= \\begin{cases} 1,& s>2,\\ z_s\\ne\\varnothing,\\ z_s\\ne z_{s-2},\\\\ 0,& \\text{otherwise}. \\end{cases}\\]\n\nThe three alternatives correspond to remaining at the same state, advancing one state, or skipping an intervening blank. The skip restriction ensures that adjacent identical target labels cannot merge into one. The final sequence probability is\n\n\\[P(y\\mid x)=\\alpha_T(S)+\\alpha_T(S-1).\\]\n\nA backward quantity \\(\\beta_t(s)\\) can be defined as the probability of valid continuations after position \\(t\\), given state \\(s\\) at that position. With this convention, it excludes the emission probability already included in \\(\\alpha_t(s)\\). The two terminal states have \\(\\beta_T(S)=\\beta_T(S-1)=1\\). The posterior occupancy of expanded-target state \\(s\\) is then\n\n\\[\\gamma_t(s)= \\frac{\\alpha_t(s)\\beta_t(s)}{P(y\\mid x)}.\\]\n\nSumming occupancies over states carrying symbol \\(k\\) gives\n\n\\[\\Gamma_t(k)=\\sum_{s:z_s=k}\\gamma_t(s).\\]\n\nIf \\(a_t(k)\\) is the corresponding softmax logit, the loss gradient is\n\n\\[\\frac{\\partial\\mathcal L_{\\mathrm{CTC}}}{\\partial a_t(k)} = q_t(k)-\\Gamma_t(k).\\]\n\nThis connects alignment marginalization directly to ordinary backpropagation through the encoder.\n\nThe alignment calculation takes \\(O(TU)\\) time. Forward probability alone can be computed with \\(O(U)\\) working memory by retaining successive rows, whereas a straightforward gradient implementation stores \\(O(TU)\\) intermediate values. These bounds exclude the encoder and the cost of producing probabilities over the alphabet.\n\nBecause long products underflow in floating-point arithmetic, implementations generally use logarithmic probabilities and log-sum-exp operations, or equivalent rescaling schemes. The dynamic program is exact within its probabilistic model; it is not a beam-search approximation. Approximation becomes relevant primarily during unrestricted decoding, where the correct target sequence is not supplied."
    },
    {
      "heading": "Optimization and implementation",
      "text": "The CTC objective replaces externally supplied frame labels with posterior distributions over alignments. Early in training, these distributions may be diffuse: many placements of the same transcript receive appreciable probability. As the encoder improves, the probability mass often concentrates around a smaller set of paths. This is how the model learns timing indirectly from sequence-level supervision.\n\nA common outcome is peaky output. Blank dominates many positions, while nonblank labels receive high probability at relatively isolated positions. Such behavior is compatible with the objective because emitting one strong label and surrounding it with blanks can explain the same transcript as a broader run of repeated labels. The loss does not require a label’s probability to remain high throughout its physical duration. Consequently, blank occupancy and the sharpness of emissions are not direct measures of transcription quality.\n\nInput and target lengths are essential implementation data. In a batch, shorter inputs are often padded to a common size, but padding positions must not be treated as valid encoder output. Similarly, target padding must be excluded from the transcription length. The length supplied to CTC must describe the sequence after any encoder subsampling. Confusing original audio length with encoded length can produce invalid alignments or train the model on artificial positions.\n\nRepeated-label constraints make length checking more subtle than testing \\(T\\ge U\\). If a target contains consecutive identical tokens, it needs extra positions for blank separation. Aggressive downsampling can therefore make some examples impossible even when their target length appears modest. An impossible example has infinite negative log-likelihood. Some software exposes an option to replace infinite losses and associated gradients with zeros, but that convention does not restore the missing alignment or make the example informative.\n\nCTC losses are available in major frameworks including PyTorch and TensorFlow. Their interfaces differ in tensor layout, whether they accept logits or log-probabilities, how they encode variable-length targets, and which index denotes blank. Blank is normally excluded from target transcriptions because it belongs to the alignment mechanism rather than the label vocabulary.\n\nNumerical precision also matters. A neural encoder may run largely in reduced precision on a graphics processing unit, while the alignment recurrence uses higher-precision accumulation. Loss reduction conventions—such as averaging examples directly or first dividing by target length—change the relative influence of short and long sequences. These details are part of the optimization specification, not merely formatting choices: two implementations can use the same mathematical CTC objective yet produce different gradient scales or effective example weights."
    },
    {
      "heading": "Decoding and language-model integration",
      "text": "Training evaluates the probability of a known transcription. Recognition instead seeks\n\n\\[\\hat y=\\arg\\max_y P(y\\mid x).\\]\n\nThe simplest procedure is greedy decoding, also called best-path decoding: choose the highest-probability symbol at each position and collapse the resulting path. This maximizes the probability of an individual path, but it does not generally maximize the sum of all paths belonging to a transcription.\n\nA two-position example demonstrates the difference. Suppose both positions assign probability \\(0.40\\) to blank, \\(0.35\\) to \\(a\\), and \\(0.25\\) to \\(b\\). The most probable individual path is two blanks, with probability \\(0.16\\), so greedy decoding returns the empty string. Yet the transcription \\(a\\) has probability\n\n\\[P(a\\mid x) = P(a,a)+P(a,\\varnothing)+P(\\varnothing,a) = 0.1225+0.14+0.14 = 0.4025.\\]\n\nIts combined probability exceeds that of the greedy result.\n\nBeam search retains multiple partial hypotheses. A CTC prefix beam search groups alignments that collapse to the same output prefix and tracks separate probabilities for paths ending in blank and nonblank symbols. This separation is necessary because adding a repeated label has different effects in the two cases: a repeated nonblank can extend the current run, whereas repetition after blank creates another output label. Pruning limits computation but can discard a hypothesis that would later become optimal.\n\nAn external language model can influence decoding through a score such as\n\n\\[S(y)=\\log P_{\\mathrm{CTC}}(y\\mid x) +\\lambda\\log P_{\\mathrm{LM}}(y) +\\eta\\,g(y),\\]\n\nwhere \\(\\lambda\\) controls language-model weight and \\(g(y)\\) is a length or insertion feature. The exact definition of \\(g\\), including whether it counts words or tokens, depends on the decoder. This score is a decision rule, not automatically the logarithm of a normalized conditional probability.\n\nLexicons and finite-state transducers provide another route to constrained decoding. They can connect token sequences to pronunciations, words, and language-model states. Character systems may operate without a fixed word lexicon, while subword systems trade a larger output alphabet for shorter label sequences.\n\nDecoder choices can materially change recognition accuracy and runtime. Greedy decoding exposes the encoder’s direct predictions at low cost; beam decoding recovers probability spread across paths; language-model integration adds information from external text. Reported results therefore need to distinguish the acoustic or visual model from the complete recognition system. A gain produced by a larger beam, a stronger language model, or additional text is not solely a gain in the CTC-trained encoder."
    },
    {
      "heading": "Encoder architectures and streaming behavior",
      "text": "CTC requires an ordered sequence of probability distributions, not a particular mechanism for computing them. Early systems frequently used recurrent neural networks, especially long short-term memory networks, because they could incorporate context over many input positions. Bidirectional recurrent encoders used both earlier and later observations to disambiguate a label.\n\nThe factorization of path probabilities does not mean that a prediction at position \\(t\\) sees only input frame \\(x_t\\). A bidirectional encoder can condition every output on the full recording. A convolutional neural network can combine neighboring image columns or acoustic frames over an expanding receptive field. A Transformer can use self-attention to connect distant positions. What remains factorized is the distribution over path symbols once the input-conditioned network outputs are fixed.\n\nThis distinction determines streaming capability. The CTC alignment algorithm is compatible with processing an ordered stream, but an unrestricted bidirectional encoder requires future input. A causal recurrent encoder can operate incrementally, while chunked attention and limited look-ahead provide intermediate choices. End-to-end latency depends on feature extraction, encoder context, subsampling, label emission timing, and decoder decisions—not on the CTC loss alone.\n\nTemporal subsampling is especially important in speech. Reducing the encoder sequence by a factor of four reduces the \\(T\\) dimension of the alignment problem and can lower both encoder and loss computation. It also reduces the number of available alignment positions. A model with a short output sequence may work well for subword labels yet fail the minimum-length constraint for a long character transcription with many repetitions.\n\nIn visual text recognition, an encoder often converts a two-dimensional text-line image into a one-dimensional feature sequence ordered horizontally. CTC then associates character labels with positions along that sequence. This arrangement works naturally for a consistently ordered line but does not itself determine how to read multiple columns, curved text, or a complex page. Rectification, line detection, or explicit layout processing may be required before the CTC stage.\n\nThe same loss can therefore support very different operational systems: an offline recognizer using full-context attention, a low-latency recognizer with a causal encoder, or a convolutional recognizer processing cropped word images. Their common training objective does not imply common memory requirements, receptive fields, or response times."
    },
    {
      "heading": "Applications and label design",
      "text": "Speech recognition is a central application because spoken units occur in sequence while their durations vary substantially. Inputs commonly consist of short-time acoustic features or learned representations of the waveform. Targets may be phonemes, characters, word pieces, or words. CTC permits these units to be learned without manually marking where each one begins and ends.\n\nThe label vocabulary changes the learning problem. Character vocabularies are relatively small and can spell words absent from the training transcripts, but they produce longer target sequences. Word-level vocabularies yield shorter sequences but require many output classes and have difficulty representing unseen words. Subword tokenization, including byte-pair encoding, occupies an intermediate position. Tokenization also changes which adjacent labels are identical and therefore changes the exact alignment-length requirements.\n\nIn optical character recognition, CTC supports recognition without explicit segmentation into individual character images. A text-line encoder can learn that several neighboring feature positions correspond to one character and that spaces or gaps need not have fixed widths. Handwriting recognition particularly benefits from avoiding rigid character boundaries because strokes may connect across letters and character widths vary widely. The widely used convolutional recurrent neural network approach introduced by Baoguang Shi, Xiang Bai, and Cong Yao combined convolutional features, recurrent sequence modeling, and CTC transcription.\n\nCTC also appears in continuous gesture and sign-recognition systems when an ordered input video is paired with an ordered sequence of symbolic labels. The monotonicity assumption is appropriate only for the particular annotation sequence being predicted. Recognizing gloss labels from signing video and translating a signed language into a written language are different tasks; translation can involve grammatical reordering that ordinary CTC does not represent directly.\n\nBiological sequence analysis provides another application. In nanopore sequencing, electrical measurements vary with molecules passing through a pore, while the desired output is a nucleotide sequence. CTC can connect many signal positions to a shorter base sequence without fixed event segmentation. Repeated bases pose a particular challenge because multiple identical output labels must be distinguished rather than collapsed, making blank-separated alignments consequential.\n\nAcross these domains, the method’s usefulness depends on a compatible ordering relation and adequate encoder resolution. It does not require that input and output have the same sampling rate or that each symbol have a fixed duration. It does require that the target be expressible through an ordered traversal of encoder positions, with enough positions to represent every label and any necessary repeat separators."
    },
    {
      "heading": "Relationship to other sequence models",
      "text": "CTC shares important mathematical machinery with hidden Markov models. Both can represent latent alignments and use forward–backward recurrences to sum over them. In CTC, however, a neural encoder directly provides input-conditioned distributions over labels and blank, and the collapse mapping determines the permitted alignments. Standard CTC has no independently learned state-transition probabilities or explicit duration distribution.\n\nRemaining at the same expanded-target state acts like a self-loop and permits a symbol to occupy several positions. This resembles duration handling in simple state models, but its interpretation differs from a generative acoustic model. The probability attached to that duration is supplied by the sequence of neural predictions, which may depend on broad input context. CTC is therefore not simply an HMM with renamed states, although both can be understood through weighted path computations.\n\nA central limitation is the absence of direct conditioning on previously emitted labels. An encoder–decoder architecture with an autoregressive decoder instead models\n\n\\[P(y\\mid x) = \\prod_{u=1}^{U}P(y_u\\mid y_{<u},x).\\]\n\nIts decoder can learn explicit dependencies among output symbols, and an attention mechanism can select relevant encoder representations for each output step. Such systems need not obey CTC’s fixed monotonic collapse rule, which makes them suitable for tasks involving reordering, including machine translation.\n\nThe recurrent neural network transducer, usually abbreviated RNN-T, provides another comparison. It marginalizes over monotonic alignments while incorporating a prediction network conditioned on previous output labels. Its alignment lattice has both an input-time dimension and an output-label dimension: blank advances time, while a label advances the output sequence. This differs from CTC, where every path position consumes an encoder step and its distribution does not depend on the decoded prefix. Transducers are consequently attractive for streaming recognition, though their additional prediction and joint-network computations change training and decoding costs.\n\nHybrid systems combine these approaches. Joint CTC–attention training applies a CTC loss and an attention-decoder loss to a shared encoder. CTC encourages monotonic alignment, while the decoder learns output dependencies. During inference, a CTC prefix score can also contribute to beam search. Another arrangement uses CTC for a fast first pass and a more expensive decoder for rescoring.\n\nThese relationships separate three choices that are sometimes conflated: the encoder architecture, the alignment model, and the output-dependency model. A Transformer encoder with CTC remains a CTC system; using attention inside the encoder does not turn it into an autoregressive attention decoder."
    },
    {
      "heading": "Evaluation, alignment interpretation, and limitations",
      "text": "Recognition quality is usually measured after decoding rather than by CTC loss alone. Speech systems commonly report word error rate,\n\n\\[\\mathrm{WER}=\\frac{S+D+I}{N},\\]\n\nwhere \\(S\\), \\(D\\), and \\(I\\) count substitutions, deletions, and insertions relative to a reference containing \\(N\\) words. Character error rate uses the analogous calculation over characters. These measures are based on edit distance and capture the final transcription, whereas CTC loss measures the probability assigned to the reference after summing alignments.\n\nThe two objectives need not improve together on every example. A model may increase the probability of the reference while leaving the decoder’s top-ranked output unchanged. Conversely, a small probability shift can change the best hypothesis and reduce an error count substantially. Evaluation also depends on text normalization, punctuation handling, case, segmentation, and the use of an external language model. A test set and its scoring procedure therefore define more than the input recordings alone.\n\nCTC can produce alignments when the transcription is known. A Viterbi algorithm replaces summation with maximization to find the highest-probability path compatible with that transcript. Forward–backward posteriors instead describe uncertainty over positions. These are useful for forced alignment and transcript synchronization, but their timing interpretation requires care. The most likely emission position is not necessarily the physical onset or end of a sound or character.\n\nPeaky emissions make this distinction particularly important. A full-context encoder may use later evidence before assigning a high probability to an earlier label position, and different training objectives can shift emission timing without changing the recognized sequence. In streaming applications, latency is therefore evaluated explicitly rather than inferred from transcription accuracy. Word boundaries derived from CTC paths may also require separate conventions for assigning intervening blanks.\n\nThe label-independence assumption limits direct sequence modeling. Although input context can encode substantial linguistic information, the CTC output layer cannot directly condition its next distribution on the actual hypothesis prefix. Ambiguous acoustics, visually similar characters, and long-range grammatical constraints may consequently benefit from an external language model or another decoder. At the same time, strong language priors can favor plausible text over unusual names or rare sequences, changing the pattern of recognition errors.\n\nFinally, CTC probabilities are conditional on the model and label system; they are not automatically calibrated measures of correctness. Blank probability is not a confidence score for the whole transcription, and a high path probability is not identical to a high sequence probability. Robustness under distribution shift, performance on unfamiliar speakers or writing styles, and reliability of confidence estimates must be measured separately from alignment feasibility and training loss."
    }
  ],
  "links": [
    "artificial-neural-network",
    "speech-recognition",
    "loss-function",
    "dynamic-programming",
    "hidden-markov-model",
    "supervised-learning",
    "unsupervised-learning",
    "deep-learning",
    "probability-distribution",
    "softmax-function",
    "backpropagation",
    "pytorch",
    "tensorflow",
    "graphics-processing-unit",
    "beam-search",
    "language-model",
    "recurrent-neural-network",
    "long-short-term-memory",
    "convolutional-neural-network",
    "transformer-architecture",
    "self-attention",
    "tokenization",
    "byte-pair-encoding",
    "encoder-decoder-architecture",
    "attention-mechanism",
    "machine-translation",
    "edit-distance",
    "test-set",
    "distribution-shift"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}