Emergent abilities are capabilities of machine-learning systems that appear absent or negligible under some conditions but become observable after changes in model size, training, data, or computation. The expression is especially associated with large language models, where researchers have reported sharp improvements in tasks such as arithmetic, question answering, and reasoning as models become larger. In an influential formulation published in 2022, an ability was described as emergent when it was not present in smaller models but appeared in larger ones, making it difficult to predict by extrapolating the performance of the smaller models.
The term describes an observed pattern, not a universally established mechanism. A sudden rise in a benchmark score can reflect a genuine change in learned computation, a smooth improvement made abrupt by the scoring rule, or an interaction among training, prompting, and evaluation. Understanding emergent abilities therefore requires distinguishing what a model has learned from how that learning becomes visible. The subject connects the practical problem of predicting future systems with questions about generalization, representation, measurement, and the limits of behavioral evidence.
Definition and conceptual boundaries
In its most common technical use, an emergent ability is defined relative to a task, a family of models, a scale variable, and a measurement procedure. A claim that arithmetic “emerges,” for example, is incomplete unless it specifies the arithmetic problems, the models being compared, the prompting method, and the criterion for success. Two evaluations of the same models can produce different apparent thresholds if one measures exact answers and the other measures how much probability the model assigns to correct answers.
The 2022 paper Emergent Abilities of Large Language Models, by Jason Wei and colleagues, emphasized abilities whose performance remained near a baseline across smaller models and then improved substantially at larger scales. The baseline might be random guessing, a trivial strategy, or the lowest score allowed by the task. This operational definition made emergence a question about observed scaling curves rather than a claim that a model had acquired consciousness, humanlike understanding, or an entirely new kind of intelligence.
Several related phenomena require separation. An unexpected ability is one that observers did not anticipate; it need not appear abruptly. A capability acquired through fine-tuning may be new to a particular model but is not necessarily an emergent consequence of scale. Conversely, a capability can be emergent under a chosen evaluation even if its broad possibility was anticipated. The phrase also does not imply that a skill was learned without relevant information: pretraining may contain demonstrations, partial solutions, or closely related structures even when the exact evaluation task was not an explicit training objective.
The distinction between capability and elicitation is particularly important. A model may succeed only after a useful instruction, worked example, or output format is supplied. Such a result can indicate that the model already supported the relevant computation but that an earlier evaluation failed to elicit it. It can also indicate that the prompt supplied essential information rather than merely exposing a latent skill. Prompt engineering therefore changes the conditions under which an ability is measured, not just the wording surrounding an otherwise fixed test.
The word “emergent” also has a broader history in philosophy and the sciences, where it refers to system-level properties arising from interactions among components. In language-model research, however, the influential scaling-based definition is narrower. It concerns the relationship between measured performance and changing resources. Neither the broader philosophical meaning nor the visual appearance of a sudden jump establishes that a new internal mechanism has appeared. That stronger conclusion requires evidence about learning dynamics or computation inside the model.
Historical development of the research question
The background to the debate lies in the expansion of deep learning and large-scale pretraining during the 2010s. Systems trained on broad datasets increasingly supported tasks beyond a narrowly specified training objective. Transfer learning showed that representations acquired in one setting could improve performance elsewhere, while language-model pretraining made next-token prediction a useful foundation for many language tasks. These developments encouraged researchers to examine capabilities as properties of a broadly trained system rather than as separate modules built for individual applications.
The 2020 paper Language Models are Few-Shot Learners, by Tom Brown and colleagues, was an important precursor. It introduced the GPT-3 family, including a model with 175 billion parameters, and evaluated tasks using instructions and examples placed in the input rather than task-specific parameter updates. Its results helped establish in-context learning as a central object of study. Performance often improved with model size, but different tasks exhibited different trajectories, including curves that appeared to change sharply over the measured range.
Work on neural scaling laws supplied a contrasting regularity. Jared Kaplan and colleagues reported in 2020 that language-model loss followed comparatively smooth relationships with model size, dataset size, and training computation over the investigated regimes. Subsequent work, including the 2022 Chinchilla study by Jordan Hoffmann and colleagues, showed the importance of balancing model parameters and training tokens. These findings sharpened the puzzle: predictable improvement in an aggregate training objective could coexist with apparently unpredictable changes in particular downstream abilities.
Wei and colleagues’ 2022 study assembled examples of such changes across model families and evaluation settings. The same period saw the development of chain-of-thought prompting, in which examples include intermediate reasoning steps. Jason Wei and colleagues’ separate 2022 paper on that method reported substantial benefits for sufficiently large models on reasoning tasks. These findings made it important to distinguish emergence in ordinary task accuracy from emergence in the effectiveness of a prompting technique.
A major challenge followed in 2023 with Are Emergent Abilities of Large Language Models a Mirage?, by Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. They argued that nonlinear or discontinuous evaluation metrics could turn smooth underlying improvements into apparent breakthroughs, and demonstrated this effect through analyses and experiments. The resulting debate did not establish that every reported transition was either genuine or illusory. It instead made measurement choice, sampling density, and the meaning of predictability central to subsequent interpretations of emergence.
Scaling variables and the appearance of thresholds
The scale of a language model is not a single quantity. Parameter count measures the number of learned numerical values, while training tokens measure exposure to data. Training computation depends on both, as well as architecture and implementation. A model with more parameters can perform worse than a smaller model if it receives inadequate training or poorer data. Consequently, a threshold described as occurring at a particular parameter count is usually a property of a specific experimental series, not a universal boundary for the ability.
A common approximate description of scaling is a power-law relationship between reducible loss and resources:
where represents a resource such as computation, is an estimated limiting loss under the model, and and are fitted constants. Such a relationship describes average predictive error, not the detailed behavior of every downstream task. A smooth decline in loss can support very different task-level curves because tasks depend on different subsets of the knowledge and computation learned during training.
Resource allocation also matters. Increasing parameters while holding the training dataset fixed can lead to a different trajectory from increasing parameters and data together. Changing the balance of code, mathematical text, multilingual material, or instruction-like examples can affect specific abilities without a comparable change in total training volume. The composition and quality of data therefore complicate claims that an ability appeared “because the model became larger.” Model size can be correlated with several other improvements made during development.
Thresholds can arise at the level of practical usability without corresponding to a discontinuity in learning. A system that answers correctly 70 percent of the time may be unsuitable for an application that requires 95 percent accuracy. Smooth progress across that interval can nevertheless create a sudden change in whether the system is adopted. This is an operational threshold imposed by the application, distinct from a statistical threshold in benchmark scores or a mechanistic transition inside the model.
The horizontal axis can itself influence interpretation. Large models are often spaced far apart in parameter count, and results are commonly plotted on logarithmic scales. If only a few model sizes are available, a smooth but steep curve may look like a jump. Conversely, averaging across many tasks can conceal a transition specific to one task. Evidence for an abrupt change is therefore stronger when it includes closely spaced scales, comparable training conditions, and repeated runs rather than a comparison between one small model and one much larger successor.
Reported abilities and their evaluation settings
Reported examples span symbolic manipulation, factual question answering, language understanding, and reasoning. Multi-digit arithmetic became a prominent case because its outputs can be checked unambiguously and problem difficulty can be varied systematically. Yet arithmetic tasks differ substantially: adding two short integers, multiplying long integers, and solving a word problem require different combinations of parsing, stored patterns, and procedural computation. Exact success on one distribution does not establish a general arithmetic algorithm.
Another important setting was BIG-bench, a collaborative benchmark introduced in 2022 that brought together more than 200 tasks intended to probe a broad range of language-model behaviors. Tasks included forms of linguistic interpretation, symbolic reasoning, and knowledge use. Analyses of selected tasks revealed curves in which smaller models performed close to baseline while larger models achieved substantially better scores. The diversity of the benchmark made such findings informative, but it also meant that no single mechanism could be assumed to explain all apparent transitions.
In-context learning is often discussed as an emergent capability because examples placed in a prompt can change a model’s behavior without updating its parameters. For instance, several input–output pairs may indicate a classification rule or a translation format. Nevertheless, in-context learning is not a single indivisible skill. Copying a demonstrated format, retrieving a familiar task, inferring a new mapping, and implementing a learning-like procedure within the forward computation are different possibilities. Their scaling behavior need not be the same.
Chain-of-thought prompting illustrates emergence in the effectiveness of an intervention. On some reasoning benchmarks, providing worked reasoning examples helps larger models much more than smaller ones. A smaller model may reproduce the style of an explanation without producing a correct solution, whereas a larger one may use intermediate text to improve its answer. The observable result is a difference between prompting conditions. It does not, by itself, show that the generated explanation faithfully reports the computation responsible for the answer.
Claims about multilingual transfer, code generation, and compositional reasoning face related limitations. Apparent novelty may reflect an ability to combine previously learned components rather than the invention of a new operation. Furthermore, data leakage can make a model appear capable of solving problems whose answers or close variants occurred in training. Reliable examples therefore depend on carefully constructed test sets, controls for overlap, and variations that separate memorized surface patterns from systematic competence. Performance on unfamiliar lengths, altered symbol systems, and newly generated examples can be particularly informative.
Measurement artifacts and statistical interpretation
A central issue is that evaluation metrics transform behavior before it is plotted. Exact-match accuracy assigns a score of one only when the entire answer is correct and zero otherwise. It ignores whether an incorrect answer differs by one character or is wholly unrelated. A model can therefore become steadily better at the components of a task while receiving almost no credit until it produces fully correct outputs often enough to affect the measured score.
An illustrative calculation shows the effect. Suppose a task requires component decisions, each correct with probability , and assume independence for simplicity. The probability of complete success is then
For , increasing from to raises complete-success probability from approximately to . Raising further to brings it to about . Smooth improvements in component accuracy can thus produce a long region of near-zero exact-match performance followed by a steep rise. Real language outputs do not generally satisfy the independence assumption, but the example establishes that abrupt-looking task scores need not imply abrupt underlying learning.
Alternative metrics can expose progress hidden by exact matching. These include the probability assigned to a correct answer, character-level similarity, or edit distance between predicted and target strings. None is automatically superior for every purpose. Partial similarity may reveal learning while failing to measure whether an answer is usable. A nearly correct program may not execute, and a nearly correct numerical answer may remain wrong for the application. Metric-induced emergence can therefore be a misleading account of internal change while still describing a meaningful practical transition.
Finite samples add another source of apparent thresholds. If a model succeeds with probability on each of independent examples, the probability of observing no successes is
An evaluation containing 100 examples has about a 37 percent chance of observing no successes when the true success probability is 1 percent. Zero measured accuracy consequently does not prove the absence of an ability. Confidence intervals, repeated evaluations, and larger samples help distinguish a true floor from an undersampled low success rate.
The baseline also requires scrutiny. Random guessing differs across tasks, and multiple-choice accuracy can be distorted by answer-position preferences or uneven label frequencies. For free-form tasks, a nominal baseline of zero may hide substantial partial knowledge. A persuasive analysis reports both application-relevant scores and measurements that retain finer information about model behavior. It also distinguishes a change that was difficult to forecast from a change that merely looks discontinuous when plotted using a particular metric.
Proposed mechanisms and evidence from controlled systems
Several mechanisms could produce nonlinear behavioral improvements without requiring a single explanation for all emergent abilities. One possibility is the acquisition of intermediate representations. A task may become tractable only after training develops a representation of syntax, number, spatial relations, or another relevant structure. Improvements in representation learning can support many downstream behaviors at once, although observing simultaneous gains is not sufficient to identify a shared internal cause.
A second possibility involves combinations of prerequisites. A word problem may require understanding the text, selecting an operation, performing a calculation, and formatting an answer. If failure at any stage prevents success, the complete task can remain difficult until all stages become sufficiently reliable. This account resembles the multiplicative effects of exact-match scoring but can also describe genuine computational dependencies. It predicts that intermediate evaluations may reveal substantial competence before end-to-end performance becomes strong.
Controlled studies of grokking provide a related, but distinct, example of delayed behavioral change. In experiments reported by Alethea Power and colleagues in 2022, small networks trained on tasks such as modular arithmetic could memorize training examples and only much later develop strong test performance. The transition occurred over training time rather than across increasing model sizes. Grokking shows that sharp changes in generalization performance can occur in neural systems under controlled conditions, but it does not establish that large-model benchmark transitions share the same mechanism.
Research in mechanistic interpretability attempts to identify the computations underlying behavior. Studies of small transformers have examined attention patterns that support copying and continuation of repeated sequences. In particular, induction heads have been investigated as components of certain in-context learning behaviors. Work published in 2022 connected their formation with a change in loss and in-context learning in studied models. The relevance of these findings is strongest for the architectures and behaviors directly tested; broad claims about reasoning require additional evidence.
A mechanistic account should ideally support interventions. If a proposed circuit causes an ability, disabling or modifying it should predictably change that ability while appropriate controls remain intact. Correlations between activations and successful answers are weaker evidence because they may reflect accompanying processes rather than necessary computation. Even successful interventions can be difficult to interpret when information is distributed across many components. The major unresolved issue is how to connect detailed findings in tractable models to the overlapping, redundant computations of much larger systems without assuming that similar behavioral curves imply identical mechanisms.
Experimental design and the problem of prediction
A strong study begins by specifying the claim before examining the results. “Emergence” can mean a sharp increase in accuracy, a failure of a particular extrapolation method, a change in an internal representation, or the first crossing of a practical threshold. These are different hypotheses. Defining the task, scale variable, metric, baseline, and prediction procedure in advance reduces the risk of selecting an impressive-looking curve after inspecting many tasks.
Model comparisons need controls for more than parameter count. Useful records include architecture, training-token budget, data mixture, tokenization, optimization settings, and post-training procedures. Evaluating intermediate checkpoints can separate changes across training time from differences between independently trained models. Repeating training with different random seeds helps determine whether an apparent threshold is stable or reflects variability in optimization. In large systems, resource costs often limit replication, so conclusions must remain proportional to the available comparisons.
Evaluation should also vary the conditions of elicitation. A task can be tested with direct questions, demonstrations, intermediate-step prompts, and constrained output formats. For inference, decoding settings and the number of sampled answers matter: selecting a correct response from many attempts measures something different from obtaining it on the first attempt. Tool access introduces another distinction, because a model that invokes an external calculator is part of a larger system whose arithmetic performance cannot be attributed solely to the model’s internal computation.
Predictability requires an explicit forecast and a later test. Fitting a curve after observing all model sizes does not show that the larger models were predictable from the smaller ones. A stronger approach fits on lower scales, states uncertainty, and evaluates predictions on withheld scales. Comparisons can ask whether continuous metrics improve forecasts of exact-match success, whether intermediate tasks predict composite performance, and whether information about data composition explains variation that parameter count alone leaves unresolved.
Benchmark construction must account for distribution shift and contamination. Newly generated problems can reduce direct memorization, while variations in wording, symbol choice, and problem size test whether a skill transfers beyond a narrow template. Separate validation sets prevent repeated prompt selection from quietly adapting the evaluation to a particular test. When hundreds of tasks are examined, reporting the full collection of curves is more informative than publishing only the most dramatic examples. Such designs make it possible to distinguish a robust phenomenon from an artifact of task selection, prompt search, or sparse measurement.
Implications for deployment, safety, and governance
The practical importance of emergent abilities lies partly in forecast uncertainty. Developers may have reliable estimates of average language-model loss while possessing much weaker predictions of particular capabilities. This gap affects release decisions, evaluation budgets, and assessments of whether existing safeguards cover a more capable successor. It does not imply that every increase in scale will produce a surprise, only that aggregate performance measures do not fully characterize a system’s operational behavior.
For AI safety, the relevant object is often the deployed system rather than the pretrained model alone. Instruction tuning, reinforcement learning from human feedback, tools, retrieval, and multi-step workflows can change what a system can accomplish. An AI agent that repeatedly plans, calls software, and checks results may complete tasks that a single model response cannot. A capability that appears abruptly at the system level may therefore result from orchestration or access to external resources rather than a new ability arising during pretraining.
Capability must also be separated from behavioral restraint. A model can possess knowledge needed for a task while declining to provide an answer under one interaction pattern. Conversely, willingness to answer is not evidence of competence. Safety evaluations consequently distinguish successful task performance from refusals, policy compliance, and robustness under changed conditions. The same distinction applies to benign applications: an unhelpful response may reflect instruction following or output constraints rather than an absence of the underlying knowledge.
High-stakes assessments require care in translating benchmark scores into claims about real-world consequences. Performance on a short cybersecurity questionnaire, for example, is not equivalent to demonstrated effectiveness in an operational environment. Real tasks involve incomplete information, changing conditions, tool failures, and verification of outcomes. Likewise, strong reasoning benchmark performance does not establish reliability across unfamiliar domains, and fluent explanations do not eliminate hallucinations. Evaluation evidence is most informative when it identifies the tested conditions and the distance between those conditions and the intended use.
Governance can therefore treat emergence as a reason to examine specific capabilities rather than as a stand-alone classification of model danger or intelligence. Parameter thresholds are convenient administrative indicators, but they do not substitute for task-level evidence. Post-training changes, newly available tools, or a more effective prompting procedure can alter measured capabilities without changing parameter count. Documenting model versions, evaluation methods, access conditions, and known failure modes makes such changes easier to distinguish from genuine discontinuities in learning. Continuing evaluation after deployment also captures abilities that were not included in the original test suite or become operationally relevant only through new system integrations.