Grokking is a phenomenon in machine learning in which a model first achieves very high performance on its training data, continues training for an extended period with poor performance on unseen examples, and then develops strong generalization. The improvement on unseen data can be relatively abrupt, even though the model’s parameters have been changing throughout training. Grokking is studied primarily as a feature of learning dynamics: it shows that fitting the training examples and acquiring a broadly applicable solution can occur on substantially different timescales.
Origin and terminology
The phenomenon was named in the paper Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, by Alethea Power and colleagues, publicly released in 2022. The experiments examined neural networks learning tasks with precise underlying rules, including arithmetic operations on finite sets. Some networks fitted their training examples well before learning solutions that worked reliably on withheld examples.
The name derives from grok, a word introduced in Robert A. Heinlein’s 1961 novel Stranger in a Strange Land, meaning to understand something deeply. In machine-learning research, however, grokking describes an observed training trajectory, not a demonstrated psychological experience or a claim that a model understands a task in a human sense.
The expression “delayed generalization” captures its central feature. Terminology varies across studies: some reserve grokking for a sharp improvement following a long interval of apparent overfitting, while others apply it more broadly to markedly delayed improvements on held-out data.
Characteristic learning dynamics
A typical grokking experiment uses supervised learning with a fixed training dataset and a separate test set. Its learning curves display three broad stages:
- Initial fitting. Training performance improves as the model learns to predict the examples it has seen.
- An extended generalization gap. Training accuracy is near perfect, but accuracy on withheld examples remains low. This period is often described as memorization, although training accuracy alone does not establish the model’s internal mechanism.
- Delayed generalization. Accuracy on withheld examples rises substantially while training accuracy remains high.
If and denote training and test accuracy after optimization steps, the characteristic separation is
where is the time at which a chosen training-performance threshold is reached and is the time at which a chosen held-out-performance threshold is reached. These thresholds are experimental conventions, not universal constants defining the phenomenon.
Accuracy can also make a transition look sharper than it is internally. In classification, a small change in predicted scores can move many examples across a decision boundary. Measures such as held-out loss, confidence, parameter norms, and internal representations can therefore reveal changes that precede an apparent jump in accuracy.
A canonical example: modular arithmetic
A widely studied setting is modular addition. The model receives two integers and and predicts
where is a fixed modulus. Inputs are often represented as tokens and processed by a small Transformer or another neural network.
There are possible ordered input pairs. An experiment uses a subset of these pairs for training and reserves other pairs for evaluation. A sufficiently expressive model can fit the observed input–output pairs without initially producing correct answers for the remaining pairs. With continued optimization, it may develop a representation of the arithmetic structure that works across the input space.
This setting is useful because the correct rule is known and the finite input space can be evaluated extensively. Researchers can distinguish performance on particular examples from performance across the underlying task.
Success on withheld pairs nevertheless establishes only a specific kind of generalization. A model trained for one modulus need not handle another modulus, different input formats, or longer arithmetic expressions. Grokking within a fixed task is not automatically general-purpose mathematical reasoning.
Explanations and mechanisms
No single explanation has been established for every instance of grokking. Different models and tasks can exhibit similar learning curves for different reasons.
Competition between memorization and structured solutions
One influential account distinguishes solutions that fit individual examples from solutions that exploit a shared rule. A network may initially reach an example-specific solution because it is easier for the optimization process to find. A more structured solution may develop slowly and eventually become dominant.
This account connects grokking to inductive bias: the preferences introduced by architecture, parameterization, initialization, and optimization. A dataset may admit many functions that fit the training examples, but these functions need not be equally favored by the training procedure.
Regularization and parameter scale
Regularization, particularly weight decay, plays an important role in many canonical experiments. A schematic regularized objective is
where denotes the model parameters, the prediction loss, and the regularization strength.
Even after training accuracy reaches its maximum, this objective can continue to change. The prediction loss may decrease further, and the penalty can favor lower-norm parameter configurations. In some settings, structured solutions are favored over solutions that retain many example-specific associations.
Explicit norm regularization and the decoupled weight decay used by some adaptive optimizers are not generally identical, although both are relevant to grokking experiments. Moreover, explicit weight decay is not a universal requirement: grokking-like behavior has also been observed without it under suitable conditions.
Internal representation changes
Research on representation learning and mechanistic interpretability examines what changes inside a model during the delay. In some modular-addition Transformers, researchers have identified representations based on periodic features related to the discrete Fourier transform. These features can support a structured computation of addition modulo .
Such analyses can distinguish the gradual construction of a useful computational mechanism from the later point at which it dominates the model’s outputs. They also limit purely behavioral interpretations: a sudden improvement in accuracy need not mean that a complete algorithm appeared in a single optimization step.
Fourier-based mechanisms are findings about particular tasks and models, not a general definition of grokking.
Factors affecting its occurrence
Whether grokking occurs, and how long it takes, depends on interacting experimental choices:
- Training-set size and structure. Too little data may not sufficiently constrain a generalizable solution. More data can reduce the delay or eliminate a distinct overfitting phase.
- Model architecture and capacity. Capacity affects which example-specific and structured solutions are available and how readily they can be learned.
- Initialization and parameter scale. Different starting conditions can alter the relative timescales of fitting and generalization.
- Optimization settings. The learning rate, optimizer, batch size, and regularization strength can change the trajectory.
- Task structure. Tasks with compact underlying rules provide particularly clear settings for studying the phenomenon.
- Training duration. An experiment may end before a delayed transition becomes visible.
These dependencies make grokking a property of a training setup rather than an intrinsic property of an architecture alone. Different random seeds can also produce different transition times or outcomes.
Research significance and limitations
Grokking provides a counterexample to the assumption that useful learning necessarily ends when training accuracy saturates. It also complicates a simple picture in which additional training after overfitting can only worsen held-out performance. In suitable settings, continued optimization can instead produce a much better generalizing solution.
This does not invalidate early stopping. Early stopping is a model-selection procedure, whereas grokking is an empirical pattern that occurs under particular conditions. Many models never undergo such a transition, and extended training can consume substantial computation without improving performance.
Reliable evaluation also requires separating exploration from final assessment. If researchers repeatedly inspect a held-out set to choose training duration or hyperparameters, that set effectively serves as a validation set. A separate final evaluation helps avoid selection effects and data leakage.
Grokking is also distinct from improvements associated with increasing model size: its defining axis is continued training, often with a fixed architecture and dataset. Nor does it by itself establish robustness under distribution shift. Strong performance on withheld examples generated by the same rule does not imply reliable behavior under different rules, formats, or environments.
The clearest demonstrations often involve small, highly structured tasks. They offer controlled settings for investigating learning mechanisms, but they do not establish that large deployed models will eventually acquire a desired capability merely by training longer.