An adversarial example is an input deliberately constructed or modified to cause a machine-learning system to produce an incorrect or otherwise undesired output. In a common formulation, an attacker makes a small change to an input that preserves its correct interpretation while changing the model’s prediction. The concept also includes conspicuous modifications, such as patches placed within an image, when these satisfy the specified attack constraints. Adversarial examples are studied both as a cybersecurity concern and as evidence about the reliability and limitations of learned representations.
Definition and mathematical formulation
For a classifier with parameters , let be an input with correct label . A standard adversarial example is an input satisfying
where defines the allowed modifications. Usually, the original input is required to be classified correctly, so that an existing error is not counted as an attack-induced failure.
In an untargeted attack, any incorrect prediction meets the attacker’s objective. In a targeted attack, the attacker seeks a particular output :
For regression, text generation, and other tasks, success is defined by a task-specific objective rather than simply by an incorrect class label.
A common construction maximizes a loss function within a bounded region:
For images, the allowed set is often defined using a norm:
where is the valid input domain and is a perturbation budget. Common choices include:
- bounds, which limit the largest absolute change to any input coordinate.
- bounds, which limit the Euclidean magnitude of the perturbation.
- constraints, which limit the number of modified coordinates. The quantity is conventionally called a norm in this context, although it is not mathematically a norm.
These constraints make attacks measurable and comparable, but they are not complete models of human perception. A numerically small perturbation need not preserve meaning, and a perceptually harmless change need not be small under a chosen norm.
Historical development
Research on adversarial manipulation of learning systems predates modern deep learning. Earlier work examined how attackers could evade spam filters, malware detectors, and other classifiers by modifying selected input features.
A major turning point was the work of Christian Szegedy and collaborators, first circulated in 2013 and presented in 2014. They demonstrated that small, carefully optimized changes could cause neural networks to misclassify images. Some examples also transferred between models, challenging the assumption that such failures were merely isolated numerical accidents.
Work by Ian Goodfellow, Jonathon Shlens, and Christian Szegedy, circulated in 2014 and presented in 2015, introduced the fast gradient sign method and emphasized the role of approximately linear behavior in high-dimensional input spaces. Subsequent research developed stronger iterative attacks, physical demonstrations, adversarial training, and methods for certifying restricted robustness properties.
Although image classification supplied many influential demonstrations, adversarial examples have also been investigated in speech recognition, text processing, object detection, and other learning tasks.
Threat models and attack categories
An adversarial example is meaningful only relative to a threat model. This specifies what an attacker knows, what can be changed, how much computation or querying is available, and what outcome counts as success.
Attacker knowledge and access
White-box attacks assume access to model details, commonly including parameters, architecture, preprocessing, and gradients.
Black-box attacks assume more limited access. A service might expose class probabilities, numerical scores, or only a final decision through an application programming interface. Attackers may search using these responses or construct examples on a substitute model.
Transfer attacks rely on an example generated for one model also affecting another. Transferability varies with the models, task, attack method, and perturbation constraints; it is not guaranteed.
Knowledge and query access are separate dimensions. An attacker can know a model’s architecture without knowing its parameters, or know its parameters while still facing restrictions on modifying the deployed system’s inputs.
Digital and physical attacks
Digital attacks directly modify the representation supplied to a model, such as image values or an audio waveform.
Physical attacks modify an object or environment that is subsequently recorded by a sensor. Printed patterns, altered signs, and wearable textures are examples studied in computer vision. Such attacks must survive some combination of changes in lighting, viewpoint, distance, sensor noise, and image processing.
An adversarial patch is a localized pattern intended to influence predictions when placed within a scene. Unlike the classic nearly imperceptible perturbation, a patch can be plainly visible.
Instance-specific and universal perturbations
An instance-specific perturbation is optimized for one input. A universal perturbation is intended to affect many inputs without being individually redesigned for each one. Universality is generally evaluated over a specified input distribution and does not imply success on every possible input or model.
How adversarial examples arise
A classifier divides its input space into regions associated with different predictions. An adversarial example can occur when an allowed modification moves an input across a decision boundary while leaving the task-relevant interpretation unchanged.
High dimensionality helps explain why individually small changes may have a substantial combined effect. For a linear score , a perturbation
changes the score by
Thus, many small, aligned coordinate changes can produce a large score change. This illustrates an important mechanism but is not a complete explanation of adversarial vulnerability in nonlinear models.
A broader explanation concerns the features learned from data. Some features can be predictive of labels without matching the stable, semantic cues people expect a system to use. A model may consequently achieve high ordinary test accuracy while remaining sensitive to carefully chosen changes.
Adversarial vulnerability is therefore not simply equivalent to overfitting. It also concerns the distinction between average-case generalization and robustness to a structured, deliberately selected set of inputs. There is no single explanation that accounts equally well for every model, modality, and attack setting.
Construction methods
Gradient-based attacks
The fast gradient sign method constructs an untargeted perturbation using a single gradient calculation:
where keeps the result within the valid input domain. Gradients are commonly computed using backpropagation.
Iterative attacks repeatedly update the input. A standard approach is projected gradient descent, which performs gradient ascent on the attack loss and projects each update back into the permitted region:
This sign-based update is commonly used for -bounded attacks. Other constraint sets call for different update rules. Random initializations and multiple restarts can help find failures missed by a single optimization run.
Query-based and discrete attacks
When gradients are unavailable, attacks can estimate useful directions from output scores or search using only decision feedback. Query limits and the information returned by the system strongly influence feasibility.
For text, arbitrary numerical perturbations do not generally correspond to valid language. Attacks may instead modify characters, substitute words, insert phrases, or restructure sentences. The effects of tokenization and the requirement to preserve meaning make text-based constraints substantially different from pixel-distance constraints.
Defenses and robustness evaluation
Adversarial training
Adversarial training incorporates adversarially modified inputs during learning. A common robust-optimization objective is
The inner problem searches for harmful modifications; the outer problem adjusts the model to reduce their effect. In practice, the inner maximum is approximated by an attack.
Adversarial training can substantially improve robustness within a chosen setting, but it increases computational cost and can reduce accuracy on unmodified inputs. Robustness learned for one perturbation family does not automatically extend to different norms, physical transformations, or semantic changes.
Certified robustness
Some methods establish guarantees rather than relying only on unsuccessful attacks. Formal verification techniques can prove properties of particular networks over specified input regions, although scalability and the complexity of supported architectures remain important constraints.
Randomized smoothing defines a classifier through predictions under added random noise and can provide certified robustness radii under specified assumptions. Such certificates apply to the stated perturbation model; they are not guarantees against every possible attack.
Misleading apparent robustness
A defense may appear robust because it obstructs a particular attack rather than because the model’s decisions are stable. This problem is often called gradient masking or gradient obfuscation. Non-differentiable preprocessing, saturated computations, and randomness can make ordinary gradient attacks fail without eliminating adversarial examples.
Credible evaluation therefore distinguishes between failure to find an attack and proof that no allowed attack exists. Relevant reporting details include the perturbation budget, input scaling, attack iterations, restarts, query access, and whether the attack accounts for the defense’s full computation.
Robust accuracy generally measures the fraction of evaluated examples correctly classified under the specified adversarial condition. Empirical robust accuracy obtained with a finite attack is not the same as certified robust accuracy, and results are not directly comparable when their threat models differ.
Relationship to other concepts
Adversarial examples are usually associated with manipulation at inference time. By contrast, data poisoning changes training data or the training process to influence the learned model. Poisoning can also implant a backdoor that is later activated by a specially constructed inference-time input.
They also differ from ordinary distribution shift. Changes in weather, recording equipment, or language use can degrade performance without an attacker. Adversarial inputs may exploit related weaknesses, but deliberate construction and an explicit adversarial objective distinguish the standard setting.
In large language models, adversarially constructed inputs may overlap with jailbreaking and prompt injection. The terms are not interchangeable: jailbreaking concerns bypassing behavioral restrictions, while prompt injection concerns untrusted content being treated as instructions. Neither necessarily involves a small, meaning-preserving perturbation.
Despite the shared word “adversarial,” generative adversarial networks are a training framework involving competing models, not a synonym for adversarial examples.
Applications and limitations
Adversarial examples serve as stress tests for recognition systems, content filters, and other automated decision processes. They also provide experimental tools for studying learned features, decision boundaries, and the gap between benchmark accuracy and reliability under deliberate manipulation.
Their practical significance depends on the surrounding system. An attack that assumes unrestricted access to a model’s input array may not be realizable through a physical sensor. Conversely, realistic attackers may be able to make conspicuous or semantic changes that fall outside the small-norm constraints used in laboratory evaluations.
A valid robustness claim must therefore specify the attacker’s capabilities, the meaning that must be preserved, the permitted modifications, and the evaluated task. Success or failure under one such specification does not establish security under all others.