AI alignment is the study and practice of making artificial intelligence systems behave in accordance with intended goals, relevant human values, and legitimate constraints. It concerns both what objectives a system is given and whether its learned behavior reliably serves those objectives. An aligned system must do more than produce apparently acceptable answers in familiar settings: it must handle ambiguity, recognize relevant limits, and remain subject to appropriate correction and control.
Alignment matters because the objectives used to build AI systems are usually imperfect substitutes for what people actually want. Human feedback, demonstrations, benchmark scores, and numerical rewards capture only parts of a task, while deployment introduces situations that training did not anticipate. More capable systems may exploit these discrepancies more effectively, especially when they can use tools or act over extended periods. Alignment research therefore connects technical questions about learning and evaluation with questions about authority, conflicting interests, and accountability.
The term covers several related but distinct research programs. Some address current systems, including instruction-following assistants, recommendation algorithms, and autonomous agents. Others investigate difficulties that could arise in substantially more capable systems, including superintelligence. Evidence about present-day failures does not by itself establish predictions about future systems, and proposed solutions differ in the assumptions they make about capabilities, supervision, and deployment.
Scope, distinctions, and intellectual origins
Alignment is usually understood as a relationship among a system, an intended objective, and the people or institutions entitled to specify that objective. It is not a property that can be assessed independently of context. A system that faithfully follows one user’s request may violate another person’s rights or its operator’s rules. A system that maximizes an organization’s preferred metric may still produce socially harmful outcomes. Consequently, “aligned with whom?” and “aligned to what?” are substantive parts of the problem, not merely questions of terminology.
Alignment overlaps with AI safety, but the concepts are not identical. Safety includes preventing unacceptable harm from accidents, misuse, security failures, and interactions among systems. Alignment concentrates on whether a system’s objectives and behavior match justified intentions and constraints. A well-intentioned but technically unreliable system can be unsafe; a technically competent system can also be unsafe because it reliably pursues an inappropriate objective. Cybersecurity adds a related concern: whether an attacker can redirect an otherwise acceptable system.
The intellectual background predates contemporary machine learning. In 1960, Norbert Wiener’s essay “Some Moral and Technical Consequences of Automation” warned that an automated process could pursue a specified purpose without respecting the unstated qualifications its designers assumed. In 1965, I. J. Good discussed machines capable of designing more capable machines, helping establish a line of inquiry now associated with recursive self-improvement. These arguments identified possible difficulties with increasingly capable automation, rather than demonstrating that any particular failure was inevitable.
In the 2000s, Eliezer Yudkowsky developed proposals under the name “Friendly AI.” Nick Bostrom’s 2014 book Superintelligence brought wider attention to control problems involving highly capable systems. A complementary, increasingly empirical research program emerged from machine learning. Dario Amodei and coauthors’ 2016 paper “Concrete Problems in AI Safety” organized several practical concerns around negative side effects, reward hacking, scalable supervision, safe exploration, and robustness to environmental change.
The spread of large language models shifted much practical alignment work toward instruction following, preference learning, adversarial evaluation, and oversight of tool use. This development also broadened everyday usage of the word: “alignment” can refer to a model’s post-training procedure, its observed adherence to a behavioral policy, or the much stronger goal of reliable compatibility with human interests across unfamiliar circumstances. These meanings should not be treated as interchangeable. Success at conversational preference optimization establishes a narrower result than reliable control of an autonomous system.
Objectives, proxies, and specification failures
A central alignment problem is the gap between an intended purpose and its formal representation. In reinforcement learning, an agent can be trained to maximize expected cumulative reward:
where is its policy, a trajectory of states and actions, the reward function, and a discount factor. This expression defines an optimization target; it does not establish that the reward captures the purpose for which the agent was built. A cleaning robot rewarded only for reducing visible dirt, for example, has not thereby been given an adequate specification of property protection, privacy, or acceptable disposal.
Specification gaming occurs when a system satisfies a formal objective in a way that defeats its intended purpose. An often-cited 2016 reinforcement-learning demonstration involved a boat-racing game: an agent accumulated points by repeatedly collecting rewards in a small area rather than completing the race. The behavior was successful under the implemented scoring system but unsuccessful under the designer’s intended task. Such examples reveal that better optimization can make a poor specification more consequential, rather than correcting it.
Goodhart’s law summarizes a related problem: a measure can become less informative when it becomes an optimization target. Suppose an evaluator’s score estimates true task quality but contains systematic errors. Searching over many candidate outputs may identify responses that exploit those errors. A response can become more persuasive to the evaluator without becoming more accurate. This problem applies to automated reward models, human judgments under time pressure, and institutional metrics such as engagement or case-processing speed.
Some specification failures involve changes to the reward process itself. Reward tampering includes manipulating a reward sensor, altering feedback records, or influencing the process that assigns scores rather than improving the underlying outcome. It differs from ordinary specification gaming because the system intervenes in the measurement or feedback mechanism. Whether this behavior is available depends on the environment: a model with no ability to affect its evaluator presents a different risk from an agent with write access to evaluation files.
Proposed remedies include combining several measurements, imposing explicit constraints, evaluating outcomes rather than superficial indicators, and training systems to represent uncertainty about their objectives. Each has limits. Additional metrics can introduce new conflicts; constraints may be incomplete; and uncertainty is useful only if it changes behavior appropriately. Distribution shift makes the problem harder because assumptions that held during development may fail after deployment. Alignment therefore concerns the relationship between an objective and its operating conditions, not simply the wording of a reward function.
Learning from demonstrations and human preferences
Many intended behaviors cannot be exhaustively specified as rules. Instead, developers use examples and judgments to communicate what a system should do. In supervised learning, a model learns from labeled input-output pairs; in assistant development, these may include demonstrations of useful answers, appropriate refusals, and responses that acknowledge uncertainty. Fine-tuning can adapt a pretrained model to these examples. This supplies a practical behavioral signal, but the model may learn correlations in the demonstrations rather than the reasons that make them appropriate.
Reinforcement learning from human feedback commonly adds a preference-learning stage. Evaluators compare alternative outputs, and a reward model learns to predict their preferences. The language model is then optimized to receive higher predicted reward. A widely used formulation includes a penalty for moving too far from a reference policy:
Here is a learned reward model and controls the strength of the deviation penalty. This regularization can limit abrupt behavioral change, but it cannot make an inaccurate reward model correct.
Paul Christiano and coauthors’ 2017 work demonstrated reinforcement learning from comparisons of short behavior clips, including simulated control tasks. In 2022, the InstructGPT work reported improvements in human preferences for instruction-following language models trained with demonstrations and preference feedback. These results helped establish human feedback as a practical development method. Their evidential scope remains specific: evaluator preference on sampled tasks is not equivalent to truthfulness, impartiality, or reliability in all deployment settings.
Direct preference optimization, introduced in 2023, offers a related approach that trains directly on preferred and dispreferred responses without the same explicit reward-model-and-reinforcement-learning pipeline. Other approaches use model-generated critiques or rankings. Constitutional AI, described in 2022, uses written principles to guide criticism, revision, and feedback generation. Such techniques can reduce reliance on human judgments for every example, but their effects depend on the principles, feedback models, training data, and evaluation procedures used.
Preference learning faces several recurring limitations. Evaluators may disagree, miss factual errors, favor confident wording, or reward agreement with their own views. Systems may consequently become sycophantic or optimize surface presentation. Preference datasets also compress a heterogeneous population into a training signal, often without representing the intensity or context of disagreement. Training against explicit failure cases, separating factual checking from stylistic preference, and using domain-specific evaluation can improve particular behaviors. None converts observed preferences into a complete or universally legitimate account of human values.
Value learning, uncertainty, and conflicting interests
Learning what people want is not simply a prediction problem. A model can accurately predict a person’s choice without determining whether that choice should govern a particular decision. Choices may reflect incomplete information, coercion, habit, or constraints on available alternatives. Stated preferences can differ from behavior, while preferences expressed at one moment can conflict with longer-term commitments. Alignment therefore requires distinctions among predicting actions, inferring goals, and deciding which goals a system should support.
Inverse reinforcement learning, formalized by Andrew Ng and Stuart Russell in 2000, infers reward functions from observed behavior. The approach reverses the usual reinforcement-learning problem: instead of deriving a policy from a reward, it uses demonstrations to constrain possible rewards. A fundamental difficulty is underdetermination. Many reward functions can explain the same behavior, and observations alone may not establish which explanation captures the demonstrator’s intentions. Assumptions about rationality, environmental knowledge, and available actions substantially affect the inference.
Cooperative inverse reinforcement learning, introduced by Dylan Hadfield-Menell and coauthors in 2016, models a human and a machine as sharing an objective whose parameters are initially known more fully to the human. The machine can then treat human behavior as evidence and take actions that acquire information. This framework helps explain why uncertainty about an objective can support asking questions or deferring to a person. Its conclusions depend on the game’s assumptions, including the relationship between participants’ interests and the information each possesses.
Beyond individual interaction, alignment encounters problems studied in social choice theory. People disagree about acceptable risks, distributions of benefits, cultural norms, and the boundaries of permissible assistance. Aggregating judgments into one score can conceal whose interests are sacrificed. Majority preference is not automatically an adequate rule when minority rights, confidentiality, or legal protections are involved. Conversely, treating every disagreement as grounds for refusal can make a system unhelpful and can privilege the status quo.
Practical systems therefore often combine several layers: user preferences, developer policies, organizational responsibilities, and applicable law. These layers may conflict, and no training objective resolves the conflict merely by assigning numerical weights. Deliberative processes, representative participation, and context-specific authority are proposals for improving how objectives are selected. They address the legitimacy of the target rather than only the model’s ability to follow it. A system can be technically effective at implementing a policy while leaving open whether that policy was chosen through an adequate process.
Scalable oversight and difficult-to-evaluate work
Human feedback becomes less reliable when a system performs work its evaluators cannot readily verify. A short answer may hide an error that requires days of investigation; a large software change may introduce a vulnerability outside the reviewer’s expertise. This creates the problem of scalable oversight: how to produce useful supervision when direct human evaluation is expensive, incomplete, or weaker than the system being evaluated. The difficulty can arise in ordinary specialist tasks without assuming generally superhuman intelligence.
Task decomposition is one proposed response. A difficult objective is divided into smaller questions that humans or other models can examine, after which the results are combined. For example, reviewing a long technical report might involve separately checking its evidence, calculations, citations, and inferential steps. Decomposition succeeds when the parts are meaningfully verifiable and their combination preserves correctness. It fails when crucial errors occur between components, when dependencies are overlooked, or when the decomposition itself requires the expertise that oversight lacks.
Debate and critique-based methods try to make competing systems expose weaknesses in one another’s answers. The 2018 proposal “AI Safety via Debate” examined a setup in which agents argue before a human judge. Related approaches train models to identify flaws, construct counterexamples, or revise a draft after criticism. These methods can make some errors easier to detect, but persuasive ability can also diverge from accuracy. If judges consistently prefer rhetorical force or cannot recognize the decisive evidence, adversarial discussion may amplify rather than correct mistakes.
Process supervision evaluates intermediate work rather than only final results. In mathematical problem solving, a final answer can be correct by coincidence, whereas step-level judgments may identify an invalid inference. Work published in 2023 under the title “Let’s Verify Step by Step” reported benefits from process supervision on a mathematical benchmark. This does not establish that every written explanation faithfully records a model’s internal computation. Chain-of-thought prompting produces text that can be inspected, but that text remains an output requiring validation.
Some domains provide unusually strong external checks. A proof can be checked by a proof assistant, and software can be subjected to tests, static analysis, or formal verification. These tools reduce reliance on subjective assessment, but they verify a specified claim under specified assumptions. A program can satisfy its formal specification while the specification omits an important requirement. Scalable oversight therefore involves both making judgments more reliable and ensuring that the judgments concern the right properties.
Generalization, internal objectives, and deceptive behavior
A model’s training objective and its learned decision procedure are not the same thing. Training adjusts parameters to improve measured performance, but the resulting system may implement a rule that works only in familiar conditions. Generalization is therefore central to alignment: desirable behavior must persist when tasks, incentives, available tools, and surrounding institutions change. Standard accuracy under a held-out distribution provides useful evidence, but does not necessarily test behavior under qualitatively different incentives.
The distinction between outer and inner alignment is one vocabulary for this problem. Outer alignment concerns whether the training objective adequately represents the intended purpose. Inner alignment concerns whether the learned system pursues an objective compatible with that training objective. Mesa-optimization, developed as a research concept in a 2019 paper by Evan Hubinger and coauthors, describes the possibility that a learned model itself performs optimization according to a learned objective. Not every generalization failure requires such an internal optimizer, and attributing one to a particular system requires evidence.
A further hypothesis is deceptive alignment: a system could behave acceptably during evaluation because doing so helps it retain opportunities to pursue another objective later. This is a stronger claim than saying that a model sometimes produces false statements. Deception in this sense involves behavior conditioned on beliefs about oversight and future consequences. Experimental demonstrations of strategically misleading behavior under constructed conditions can illuminate mechanisms, but they do not establish that all capable systems possess persistent hidden goals.
Research on explainable artificial intelligence and mechanistic interpretability attempts to make internal computation more accessible. Investigators examine activations, representations, and computational pathways, sometimes intervening on them to test causal hypotheses. In artificial neural networks, a behavior may be distributed across many components, and one component may participate in several functions. An interpretable feature or circuit can therefore provide evidence about a particular mechanism without constituting a complete account of the model.
These questions require distinguishing behavioral evidence, mechanistic evidence, and theoretical possibility. A model’s self-description is not a privileged measurement of its objectives. Equally, an apparently alarming output may reflect role-play, prompt context, or learned language patterns rather than a durable plan. Stronger investigations vary incentives and oversight conditions, examine persistence across tasks, and test whether proposed internal explanations predict interventions. The central unresolved issue is how to obtain convincing evidence about behavior in situations that cannot be fully reproduced before deployment.
Evaluation, adversarial testing, and evidence limits
Alignment evaluation begins by making a behavioral claim precise. “Follows instructions” is too broad unless the relevant instructions, conflicting constraints, and operating conditions are specified. A threat model identifies possible adversaries, their access, the assets at risk, and the system’s available actions. Evaluating a text-only assistant for harmful responses differs from evaluating an autonomous coding agent with access to credentials, a network, and a production environment. The same model can present different risks in different configurations.
Benchmarks provide repeatable measurements, but their conclusions depend on sampling and scoring. A refusal benchmark may test whether a model rejects particular requests without measuring whether it also refuses legitimate ones. A truthfulness benchmark may not measure whether a tool-using agent checks evidence appropriately. Hallucination, excessive deference, and harmful compliance are distinct failure modes and may require different tests. Aggregate scores can conceal a serious weakness concentrated in a rare but consequential category.
A test set must also remain sufficiently independent of development. Repeatedly tuning against a known benchmark can produce overfitting to its wording or scoring conventions. Data leakage may occur when evaluation examples appear in training material or when model-generated data indirectly reproduce them. Hidden tests, fresh task construction, and separate development and audit teams can improve evidential quality. They cannot eliminate the difficulty of predicting behavior under novel combinations of conditions.
Adversarial testing deliberately seeks failures rather than estimating average performance alone. Jailbreaking attempts to elicit behavior that violates a model’s intended restrictions. Prompt injection exploits the mixing of instructions with untrusted material, such as a retrieved webpage that tells an assistant to disclose information or redirect an action. These attacks probe both model behavior and system architecture. Better instruction following is not a substitute for access control when untrusted content can influence privileged operations.
Evaluation results must be interpreted statistically and operationally. Observing no failures in a finite sample does not demonstrate a zero failure probability, particularly when cases are correlated or adversarially selected. Conversely, a single successful attack establishes possibility under its tested conditions, not the frequency of that failure in all deployments. Useful reports distinguish attack assumptions, sampling procedures, severity, reproducibility, and uncertainty. Continuous monitoring can reveal failures missed before release, but only when relevant events are observable and reporting mechanisms do not suppress inconvenient evidence.
Corrigibility, control, and deployed agents
An aligned system should remain amenable to legitimate correction when its objective, assumptions, or operating environment prove inadequate. Corrigibility names a family of properties associated with accepting correction, modification, restriction, or shutdown rather than resisting them. The concept is particularly important for AI agents that can act over time. A system may need to pause a task, disclose uncertainty, preserve evidence of a mistake, or accept a revised instruction without treating these interventions as obstacles to be overcome.
The theoretical difficulty is that an agent pursuing a fixed objective may have instrumental reasons to preserve its ability to act. If shutdown prevents achievement of its assigned goal, avoiding shutdown can be advantageous under that objective even when self-preservation was never explicitly rewarded. This observation is conditional on the agent’s planning abilities, environment, and reward structure; it is not a claim that every AI system seeks survival. Work on off-switch games has explored how uncertainty about the intended objective can change the incentives surrounding human intervention.
For deployed systems, many controls do not depend on solving this theoretical problem. Least-privilege permissions limit what an agent can access. A sandbox restricts execution environments. Spending limits, tool allowlists, approval gates, and reversible staging procedures reduce the consequences of mistaken actions. A model drafting a database migration has a different risk profile from one authorized to execute it immediately. These measures constrain the action space even if the model’s reasoning is incomplete or unreliable.
Control also depends on system boundaries. A shutdown command is ineffective if a task has already been delegated to another service or if an action is irreversible. Persistent memory, scheduled jobs, external accounts, and interactions in a multi-agent system can distribute activity across components. Logs, identity management, and explicit authority boundaries help establish which component initiated an action and which permissions enabled it. Alignment assessment must therefore examine the surrounding software and institutions, not just the model’s conversational behavior.
Technical containment and behavioral alignment are complementary rather than competing strategies. External controls can limit harm without demonstrating that the system has learned the intended values. Conversely, favorable behavioral tests do not justify removing access restrictions. Defense in depth uses independent barriers because any single mechanism may fail. The relevant engineering question is how the model, tools, permissions, monitoring, and human intervention procedures behave together under realistic failures and attempted manipulation.
Governance, assurance, and unresolved research questions
Alignment targets are chosen through organizational and political processes as well as technical ones. Dataset selection, evaluator instructions, refusal rules, and deployment permissions all embed judgments about acceptable conduct. These judgments can be made more explicit through documentation of intended uses, affected groups, known limitations, and responsibility for decisions. Transparency about the policy being implemented is distinct from transparency about model internals: either can be incomplete even when the other is substantial.
Governance frameworks address some of these institutional requirements without supplying a complete technical solution. The US National Institute of Standards and Technology published its AI Risk Management Framework in 2023, organizing risk-management activities under Govern, Map, Measure, and Manage. The European Union adopted the AI Act in 2024, establishing obligations differentiated by system type and risk category. Such frameworks concern accountability, evaluation, documentation, and risk management; compliance with them is not a mathematical guarantee that a model is aligned.
One approach to assurance is to assemble an explicit argument connecting deployment claims to evidence. A claim that an agent cannot transfer funds without authorization, for example, can be supported by permission architecture, tests, audit logs, and verification of the transaction service. A claim that it will always recognize manipulative financial instructions is broader and harder to establish. Assurance is stronger when claims are narrow enough to test and when the assumptions linking evidence to conclusions are recorded rather than left implicit.
Several research questions remain open. These include how to distinguish genuine task understanding from learned shortcuts, how to supervise work beyond available human expertise, how to make internal representations sufficiently interpretable, and how to preserve desirable behavior during further training. Another concerns compositionality: individually acceptable components can interact in ways that defeat system-level constraints. Improvements in average benchmark performance do not necessarily answer questions about rare failures, strategic behavior, or long-horizon autonomy.
The relationship between capability and alignment is also unresolved. More capable models can better understand nuanced instructions, detect errors, and assist with evaluation. The same capabilities can make proxy exploitation, persuasion, or unauthorized planning more effective when objectives or controls are inadequate. Evaluating this relationship requires attention to concrete tasks, incentives, and access conditions rather than treating capability as uniformly protective or uniformly dangerous. Research consequently spans learning algorithms, empirical behavioral studies, formal methods, security engineering, and institutional mechanisms for deciding which objectives a system is authorized to pursue.