Offline reinforcement learning is a branch of reinforcement learning in which an agent learns how to make sequential decisions using a previously collected dataset, without interacting with the environment during training. The dataset may contain trajectories generated by people, automated controllers, earlier learning systems, or mixtures of these sources. Its defining problem is to improve decision-making from recorded experience while accounting for the fact that the consequences of alternative actions are only partially observed.
The approach is important wherever collecting new experience is expensive, slow, or potentially harmful. Examples include robotic manipulation, industrial control, recommendation, and research on decisions represented in clinical records. Unlike ordinary supervised learning, offline reinforcement learning must reason about how present actions change future observations and rewards. Unlike online reinforcement learning, it cannot resolve uncertainty by experimenting. Its principal methods therefore combine policy optimization with restrictions on unsupported actions, conservative value estimation, learned environment models, or sequence prediction.
Mathematical formulation and relation to neighboring problems
The standard mathematical setting is a Markov decision process, usually written as . Here, and are the state and action spaces, specifies transition probabilities, is the expected immediate reward, is the initial-state distribution, and discounts future rewards. A policy determines which actions are selected. The usual objective is to maximize expected discounted return:
In the offline setting, the learner receives a fixed collection of transitions or complete trajectories. A transition dataset can be represented as , where records whether a transition ends an episode. The data-generating policy is conventionally called the behavior policy, denoted by . In practice, a dataset may combine many behavior policies, some of which depend on time or on information absent from the recorded state. Consequently, a single stationary is often an analytical simplification rather than a literal description of data collection.
Offline learning is not synonymous with off-policy learning. An off-policy algorithm learns about a target policy using experience generated by another policy, but it may continue collecting fresh experience. For example, deep Q-networks train on a replay buffer while the agent continues interacting with its environment. Offline reinforcement learning removes that continuing source of corrective evidence. Running an otherwise unchanged off-policy algorithm on a frozen buffer is possible, but it can fail because the algorithm repeatedly evaluates actions that the buffer scarcely represents.
Behavior cloning provides another important comparison. It uses supervised learning to predict recorded actions from recorded states, commonly by minimizing negative log likelihood. This objective does not require rewards and seeks to reproduce the behavior rather than improve its long-term return. Offline reinforcement learning instead uses rewards and transitions to identify beneficial decisions, potentially combining successful portions of different trajectories. Neither approach is universally superior: when demonstrations are consistently strong and coverage is narrow, cloning can be an effective baseline.
The temporal structure also distinguishes offline reinforcement learning from a contextual bandit. A contextual bandit evaluates an action’s immediate outcome without modeling how that action changes later decision contexts. Sequential problems require credit assignment across time. A controller that sacrifices immediate reward to reach a more favorable state cannot generally be learned correctly by treating every logged decision as an independent prediction example.
Historical development and major algorithmic milestones
Offline reinforcement learning developed from earlier work on approximate dynamic programming, batch value estimation, and learning control from stored transitions. The term “batch reinforcement learning” predates the recent emphasis on large neural networks. Historically, “batch” has sometimes also described methods that alternate between collecting a batch of experience and updating a policy; contemporary “offline reinforcement learning” usually refers more narrowly to training without additional environment interaction.
An influential early method was least-squares policy iteration, introduced by Michail Lagoudakis and Ronald Parr in 2003. It combined a least-squares approximation to policy evaluation with policy improvement, reusing a fixed set of transitions across iterations. Fitted Q iteration, developed by Damien Ernst, Pierre Geurts, and Louis Wehenkel in 2005, framed successive Bellman updates as supervised regression problems. Its use of regression trees demonstrated that reinforcement learning could exploit flexible prediction methods while repeatedly recycling a batch of experience.
The wider adoption of deep learning made it possible to represent policies and value functions for images, high-dimensional observations, and continuous control. However, strong online results did not automatically translate into reliable offline performance. Deep algorithms could assign large values to unsupported actions and then reinforce those errors through bootstrapping. This helped shift attention from simply reusing data toward explicitly controlling the mismatch between recorded behavior and the learned policy.
Several influential algorithms appeared between 2019 and 2022. Batch-Constrained deep Q-learning, introduced by Scott Fujimoto, David Meger, and Doina Precup in 2019, restricted action selection using a learned approximation to dataset actions. Bootstrapping Error Accumulation Reduction, also introduced in 2019, used distributional constraints to discourage unsupported choices. Conservative Q-Learning, introduced by Aviral Kumar and colleagues in 2020, penalized excessive values for actions beyond those directly supported by data. These methods established policy restriction and value pessimism as central design principles.
Other developments broadened the field beyond constrained Q-learning. Model-based methods such as MOPO and MOReL, both introduced in 2020, used learned dynamics together with mechanisms for limiting model exploitation. Decision Transformer and Trajectory Transformer, introduced in 2021, recast aspects of offline decision-making as sequence modeling. Implicit Q-Learning, presented at ICLR in 2022, showed how policy improvement could be organized around value estimates fitted without maximizing over arbitrary actions during critic training.
Benchmarking also became more standardized. D4RL, introduced in 2020, assembled datasets designed to expose difficulties such as narrow coverage, mixed-quality behavior, and the need to combine trajectory fragments. RL Unplugged, introduced in the same year, provided datasets and evaluation protocols for diverse reinforcement-learning domains. These resources made algorithm comparisons more systematic, while also revealing that results depend strongly on dataset construction and evaluation choices.
Distribution shift, coverage, and limits of identification
The central statistical difficulty is distribution shift between the data used for learning and the situations induced by the learned policy. One useful description is the discounted state–action occupancy distribution,
Training data approximate the occupancy of the behavior policy or a mixture of behaviors, whereas deployment generates observations according to . A policy can therefore move the system into states that were uncommon in training even when its individual actions appear locally reasonable.
The problem is particularly acute in Q-learning. A fitted Bellman target often has the form . The maximum searches over actions whether or not the dataset contains useful evidence about them. If function approximation accidentally assigns a high value to an unsupported action, that value enters the target for a recorded transition. Further updates propagate the mistake backward through the dataset. Policy optimization then preferentially selects precisely the actions with favorable estimation errors. This feedback is often called extrapolation error or bootstrapping error.
Coverage determines which improvements can be justified. If the behavior policy never takes a particular action in a relevant state, the dataset alone may not reveal what that action does. Two environments can agree on every observed transition while assigning very different outcomes to that unobserved action. No learning algorithm can distinguish those environments from the same observations without additional structural assumptions. An apparent failure of optimization may therefore reflect missing information rather than an inadequate optimizer or model architecture.
Theoretical analyses often express coverage through density ratios or concentrability coefficients, involving quantities such as . Large ratios indicate that the target policy relies on events poorly represented in the data. Some guarantees assume coverage only for a suitable comparator policy rather than for every possible policy. Their meaning depends on the precise function class, horizon, and error measure. A guarantee stated under bounded ratios does not automatically apply to a high-dimensional dataset with unknown or highly concentrated behavior.
Dataset size and useful coverage are consequently different properties. A million nearly identical trajectories may provide excellent estimates near one operating regime while offering little evidence about alternatives. Conversely, diverse but moderately sized data can support useful recombinations of decisions. In navigation, one group of trajectories might connect a starting area to an intersection and another connect that intersection to a destination. Learning a complete route requires compatible state information and connecting transitions; it does not permit inventing a shortcut through an unobserved region.
Observational records introduce another limitation. If the recorded state omits information that influenced both action selection and outcomes, the apparent action–outcome relationship can be misleading. This is related to causation and hidden confounding, not merely to neural-network prediction error. Many standard offline reinforcement-learning analyses assume a sufficiently informative state representation and therefore do not resolve this problem.
Behavior constraints and policy regularization
One family of methods limits how far the learned policy can depart from recorded behavior. Its motivation is not that the behavior policy is necessarily optimal, but that predictions are usually better supported near observed actions. A generic regularized objective is
where measures a discrepancy and controls the strength of the restriction. Related algorithms impose a hard constraint instead of a penalty. The particular discrepancy determines which departures are discouraged.
A likelihood-based penalty favors actions that a fitted behavior model considers probable. Other approaches compare distributions using kernel-based distances or constrain sampled actions directly. These alternatives are not interchangeable. Requiring the new policy to reproduce the behavior distribution closely can prevent it from concentrating on the best actions in a mixed-quality dataset. A support-oriented restriction instead attempts to exclude unsupported actions while permitting substantial redistribution among actions that the data represent.
Batch-Constrained deep Q-learning illustrates a generative approach to action restriction. In its continuous-control formulation, a variational autoencoder models actions conditioned on states. The algorithm samples candidate actions from this model, permits a bounded perturbation, and selects a candidate using learned action values. The generative model approximates which actions are plausible under the dataset, while the critic distinguishes their expected returns. Its effectiveness depends on the behavior model: omitting useful action modes can make the policy unnecessarily restrictive, while generating unsupported actions can undermine the constraint.
TD3+BC, introduced in 2021, demonstrated a simpler design. It modifies the actor objective of a continuous-control algorithm by adding a behavior-cloning term, commonly a squared distance between the actor’s action and the dataset action. With suitable normalization and scaling, this relatively small modification produced competitive benchmark results. It also highlighted the importance of strong, carefully tuned baselines: elaborate uncertainty models or distribution estimators are not automatically required to improve on naive offline training.
Advantage-weighted regression provides a complementary route. Instead of generating arbitrary actions and estimating their quality, it increases the likelihood of recorded actions according to estimated advantages. A typical objective weights the log likelihood of a dataset action by , with temperature . High-advantage actions receive greater influence, allowing the policy to favor better decisions within the observed action distribution. Weight clipping and normalization are common because exponential weights can concentrate learning on a small number of samples.
These methods face a recurring trade-off. Strong regularization reduces reliance on unsupported predictions but may preserve weaknesses in the dataset. Weak regularization allows greater improvement but increases exposure to extrapolation error. Moreover, action-level similarity alone cannot guarantee trajectory-level safety: a small change at each step can accumulate into a substantial change in visited states. Policy constraints address an important source of error without removing the need to examine long-horizon consequences.
Conservative value learning and implicit policy improvement
A second major family modifies value estimation rather than relying only on an explicit behavioral constraint. The central idea is pessimism: when evidence is weak, the algorithm should avoid assigning values that make uncertain actions look artificially attractive. This differs from the optimism often used to encourage exploration in online reinforcement learning. With no opportunity to test an optimistic guess, an offline learner has less reason to treat uncertainty as evidence of opportunity.
Conservative Q-Learning (CQL) adds a penalty to a Bellman-fitting objective that pushes down values of broadly sampled actions relative to values of dataset actions. For discrete actions, one representative regularizer is
The log-sum-exp term emphasizes high-valued actions, while the second term anchors the penalty to recorded actions. In continuous action spaces, the required action integrals or sums are approximated using samples. The regularizer therefore targets values that could otherwise become attractive merely through extrapolation.
Under specified assumptions, CQL analysis establishes lower-bound properties for expected policy values. Such results do not mean that every neural estimate is a calibrated lower confidence bound at every state–action pair. Practical implementations introduce approximation error, finite-sample effects, and optimization choices. Excessive conservatism can also suppress genuinely valuable actions, especially when successful behavior is rare. The penalty strength must balance protection against unsupported optimism with the ability to distinguish good decisions from poor ones.
Implicit Q-Learning (IQL) approaches the problem by avoiding explicit maximization over out-of-dataset actions during value learning. It fits a state-value function using asymmetric squared error, or expectile regression, on dataset action values. With expectile parameter , positive residuals receive greater weight, so the fitted state value emphasizes the upper part of the observed action-value distribution. Its action-value function then uses a target of the form , rather than a target formed by maximizing over arbitrary candidate actions.
IQL extracts a policy through advantage-weighted behavior cloning. Thus, its critic and actor updates remain tied to actions represented in the dataset, while still favoring better-than-average actions. The expectile is not a quantile: it is defined by asymmetric squared deviations rather than asymmetric absolute deviations. This distinction matters because the resulting value depends on both the probability and the magnitude of residuals, not simply on a percentile rank.
Other methods use ensembles, uncertainty estimates, or robust optimization to obtain conservative targets. Disagreement among value functions can indicate weak support, but it is not automatically a complete uncertainty measure. Networks trained with similar architectures and data may share the same error. More generally, pessimistic offline learning depends on identifying where evidence is insufficient; simply subtracting an arbitrary penalty from all values does not provide a meaningful guarantee.
Learned dynamics and model-based planning
Model-based offline reinforcement learning fits an approximate environment and uses it for planning or additional training. A learned transition model , often accompanied by a reward model, allows the learner to generate synthetic trajectories without interacting with the real system. This can make recorded transitions more reusable and support explicit comparisons between candidate action sequences. Models may operate directly on physical variables or through a learned latent space when observations are images or other high-dimensional signals.
The main danger is model exploitation. A policy optimized inside an imperfect model can discover action sequences that produce high simulated rewards because the model is wrong. Even a model with low average prediction error on held-out logged transitions may behave poorly in the regions reached by the optimized policy. Optimization changes the distribution on which accuracy matters. This problem is especially serious when predictions are rolled forward repeatedly, allowing small local errors to alter later states and compound over time.
MOPO, short for Model-based Offline Policy Optimization, addresses this issue by penalizing synthetic rewards using a model-uncertainty estimate. Its modified reward has the schematic form , where measures uncertainty or predictive disagreement. The algorithm trains a policy using generated experience while discouraging trajectories through uncertain regions. Short model rollouts, initialized from states in the real dataset, can reduce the distance over which errors accumulate, although they do not eliminate model bias.
MOReL, or Model-Based Offline Reinforcement Learning, uses a related pessimistic construction. It identifies state–action regions considered insufficiently known and treats entering them as leading to an undesirable absorbing state in the learned decision process. This turns uncertainty into an explicit planning cost. The distinction between known and unknown regions is itself estimated, so theoretical conclusions depend on whether the uncertainty mechanism correctly captures model error and whether the dataset supports the relevant comparisons.
A learned world model can also exploit structural information that model-free value learning does not directly encode. Physical constraints, object relationships, or known equations may reduce the space of plausible dynamics. Such information can sometimes support generalization beyond exact recorded state–action pairs. However, claims of reliable extrapolation require the structural assumptions to be appropriate. A flexible predictive model trained only to reproduce observations does not automatically identify the effects of unseen interventions.
Model-based evaluation therefore requires more than reporting one-step prediction accuracy. Relevant tests include multi-step consistency, reward accuracy, calibration across operating regimes, and the performance of policies optimized against the model. Even these tests remain limited when all available validation data come from the same narrow behavior distribution. The essential question is not whether the model can reconstruct familiar trajectories, but whether its errors remain controlled under the decisions it encourages.
Sequence modeling and return-conditioned policies
Sequence-modeling approaches represent decision-making as prediction over trajectories rather than exclusively through Bellman recursion. A trajectory contains temporally ordered states, actions, and rewards; an autoregressive model can learn statistical relationships among these elements. This connects offline reinforcement learning with representation learning and architectures originally developed for natural language processing. The connection concerns the modeling machinery rather than an equivalence between language prediction and optimal control.
Decision Transformer, introduced by Lili Chen and colleagues in 2021, trains a causal Transformer on sequences containing returns-to-go, states, and actions. For a finite trajectory, a return-to-go token typically represents
The model predicts an action conditioned on the recent trajectory and a desired return. During deployment, the return-conditioning variable is updated as rewards are received. Training can use ordinary supervised prediction losses, without fitting a Bellman equation or estimating an action-value function.
Return conditioning changes the imitation objective. Instead of reproducing the unconditional mixture of all behaviors, the model can associate better outcomes with particular action patterns. On datasets containing trajectories of different quality, this can select behavior resembling higher-return examples. Nevertheless, conditioning on a large numerical target does not ensure that the target is achievable. If high returns are absent from the data, or reflect favorable randomness rather than controllable action choices, the model may have no basis for producing the requested outcome.
Trajectory Transformer, introduced by Michael Janner and colleagues in 2021, models trajectory components autoregressively and uses search over predicted sequences for planning. In its original formulation, continuous quantities are discretized into tokens. This differs from directly predicting actions from a desired return: the learned trajectory distribution can support evaluation of alternative continuations. Its planning quality still depends on the accuracy of the generated transitions and rewards, and search can exploit prediction errors just as optimization can exploit errors in conventional dynamics models.
Diffusion models provide another way to represent complex action or trajectory distributions. Diffusion-based planners generate candidate trajectories and can condition generation on goals, rewards, or other constraints. Their expressive distributions can represent multiple distinct successful behaviors more naturally than a single deterministic action predictor. However, diffusion-based policy learning is not necessarily offline reinforcement learning: a model trained only to imitate demonstrations may be better described as imitation learning unless reward-based improvement or planning is involved.
Sequence methods make history dependence and large-scale pretraining relatively natural, but long context does not solve every decision-making problem. Long-horizon credit assignment, hidden confounding, and missing coverage remain. Moreover, classic value-based methods can sometimes combine locally useful transitions into policies that were never recorded as complete trajectories. The degree to which sequence models achieve comparable recombination depends on their representation, training objective, conditioning scheme, and planning procedure rather than on model size alone.
Offline evaluation, model selection, and benchmarks
Off-policy evaluation estimates a target policy’s return using data collected by other policies. It is indispensable when direct deployment is costly or unavailable. Learning and evaluation are distinct tasks: a policy can be easy to represent but difficult to evaluate from existing data, and an algorithm’s predicted training value need not be an independent estimate of its real performance. Reliable policy comparison requires attention to both statistical uncertainty and the assumptions linking the logs to the target policy.
Importance sampling reweights recorded trajectories according to how likely their actions are under the target and behavior policies. A trajectory weight can take the form
Under appropriate support and logging assumptions, weighted returns can estimate target-policy performance. In long horizons, however, products of probability ratios can have enormous variance. Estimated behavior probabilities introduce further error, and deterministic or highly concentrated target policies make overlap especially difficult in continuous action spaces. No reweighting scheme can recover outcomes for actions that have zero probability in the relevant logged data.
Model-based evaluation predicts outcomes through an estimated environment, while fitted Q evaluation repeatedly estimates the value of a fixed target policy from transitions. Doubly robust estimators combine a value model with importance-weighted corrections, seeking favorable bias or variance properties under stated assumptions. Their names should not be interpreted as guarantees against arbitrary misspecification. Dependence between samples, hidden confounding, inadequate overlap, and poorly estimated nuisance models can undermine the conditions required by the analysis.
Policy selection adds another layer of difficulty. If many algorithms, random seeds, and hyperparameter settings are compared using the same noisy evaluator, the selected winner may exploit evaluation error. Splitting data by complete trajectories, collection periods, or source systems can be more appropriate than randomly splitting individual transitions. Data leakage occurs when related trajectory fragments or future information cross the intended evaluation boundary. A held-out transition prediction loss, meanwhile, measures predictive fit rather than necessarily measuring policy quality.
D4RL made several dataset characteristics explicit, including random behavior, partially trained policies, replay buffers, expert demonstrations, and mixtures of quality levels. Its maze and navigation tasks helped expose the difficulty of connecting trajectory fragments and handling sparse rewards. Some D4RL tasks use normalized scores of the form
The reference policies define a scale, not a universal probability of success; scores can fall below zero or exceed 100.
A crucial distinction separates offline training from offline-only evaluation. Many benchmark studies train on a fixed dataset but evaluate resulting policies by executing them in a simulator. That is valid for measuring algorithm performance, yet it does not solve the deployment problem of choosing a policy when no such testing environment exists. Repeated simulator-based tuning can also make reported results more favorable than a genuinely interaction-free development process would permit.
Applications, deployment boundaries, and unresolved problems
Robotics is a prominent application because physical interaction consumes time and can damage equipment. Offline datasets can combine teleoperation, scripted controllers, and earlier robot policies. Reward-based learning may identify useful actions in demonstrations that vary in quality or combine skills across trajectories. Important practical complications include camera changes, object variation, imperfect synchronization, and differences between the robot that collected the data and the robot on which the learned policy will run. Strong results on one manipulation setup do not establish robustness to these changes.
In industrial systems, historical controller logs can support research on energy use, process efficiency, and maintenance-related decisions. Such records often provide extensive coverage of normal operation but little information about unusual or unsafe conditions. Existing controllers may also have enforced constraints that are not explicitly represented in the learning objective. Replacing their behavior with a reward-maximizing policy can therefore remove protections that the dataset does not identify as separate rules. Offline evidence alone does not establish compliance with physical operating limits.
Recommender systems provide large quantities of logged interactions and naturally involve delayed effects, such as whether a recommendation changes subsequent engagement. However, displayed items are selected by earlier systems, so unshown alternatives have no directly observed user response. Position effects, changing user populations, and evolving item catalogs further complicate interpretation. Whether a contextual-bandit model or a sequential reinforcement-learning model is appropriate depends on the actual decision process and the temporal effects being studied.
Clinical records have motivated offline reinforcement-learning research because they contain sequences of observations, interventions, and outcomes. These are observational data rather than randomized experiments. Treatment choices can reflect clinician knowledge not captured in structured variables, and recorded interventions may be proxies for a changing underlying condition. Missing measurements, irregular sampling, and differences between institutions complicate the state representation. Retrospective estimates of a policy’s value do not by themselves establish treatment efficacy or clinical safety.
Offline-to-online learning provides an intermediate setting. A policy is initialized from static data and subsequently improved through new interaction. This can reduce the amount of online experience required, but it introduces a transition problem: a conservative offline policy may explore too little, while immediately removing conservatism may expose unreliable value estimates. The resulting system is not fully offline during its later phase, and its gains must be assessed together with the cost and risk of the additional interactions.
Open research problems include reliable uncertainty estimation with neural networks, policy selection without environment access, long-horizon learning under weak coverage, and generalization across tasks and embodiments. Richer observations and pretrained representations can improve the information extracted from a dataset, but they cannot reveal arbitrary missing consequences without additional assumptions. Likewise, optimizing an accurately estimated reward can still produce reward hacking when that reward omits important aspects of the intended task. Offline reinforcement learning must therefore distinguish three separate questions: what behavior the data support, what outcomes the algorithm can estimate, and what objective the deployed system is actually optimizing.