{
  "id": "policy-gradient-method",
  "title": "Policy-Gradient Method",
  "lang": "en",
  "source_lang": "en",
  "summary": "A family of reinforcement-learning methods that optimize parameterized policies by estimating gradients of expected return.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Policy-gradient methods",
    "policy gradient",
    "policy-gradient learning"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/policy-gradient-method/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/policy-gradient-method/",
      "json": "https://aiwiki.page/zh/policy-gradient-method.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A policy-gradient method is an approach to reinforcement learning that improves an agent’s behavior by estimating how changes in its policy parameters affect expected cumulative reward. Rather than first learning a value for every possible action and then selecting the action with the highest estimated value, it directly adjusts a parameterized policy: a rule specifying which actions to take in different situations. The policy may be a small statistical model or a large neural network, and its outputs may describe discrete choices, continuous controls, or distributions over sequences.\n\nPolicy-gradient methods are important because they accommodate stochastic behavior, high-dimensional observations, and continuous action spaces within a common optimization framework. They underpin many actor–critic algorithms, robot-control systems, game-playing agents, and methods for adapting language models using reward signals. Their central mathematical result expresses a gradient of long-term performance using quantities that can be estimated from interaction, without requiring derivatives of the environment’s dynamics.\n\nThe family includes Monte Carlo estimators such as REINFORCE, natural policy gradients, trust-region methods, proximal policy optimization, and deterministic or entropy-regularized actor–critic algorithms. These approaches differ chiefly in how they estimate action quality, control sampling variance, reuse previously collected experience, and prevent destabilizing changes to behavior."
    },
    {
      "heading": "Policies, trajectories, and optimization objectives",
      "text": "The standard mathematical setting is a Markov decision process, consisting of states \\(s\\), actions \\(a\\), transition probabilities \\(P(s' \\mid s,a)\\), rewards, and an initial-state distribution. A policy \\(\\pi_\\theta(a\\mid s)\\), parameterized by a vector \\(\\theta\\), gives the conditional probability or probability density of taking action \\(a\\) in state \\(s\\). In partially observed environments, its input may instead be an observation history or an internal memory state. Learning changes \\(\\theta\\), thereby changing both immediate action choices and the future states the agent encounters.\n\nA trajectory is a sequence such as\n\n\\[\\tau=(s_0,a_0,r_0,s_1,a_1,r_1,\\ldots).\\]\n\nFor a finite horizon \\(T\\), a common objective is\n\n\\[J(\\theta)= \\mathbb{E}_{\\tau\\sim\\pi_\\theta} \\left[\\sum_{t=0}^{T-1}\\gamma^t r_t\\right],\\]\n\nwhere \\(\\gamma\\in[0,1]\\) is a discount factor. The expectation averages over initial states, policy randomness, and environmental randomness. For continuing tasks, an infinite-horizon discounted objective with \\(\\gamma<1\\), or a long-run average-reward objective, is often used. These choices are mathematically distinct: discounting changes the relative importance of events at different times and is not merely a numerical convenience.\n\nThe basic update is gradient ascent,\n\n\\[\\theta_{k+1}=\\theta_k+\\alpha_k\\widehat{\\nabla_\\theta J(\\theta_k)},\\]\n\nwith learning rate \\(\\alpha_k\\) and an estimated gradient. Implementations often minimize the negative of a surrogate objective using software designed for loss minimization. The policy gradient is therefore an instance of mathematical optimization, but its data distribution changes as the policy changes. Actions influence what experience is collected, making the problem different from ordinary supervised learning on a fixed dataset.\n\nPolicy parameterization determines which behaviors can be represented. For discrete actions, a neural network often produces logits followed by a softmax distribution. For continuous actions, a common choice is a Gaussian distribution whose mean, and sometimes covariance, depend on the state. A deterministic policy instead produces a single action \\(\\mu_\\theta(s)\\). A stochastic policy can retain randomness when uncertainty or strategic interaction makes randomization useful, rather than using randomness only during initial exploration.\n\nDirect policy optimization is not identical to every form of policy search. Searching over policy parameters using evolutionary methods or finite-difference perturbations need not use a policy-gradient estimator. Conversely, an actor–critic algorithm remains a policy-gradient method even though it learns value functions internally. The defining feature is the use of a gradient of a policy-performance objective, or a carefully constructed approximation to such a gradient, to update the policy."
    },
    {
      "heading": "The policy-gradient theorem and its derivation",
      "text": "The key difficulty is that changing a policy alters the entire distribution of future trajectories. Nevertheless, under suitable differentiability and integrability conditions, the gradient can be estimated without differentiating the transition probabilities. For an environment whose dynamics and rewards do not explicitly depend on \\(\\theta\\), a finite trajectory has probability\n\n\\[p_\\theta(\\tau)= \\rho_0(s_0)\\prod_{t=0}^{T-1} \\pi_\\theta(a_t\\mid s_t)P(s_{t+1}\\mid s_t,a_t).\\]\n\nConsequently,\n\n\\[\\nabla_\\theta\\log p_\\theta(\\tau) = \\sum_{t=0}^{T-1}\\nabla_\\theta\\log\\pi_\\theta(a_t\\mid s_t).\\]\n\nOnly the policy factors contribute to this derivative.\n\nWriting the discounted trajectory return as \\(R(\\tau)\\), the likelihood-ratio identity gives\n\n\\[\\nabla_\\theta J(\\theta) = \\mathbb{E}\\left[ R(\\tau) \\sum_{t=0}^{T-1} \\nabla_\\theta\\log\\pi_\\theta(a_t\\mid s_t) \\right].\\]\n\nThis is a score-function estimator: it differentiates the log probability of sampled events rather than differentiating the events themselves. It can therefore handle discrete actions and nondifferentiable environments. For example, a simulator may contain collisions, discrete branching, or a proprietary transition mechanism; a policy gradient can still be estimated if actions can be sampled and rewards observed.\n\nCausality removes terms involving rewards obtained before the action being differentiated. Define the reward-to-go from time \\(t\\) as\n\n\\[G_t=\\sum_{k=t}^{T-1}\\gamma^{k-t}r_k.\\]\n\nThe exact gradient for the stated finite-horizon discounted objective can then be written\n\n\\[\\nabla_\\theta J(\\theta) = \\mathbb{E}\\left[ \\sum_{t=0}^{T-1} \\gamma^t \\nabla_\\theta\\log\\pi_\\theta(a_t\\mid s_t)G_t \\right].\\]\n\nThe outer factor \\(\\gamma^t\\) matters for this particular objective and sampling convention. Expressions without it may correspond to a different state weighting, an undiscounted episodic objective, or an implementation approximation.\n\nThe policy-gradient theorem expresses the same principle through value functions. Let\n\n\\[Q^\\pi(s,a)= \\mathbb{E}_\\pi\\left[ \\sum_{k=0}^{\\infty}\\gamma^k r_{t+k} \\mid s_t=s,a_t=a \\right].\\]\n\nFor the normalized discounted state-occupancy distribution\n\n\\[d_\\gamma^\\pi(s)= (1-\\gamma)\\sum_{t=0}^{\\infty} \\gamma^t\\Pr_\\pi(s_t=s),\\]\n\nthe infinite-horizon discounted gradient is\n\n\\[\\nabla_\\theta J(\\theta)= \\frac{1}{1-\\gamma} \\mathbb{E}_{s\\sim d_\\gamma^\\pi,\\;a\\sim\\pi_\\theta} \\left[ \\nabla_\\theta\\log\\pi_\\theta(a\\mid s) Q^\\pi(s,a) \\right].\\]\n\nThe theorem does not claim that the state distribution is independent of the policy. Rather, it accounts for that dependence without requiring an explicit derivative of the occupancy distribution. This is what makes it useful: samples of states and actions, together with estimates of action value, suffice to construct an update. When rewards, initial conditions, or environment parameters themselves depend explicitly on the policy parameters, additional derivative terms may be required."
    },
    {
      "heading": "REINFORCE, baselines, and sampling variance",
      "text": "REINFORCE, introduced by Ronald J. Williams in 1992, is a foundational family of likelihood-ratio learning rules. In its familiar episodic form, the agent samples complete trajectories, computes their returns, and increases the log probability of actions in proportion to the return associated with them. An action that appears in a high-return trajectory receives a positive update, while an action associated with sufficiently poor relative performance can receive a negative one. The rule does not require an explicit model of environmental dynamics or a differentiable reward function.\n\nFor \\(N\\) sampled episodes, a reward-to-go estimator is\n\n\\[\\widehat g= \\frac{1}{N}\\sum_{i=1}^{N} \\sum_{t=0}^{T_i-1} \\gamma^t \\nabla_\\theta\\log\\pi_\\theta(a_t^{(i)}\\mid s_t^{(i)}) G_t^{(i)}.\\]\n\nFor on-policy samples and the matching objective, this is an unbiased gradient estimator under the usual regularity conditions. Unbiasedness does not imply that a small batch gives an accurate direction. Returns can vary greatly because of random transitions, exploratory actions, different initial conditions, and rare events occurring many steps after the action being updated.\n\nA central variance-reduction device is a baseline. A state-dependent quantity \\(b(s)\\) may be subtracted from the return without changing the expected gradient, because\n\n\\[\\mathbb{E}_{a\\sim\\pi_\\theta(\\cdot\\mid s)} \\left[\\nabla_\\theta\\log\\pi_\\theta(a\\mid s)b(s)\\right]=0.\\]\n\nThis follows from the fact that the action probabilities sum or integrate to one. The baseline must not depend on the sampled action in an unrestricted way; action-dependent control variates require additional correction. A learned baseline is normally treated as fixed during the actor-gradient calculation, even if it depends on parameters that are updated elsewhere.\n\nA common baseline is the state value\n\n\\[V^\\pi(s)=\\mathbb{E}_{a\\sim\\pi}[Q^\\pi(s,a)].\\]\n\nSubtracting it gives the advantage function,\n\n\\[A^\\pi(s,a)=Q^\\pi(s,a)-V^\\pi(s),\\]\n\nwhich measures whether an action is better or worse than the policy’s usual behavior in that state. Thus an action can be reinforced even when its absolute return is negative, provided it is better than the relevant baseline. Although \\(V^\\pi\\) is a natural and useful baseline, the variance-minimizing baseline generally also depends on the magnitude of the score vector, so the value function is not universally optimal for variance reduction.\n\nOther techniques include using reward-to-go rather than full-episode return, increasing batch size, and exploiting problem structure to construct control variates. Batchwise advantage normalization is common in implementations, but it is an additional data-dependent transformation rather than an exact consequence of the baseline identity. Sparse rewards remain especially difficult: if all sampled episodes have zero reward, an ordinary return-weighted estimator may provide no useful direction toward a successful behavior. Larger networks or faster gradient computation alone do not resolve this exploration problem."
    },
    {
      "heading": "Actor–critic learning and temporal credit assignment",
      "text": "An actor–critic method combines a policy, called the actor, with a learned evaluator, called the critic. The critic estimates a value function or an action-value function; the actor uses that estimate to improve its action probabilities. This separates two related problems: predicting future outcomes and deciding how behavior should change. The actor may have parameters \\(\\theta\\) and the critic parameters \\(\\phi\\), although neural implementations sometimes share an observation encoder.\n\nA state-value critic can be trained using temporal-difference learning. Its one-step error is\n\n\\[\\delta_t=r_t+\\gamma V_\\phi(s_{t+1})-V_\\phi(s_t),\\]\n\nwith the bootstrap term removed at a genuine terminal state. If \\(V_\\phi=V^\\pi\\), the conditional expectation of \\(\\delta_t\\), given \\(s_t,a_t\\), equals \\(A^\\pi(s_t,a_t)\\). This makes the temporal-difference error a practical learning signal for the actor. Unlike a complete Monte Carlo return, it can be computed after a single transition, allowing updates before an episode finishes.\n\nBootstrapping creates a bias–variance trade-off. A Monte Carlo return avoids dependence on an estimated value at the end of the sampled segment but may be very noisy. A short bootstrapped return has less exposure to subsequent randomness but inherits errors from the critic. Longer \\(n\\)-step returns interpolate between these extremes. A critic can therefore accelerate learning while also introducing systematic errors into the actor’s update, particularly when function approximation is inaccurate in newly visited states.\n\nGeneralized advantage estimation, developed by John Schulman and collaborators and circulated in 2015, combines temporal-difference residuals:\n\n\\[\\widehat A_t^{\\mathrm{GAE}(\\gamma,\\lambda)} = \\sum_{\\ell=0}^{L-1} (\\gamma\\lambda)^\\ell\\delta_{t+\\ell}.\\]\n\nHere \\(\\lambda\\in[0,1]\\) controls the effective length of the estimator, while \\(L\\) depends on available rollout data. At \\(\\lambda=0\\), it uses the one-step residual. Toward \\(\\lambda=1\\), it approaches a Monte Carlo-style return minus a baseline, subject to the treatment of segment endpoints. Its practical value is the ability to tune temporal credit assignment without explicitly selecting just one return length.\n\nTerminal states and data-collection cutoffs must be distinguished. If an episode truly ends, no future task reward remains. If a rollout stops only because a fixed sampling window has elapsed, the underlying task may continue, and bootstrapping can still be appropriate. Confusing these cases systematically distorts value targets. Similar care is needed when environments reset automatically, because the observation after reset must not be mistaken for the successor state of a terminated trajectory.\n\nClassical convergence analyses often place actor and critic on different time scales, allowing the critic to track the current policy while the actor changes more slowly. Deep actor–critic implementations do not necessarily satisfy those assumptions. When actor and critic share parameters, fitting the critic can also change the actor’s representation, so their interaction is more complex than two completely separate optimization problems."
    },
    {
      "heading": "Natural gradients, trust regions, and proximal updates",
      "text": "An ordinary parameter gradient depends on the coordinates used to represent a policy. Two parameter changes of equal Euclidean length can produce very different changes in action probabilities, and reparameterizing the same policy family can change the apparent steepest direction. The natural gradient addresses this by measuring local change in the space of probability distributions. For a policy, the relevant Fisher information matrix has the form\n\n\\[F(\\theta)= \\mathbb{E}\\left[ \\nabla_\\theta\\log\\pi_\\theta(a\\mid s) \\nabla_\\theta\\log\\pi_\\theta(a\\mid s)^\\top \\right],\\]\n\nunder a specified state–action distribution.\n\nThe natural-gradient direction is \\(F(\\theta)^{-1}\\nabla_\\theta J\\), interpreted with suitable handling of singularity or damping. Locally, the Fisher matrix is the curvature of the Kullback–Leibler divergence between nearby policies. A natural-gradient step can therefore be derived by maximizing a linear approximation to performance while constraining a quadratic approximation to policy divergence. For large neural networks, forming and inverting the full matrix is generally impractical; algorithms instead use matrix–vector products, iterative solvers, or structured approximations.\n\nTrust region policy optimization (TRPO), introduced in 2015, uses a surrogate objective evaluated with samples from an old policy and constrains an average KL divergence. A typical surrogate is\n\n\\[L(\\theta)= \\mathbb{E}_{s,a\\sim\\pi_{\\mathrm{old}}} \\left[ \\frac{\\pi_\\theta(a\\mid s)} {\\pi_{\\mathrm{old}}(a\\mid s)} \\widehat A_{\\mathrm{old}}(s,a) \\right].\\]\n\nThe probability ratio accounts for changing action probabilities at sampled states. Restricting the size of the update is intended to keep this local approximation useful. Practical TRPO combines an approximate constrained step with a line search; its theoretical motivation does not imply an unconditional improvement guarantee for every sampled neural-network implementation.\n\nProximal policy optimization (PPO), introduced in 2017, replaces the explicit constrained solve with objectives that are easier to optimize using minibatch gradient methods. Its widely used clipped objective is\n\n\\[L^{\\mathrm{clip}}(\\theta)= \\mathbb{E}\\left[ \\min\\left( r_t(\\theta)\\widehat A_t,\\, \\operatorname{clip}(r_t(\\theta),1-\\epsilon,1+\\epsilon) \\widehat A_t \\right) \\right],\\]\n\nwhere\n\n\\[r_t(\\theta)= \\frac{\\pi_\\theta(a_t\\mid s_t)} {\\pi_{\\mathrm{old}}(a_t\\mid s_t)}.\\]\n\nFor a positive advantage, excessive increases in the sampled action’s probability cease to improve the clipped objective. For a negative advantage, the analogous protection concerns excessive probability decreases.\n\nClipping does not impose a hard bound on every action probability or on the overall policy divergence. Parameters are shared across samples, and multiple optimization passes can still move the policy substantially. Implementations may therefore monitor KL divergence or stop updates early. Natural-gradient, TRPO, and PPO methods address the same broad problem—preventing a noisy local performance estimate from supporting an excessively large behavioral change—but achieve it through different mathematical and computational mechanisms."
    },
    {
      "heading": "Off-policy, deterministic, and entropy-regularized variants",
      "text": "An on-policy estimator uses experience generated by the policy being differentiated. An off-policy method learns from a behavior policy \\(\\beta\\), potentially including older policies stored in a replay buffer. Reusing experience can greatly improve interaction efficiency, especially when collecting data is expensive. However, the action distribution and usually the state distribution differ from those of the target policy, so simply applying an on-policy formula to replayed data is generally incorrect.\n\nImportance sampling corrects changes of distribution using likelihood ratios. For an entire trajectory, a ratio can contain the product\n\n\\[\\prod_t\\frac{\\pi_\\theta(a_t\\mid s_t)}{\\beta(a_t\\mid s_t)}.\\]\n\nSuch products may have very high variance over long horizons. Per-decision corrections, truncated ratios, and specialized actor–critic estimators trade exactness against stability. Adequate support is essential: if the behavior policy never takes an action that the target policy may choose, the missing outcomes cannot be recovered by weighting the observed samples.\n\nThe deterministic policy-gradient theorem, presented by David Silver and collaborators in 2014, gives a different route for continuous actions. For a deterministic actor \\(a=\\mu_\\theta(s)\\), an on-policy form is\n\n\\[\\nabla_\\theta J(\\theta)= \\frac{1}{1-\\gamma} \\mathbb{E}_{s\\sim d_\\gamma^\\mu} \\left[ \\nabla_\\theta\\mu_\\theta(s) \\nabla_a Q^\\mu(s,a)\\big|_{a=\\mu_\\theta(s)} \\right].\\]\n\nThe actor is differentiated through the critic’s action input, not through the environment. This requires an action-value function differentiable with respect to continuous actions. Deep deterministic policy gradient, described in 2015, combines this idea with neural function approximation, replay memory, and slowly updated target networks.\n\nPractical replay-based deterministic actor updates commonly average over states in the replay buffer rather than the exact current-policy occupancy distribution. Such updates should not automatically be identified with an unbiased gradient of the original start-state objective. Another difficulty is critic exploitation: an actor may choose actions that look favorable only because the critic overestimates them. Twin delayed deep deterministic policy gradient, introduced in 2018, counteracts this with two critics, a minimum-based target, delayed actor updates, and target-policy smoothing.\n\nMaximum-entropy methods optimize reward together with policy entropy. A representative objective is\n\n\\[J_{\\mathrm{ent}}(\\pi)= \\mathbb{E}_\\pi\\left[ \\sum_{t=0}^{\\infty}\\gamma^t \\left(r_t+\\alpha\\mathcal H(\\pi(\\cdot\\mid s_t))\\right) \\right],\\]\n\nwhere \\(\\mathcal H\\) is entropy and \\(\\alpha\\) sets its relative weight. Soft actor–critic, introduced in 2018, combines an off-policy critic with a stochastic actor and entropy regularization. Continuous actions are often generated through a differentiable transformation of parameter-free noise, permitting a reparameterized actor gradient. Here entropy is part of the optimization criterion, rather than merely an informal exploration aid; changing its weight changes the behavior being optimized."
    },
    {
      "heading": "Neural implementation and experimental practice",
      "text": "With deep learning, the policy gradient is usually computed by differentiating a scalar surrogate through a neural network. A batch contains observations, sampled actions, old log probabilities, rewards, termination information, and quantities needed for value or advantage estimates. Backpropagation computes derivatives of log probabilities or critic outputs with respect to network parameters. The environment interaction itself ordinarily lies outside this differentiable computation.\n\nA crucial implementation distinction is between differentiating an estimator and differentiating every quantity appearing numerically inside it. In a standard likelihood-ratio actor update, the sampled advantage is treated as a fixed weight. If automatic differentiation also follows the advantage through the critic, it adds terms not present in the intended estimator. Similarly, old-policy log probabilities in a probability ratio must remain fixed during the policy update. Frameworks such as PyTorch and TensorFlow provide mechanisms for detaching tensors from the computational graph.\n\nThe action distribution requires careful handling. For continuous bounded actions, a common construction samples a Gaussian variable \\(u\\) and applies \\(a=\\tanh u\\), followed by any required rescaling. The action log density must include the transformation’s Jacobian:\n\n\\[\\log\\pi(a\\mid s) = \\log p(u\\mid s) -\\sum_i\\log\\left(1-\\tanh^2u_i\\right),\\]\n\nbefore accounting for additional scaling. Simply clipping a Gaussian sample at the action limits produces a different distribution, including probability mass at the boundaries; evaluating it as an unchanged Gaussian gives an inconsistent log probability.\n\nData collection and optimization also operate at different scales. Parallel environments can supply large rollout batches, while accelerators process minibatches efficiently. Asynchronous workers may act using stale policy parameters, introducing an off-policy component even when the intended algorithm is largely on-policy. For autoregressive policies, the likelihood of a sampled sequence is the product of conditional token probabilities, usually evaluated as a sum of log probabilities. Long sequences therefore create many policy decisions from a single generated output.\n\nReward scaling, observation normalization, gradient clipping, optimizer settings, and rollout length can substantially affect results. An adaptive optimizer such as Adam changes how the estimated gradient is converted into a parameter update, but does not eliminate errors in the estimator. Reward transformations also require care: multiplying all rewards by a positive constant preserves the unregularized ordering of policies, whereas adding a constant per step can change preferences when episode lengths differ. Entropy and other penalty terms must be considered separately when rewards are rescaled.\n\nEvaluation normally separates training-time exploration from the policy behavior being measured and reports variability across random seeds. Environment interactions, gradient updates, and elapsed computation measure different aspects of cost. High return on training environments is not itself evidence of generalization to different dynamics, observations, or task distributions, and sensitivity to hyperparameters can make comparisons unreliable when tuning budgets differ substantially."
    },
    {
      "heading": "Applications in control, games, and language generation",
      "text": "Continuous control is a major application because direct action optimization can avoid enumerating a large or infinite action set. A robot policy may output joint torques, target positions, or parameters of a lower-level controller. Neural policies can consume state estimates or visual observations and learn coordinated behavior over many actuators. Simulation permits large volumes of interaction, but transferring a policy to physical hardware introduces discrepancies in contact dynamics, sensing, latency, and actuator behavior. Methods such as dynamics randomization aim to reduce sensitivity to these discrepancies, without guaranteeing transfer.\n\nGames provide repeatable environments and clear performance signals. In the 2016 AlphaGo system, a policy network initially trained from human moves was subsequently improved through reinforcement learning using policy-gradient updates against earlier versions of itself. This policy-learning component was combined with a value network and Monte Carlo tree search. AlphaGo illustrates that a policy-gradient method may be one component of a larger decision system rather than the entire action-selection procedure.\n\nOther game-playing systems use actor–critic methods to learn from large-scale parallel experience or self-play. The policy may select among discrete commands, choose spatial locations, or coordinate multiple action components. In multiplayer settings, the learning environment changes as other agents change their policies. Consequently, a stationary single-agent Markov decision process may be an imperfect description, and success against training opponents need not imply robustness against unseen strategies.\n\nFor a large language model, token generation can be viewed as sequential decision-making. The state is the prompt and generated prefix, the action is the next token, and the policy is the model’s conditional token distribution. A reward may arrive only after the full response, making the connection to episodic policy gradients direct. The same machinery can optimize sequence-level criteria that are not differentiable with respect to individual token choices, provided sampled outputs can be scored.\n\nIn reinforcement learning from human feedback, a learned reward model may supply scores, while a penalty for divergence from a reference model discourages excessive drift. PPO became widely used for this stage of language-model training. The critic can predict expected future reward from a partial response, and token-level log-probability ratios support policy updates. Sampling, reward-model evaluation, value estimation, and reference-policy evaluation may all contribute substantially to computational cost.\n\nA higher estimated reward is not automatically a more desirable output. Optimization can exploit weaknesses in a learned reward model, an instance of reward hacking, and behavior can change under distribution shift. Not all preference-based adaptation uses policy gradients: direct preference optimization, for example, trains from preference pairs using a different objective without a conventional rollout-based actor–critic loop. The distinction concerns the training mechanism, even when methods address similar application goals."
    },
    {
      "heading": "Historical development and theoretical limitations",
      "text": "Policy-gradient methods emerged from work connecting stochastic learning systems, adaptive control, and optimization. Williams’s 1992 REINFORCE paper supplied a general and influential family of gradient estimators for stochastic units. Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour presented policy-gradient results with function approximation in 1999. Their work also clarified the role of compatible function approximation, under which a suitably parameterized approximate critic can preserve the correct actor-gradient direction when the necessary fitting conditions hold.\n\nSham Kakade introduced a natural policy-gradient formulation in work presented in 2001. Subsequent research developed natural actor–critic algorithms, more effective advantage estimators, and practical approximations to second-order updates. The deterministic policy-gradient theorem appeared in 2014; TRPO and deep deterministic policy gradient followed in 2015. Asynchronous advantage actor–critic in 2016 demonstrated effective parallel training with deep policies, while PPO in 2017 and soft actor–critic in 2018 became widely used algorithmic families. These developments addressed different bottlenecks: gradient noise, update geometry, continuous actions, parallel data collection, and reuse of experience.\n\nConvergence claims depend strongly on assumptions. Classical stochastic-approximation results may require boundedness, suitable step-size schedules, sufficient exploration, and accurate or specially structured critics. In general neural-network parameterizations, the objective is nonconvex, and an exact gradient does not ensure convergence to a globally optimal policy. Stronger guarantees are available for particular tabular policy parameterizations and carefully specified update rules, but they do not transfer automatically to arbitrary deep networks or finite-sample implementations.\n\nLong-horizon credit assignment and exploration remain separate sources of difficulty. Even a low-variance estimator cannot learn from successful outcomes that are never observed. Conversely, observing a rare success may still provide a very noisy signal about which earlier actions caused it. Sparse rewards, nearly deterministic policies, poorly scaled probability distributions, and inaccurate critics can all reduce effective learning. A policy can also improve its measured average return while performing poorly on rare but important events.\n\nPolicy parameterization creates additional constraints. A unimodal Gaussian actor cannot directly represent every multimodal action distribution, while an autoregressive policy can represent complex structured actions at the cost of additional sampling and likelihood computation. Partial observability may require memory, and learning a recurrent policy introduces long-range differentiation and state-management issues. Restricted behavior classes can make an apparently unsuccessful optimizer indistinguishable from an insufficiently expressive policy unless those sources of error are examined separately.\n\nResearch consequently studies finite-sample guarantees, off-policy corrections, structured exploration, constrained objectives, and the interaction between representation learning and policy improvement. In safety-constrained formulations, expected reward may be maximized subject to bounds on expected costs, introducing additional value functions and optimization variables. Such constraints do not by themselves bound the probability of every catastrophic event: average-cost limits, chance constraints, and worst-case guarantees are different mathematical requirements, each demanding its own assumptions and estimation procedures."
    }
  ],
  "links": [
    "reinforcement-learning",
    "policy-reinforcement-learning",
    "artificial-neural-network",
    "proximal-policy-optimization",
    "markov-decision-process",
    "expected-value",
    "learning-rate",
    "mathematical-optimization",
    "supervised-learning",
    "reinforce-algorithm",
    "temporal-difference-learning",
    "natural-gradient",
    "fisher-information",
    "kullback-leibler-divergence",
    "importance-sampling",
    "deep-learning",
    "backpropagation",
    "pytorch",
    "tensorflow",
    "computational-graph",
    "adam-optimizer",
    "generalization",
    "hyperparameter",
    "alphago",
    "monte-carlo-tree-search",
    "large-language-model",
    "reinforcement-learning-from-human-feedback",
    "reward-hacking",
    "distribution-shift",
    "direct-preference-optimization"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}