{
  "id": "proximal-policy-optimization",
  "title": "Proximal Policy Optimization",
  "lang": "en",
  "source_lang": "en",
  "summary": "Proximal Policy Optimization is a family of reinforcement-learning algorithms that stabilizes policy-gradient training by discouraging excessively large policy updates.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "PPO",
    "Proximal Policy Optimization algorithms"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/proximal-policy-optimization/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/proximal-policy-optimization/",
      "json": "https://aiwiki.page/zh/proximal-policy-optimization.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Proximal Policy Optimization (PPO) is a family of reinforcement-learning algorithms for training an agent to choose actions that maximize expected cumulative reward. Introduced in 2017 by John Schulman and colleagues, PPO uses a relatively simple optimization objective to discourage policy updates that change behavior too abruptly. Its most widely used variant, PPO-Clip, limits the incentive to move action probabilities too far from those of the policy that collected the training data. PPO is used in simulated control, robotics research, game playing, and the fine-tuning of language models."
    },
    {
      "heading": "Background and motivation",
      "text": "A reinforcement-learning agent interacts with an environment: it observes a state, selects an action, receives a reward, and continues to a new state. A policy specifies how actions are selected. In PPO, the policy is usually stochastic and parameterized by an artificial neural network.\n\nWriting the policy as \\(\\pi_\\theta(a\\mid s)\\), where \\(\\theta\\) denotes its parameters, a common objective is\n\n\\[J(\\theta) = \\mathbb{E}_{\\tau\\sim\\pi_\\theta} \\left[ \\sum_{t=0}^{T-1}\\gamma^t R_t \\right],\\]\n\nwhere \\(\\tau\\) is a trajectory, \\(R_t\\) is the reward at time \\(t\\), and \\(\\gamma\\) is a discount factor. This formulation commonly assumes a Markov decision process.\n\nPolicy-gradient methods estimate how changing the policy parameters would change expected return. Their updates increase the probability of actions associated with better-than-expected outcomes and decrease the probability of actions associated with worse-than-expected outcomes.\n\nA central difficulty is selecting the update size. A large update may sharply reduce performance, making the data collected by the previous policy a poor guide to the new policy. Very small updates, however, can make learning slow and waste expensive environment interactions. PPO addresses this problem by modifying the objective used to update the policy."
    },
    {
      "heading": "Historical development",
      "text": "PPO followed Trust Region Policy Optimization (TRPO), introduced in 2015. TRPO optimizes a surrogate policy objective subject to a constraint on the average divergence between the old and new policies. Its motivation comes from bounds connecting policy changes to changes in expected return, although its practical algorithm uses approximations to those bounds.\n\nTRPO requires more elaborate numerical machinery than ordinary first-order training, including operations involving curvature information. PPO was designed to retain some of the practical stability associated with conservative policy updates while using simpler minibatch optimization.\n\nThe 2017 paper Proximal Policy Optimization Algorithms presented both a clipped objective and an adaptive divergence-penalty objective. The clipped version became the more commonly used form. Consequently, “PPO” often refers specifically to PPO-Clip rather than to the entire family."
    },
    {
      "heading": "The clipped policy objective",
      "text": "Probability ratio and advantage\n\nPPO collects a batch of trajectories using a fixed behavior policy, denoted by \\(\\pi_{\\theta_{\\mathrm{old}}}\\). During optimization on that batch, it compares the current policy with this behavior policy through the probability ratio\n\n\\[r_t(\\theta) = \\frac{\\pi_\\theta(a_t\\mid s_t)} {\\pi_{\\theta_{\\mathrm{old}}}(a_t\\mid s_t)}.\\]\n\nFor continuous actions, the numerator and denominator are probability densities.\n\nThe ratio indicates how the update changes the probability of the action actually sampled. A ratio greater than one means that the current policy makes that action more likely.\n\nEach sample also has an estimated advantage, \\(\\hat A_t\\), which measures how much better or worse the action was than the policy’s typical behavior in that state. The underlying advantage function is\n\n\\[A^\\pi(s,a)=Q^\\pi(s,a)-V^\\pi(s),\\]\n\nwhere \\(Q^\\pi\\) is the action-value function and \\(V^\\pi\\) is the state-value function.\n\nA basic surrogate objective would maximize the empirical average of \\(r_t(\\theta)\\hat A_t\\). Repeated optimization of that objective on a fixed batch can encourage excessive changes in action probabilities.\n\nClipping\n\nPPO-Clip instead maximizes\n\n\\[L^{\\mathrm{CLIP}}(\\theta) = \\hat{\\mathbb{E}}_t \\left[ \\min \\left( r_t(\\theta)\\hat A_t,\\, \\operatorname{clip} \\bigl(r_t(\\theta),1-\\epsilon,1+\\epsilon\\bigr)\\hat A_t \\right) \\right],\\]\n\nwhere \\(\\hat{\\mathbb{E}}_t\\) denotes an empirical average and \\(\\epsilon>0\\) is a clipping hyperparameter.\n\nThe clipping function restricts its argument to the interval \\([1-\\epsilon,1+\\epsilon]\\). The minimum selects the more pessimistic of the unclipped and clipped surrogate terms:\n\n- Positive advantage: increasing the action’s probability improves the objective, but the clipped term stops rewarding further increases once \\(r_t>1+\\epsilon\\).\n- Negative advantage: decreasing the action’s probability improves the objective, but the clipped term stops rewarding further decreases once \\(r_t<1-\\epsilon\\).\n\nChanges in the opposite, harmful direction remain penalized rather than being hidden by clipping.\n\nFor example, if an action has positive estimated advantage and \\(\\epsilon=0.2\\), increasing its probability ratio from \\(1\\) to \\(1.1\\) improves its contribution to the objective. Increasing the ratio beyond \\(1.2\\) produces no additional improvement in that sample’s clipped surrogate term.\n\nWhat clipping does not guarantee\n\nThe clipping interval is not a hard constraint on policy probabilities. PPO clips a term in the objective, not the updated policy itself. Shared neural-network parameters, updates from other samples, and auxiliary losses can still move a probability ratio outside the interval.\n\nNor does clipping guarantee a fixed upper bound on the divergence between the complete old and new action distributions. PPO-Clip therefore does not inherit a general guarantee of monotonic improvement merely from its clipped objective."
    },
    {
      "heading": "Actor–critic training and advantage estimation",
      "text": "PPO commonly uses an actor–critic architecture:\n\n- The actor represents the policy.\n- The critic estimates expected return from a state.\n\nThe actor and critic may use separate networks or share some parameters. The critic supplies a baseline that helps reduce the variance of policy-gradient estimates.\n\nA common estimator is Generalized Advantage Estimation (GAE). It combines temporal-difference residuals,\n\n\\[\\delta_t = R_t+\\gamma V_\\phi(s_{t+1})-V_\\phi(s_t),\\]\n\ninto\n\n\\[\\hat A_t = \\sum_{l=0}^{T-t-1} (\\gamma\\lambda)^l\\delta_{t+l}.\\]\n\nHere, \\(\\phi\\) denotes critic parameters and \\(\\lambda\\) controls a bias–variance trade-off. Terminal-state masks and appropriate bootstrapping are required in implementations. In particular, a genuinely terminal state and an episode stopped only because of a time limit may require different treatment.\n\nAdvantages are often normalized within a batch or minibatch. This is a practical implementation choice rather than part of the defining clipped objective.\n\nCombined training loss\n\nA typical implementation minimizes a loss function of the form\n\n\\[\\mathcal{L} = -L^{\\mathrm{CLIP}} +c_V\\mathcal{L}_{V} -c_H\\hat{\\mathbb{E}}_t \\left[ \\mathcal{H}\\bigl(\\pi_\\theta(\\cdot\\mid s_t)\\bigr) \\right],\\]\n\nwhere:\n\n- \\(\\mathcal{L}_{V}\\) trains the value function, often using squared error against estimated returns;\n- \\(\\mathcal{H}\\) is policy entropy, which can encourage exploration;\n- \\(c_V\\) and \\(c_H\\) weight these additional terms.\n\nValue-function clipping appears in some implementations but is not essential to PPO-Clip. Updates are typically computed through backpropagation using an optimizer such as Adam."
    },
    {
      "heading": "Training procedure",
      "text": "A standard PPO training cycle consists of the following steps:\n\n- Collect experience. Run the current policy in one or more environments and store observations, actions, rewards, value estimates, and the log probabilities of sampled actions.\n- Compute learning targets. Estimate advantages and value-function targets from the collected trajectories.\n- Freeze behavior-policy probabilities. Preserve the probabilities or log probabilities produced during collection.\n- Optimize on the batch. Shuffle the data into minibatches and perform several epochs of actor and critic updates.\n- Collect fresh experience. Use the updated policy to generate the next batch.\n\nThe behavior-policy probabilities remain fixed throughout optimization on a given batch. Recomputing them from the changing current policy would destroy the intended meaning of the ratio.\n\nPPO is generally classified as on-policy, despite reusing each collected batch for several optimization epochs. Its data come from a recent policy and are normally discarded after the update cycle. It does not ordinarily use a long-lived replay buffer in the manner of many off-policy algorithms.\n\nParallel environment execution is common because it increases data-collection throughput. Batch size, optimization epochs, minibatch size, clipping threshold, and learning rate jointly affect how far the policy moves during each cycle."
    },
    {
      "heading": "Penalty-based PPO and divergence monitoring",
      "text": "An alternative formulation penalizes policy change using Kullback–Leibler divergence:\n\n\\[L^{\\mathrm{KLPEN}}(\\theta) = \\hat{\\mathbb{E}}_t \\left[ r_t(\\theta)\\hat A_t - \\beta D_{\\mathrm{KL}} \\left( \\pi_{\\theta_{\\mathrm{old}}}(\\cdot\\mid s_t) \\,\\Vert\\, \\pi_\\theta(\\cdot\\mid s_t) \\right) \\right].\\]\n\nThe penalty coefficient \\(\\beta\\) can be adapted according to whether the measured divergence is above or below a target.\n\nEven when using PPO-Clip, implementations often monitor approximate divergence and stop optimization on a batch early if it exceeds a threshold. Another diagnostic, the clip fraction, measures how often sampled probability ratios fall outside the clipping interval. Neither statistic, by itself, establishes that the policy has improved."
    },
    {
      "heading": "Applications",
      "text": "Control and simulated environments\n\nPPO supports both discrete and continuous action spaces and is widely used as a baseline in deep reinforcement learning. Applications include simulated locomotion, manipulation, navigation, and game environments.\n\nIts comparatively simple optimizer and compatibility with parallel simulation make it useful when many environment interactions can be generated. Successful simulated behavior does not by itself establish reliable real-world performance: simulator inaccuracies and distribution shift can affect deployment.\n\nLanguage-model training\n\nPPO has been used in reinforcement learning from human feedback to train large language models. In this setting, the policy generates tokens conditioned on a prompt and previously generated text. A learned reward model evaluates generated responses.\n\nSuch systems often penalize divergence from a fixed reference model to discourage excessive departure from the starting language model. This reference-model penalty has a different purpose from PPO’s comparison with the behavior policy:\n\n- The behavior policy changes between rollout cycles and supplies the denominator in PPO’s probability ratio.\n- The reference model may remain fixed and constrains longer-term departure from a chosen model.\n\nPPO is not required for all preference-based training. Direct Preference Optimization, for example, uses a different objective and does not require the same online PPO rollout-and-critic procedure."
    },
    {
      "heading": "Strengths and limitations",
      "text": "PPO combines a relatively compact objective with flexible neural-network architectures and ordinary minibatch optimization. Its ability to perform several passes over a newly collected batch makes better use of that batch than a single gradient update would.\n\nNevertheless, important limitations remain:\n\n- Interaction cost. Because PPO repeatedly requires fresh trajectories, it can be less sample-efficient than well-suited off-policy methods.\n- Sensitivity to implementation. Reward scaling, observation normalization, initialization, advantage estimation, episode-boundary handling, and optimizer settings can substantially affect results.\n- No universal stability guarantee. Clipping reduces some incentives for large updates but cannot prevent all destructive changes.\n- Dependence on critic quality. Inaccurate value estimates can produce misleading advantage estimates.\n- Exploration difficulties. PPO does not inherently solve sparse rewards, long-horizon credit assignment, or environments requiring coordinated exploratory behavior.\n- Reward misspecification. PPO optimizes the supplied reward; it does not establish that the reward captures the intended task. Reward hacking remains possible.\n- Limited data-reuse horizon. Too many optimization passes can make the current policy substantially different from the behavior policy and encourage overfitting to the collected batch.\n\nPPO’s “proximal” character therefore describes a practical method for encouraging conservative policy updates, rather than an exact projection onto a guaranteed safe set of policies."
    }
  ],
  "links": [
    "reinforcement-learning",
    "fine-tuning",
    "policy-reinforcement-learning",
    "artificial-neural-network",
    "markov-decision-process",
    "policy-gradient-method",
    "hyperparameter",
    "variance",
    "temporal-difference-learning",
    "loss-function",
    "backpropagation",
    "adam-optimizer",
    "parallel-computing",
    "learning-rate",
    "kullback-leibler-divergence",
    "deep-learning",
    "distribution-shift",
    "reinforcement-learning-from-human-feedback",
    "large-language-model",
    "direct-preference-optimization",
    "reward-hacking",
    "overfitting"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}