{
  "id": "reinforce-algorithm",
  "title": "REINFORCE Algorithm",
  "lang": "en",
  "source_lang": "en",
  "summary": "REINFORCE is a Monte Carlo policy-gradient algorithm that learns a stochastic policy by weighting sampled action log-probability gradients by observed returns.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Monte Carlo policy gradient",
    "REINFORCE policy gradient"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/reinforce-algorithm/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/reinforce-algorithm/",
      "json": "https://aiwiki.page/zh/reinforce-algorithm.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "REINFORCE is an reinforcement learning algorithm that adjusts a stochastic policy to maximize expected cumulative reward. It belongs to the family of policy-gradient methods: rather than first learning which action has the highest value, it directly changes the parameters of the distribution from which actions are sampled. Its characteristic update multiplies the gradient of an action’s log-probability by a sampled return, increasing the probability of actions associated with favorable outcomes and decreasing it for unfavorable outcomes relative to any baseline used."
    },
    {
      "heading": "Origins and terminology",
      "text": "Ronald J. Williams introduced the REINFORCE family of statistical gradient-following algorithms in a 1992 paper, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. The work described methods for adjusting stochastic units, including units in artificial neural networks, using a scalar reinforcement signal.\n\nIn contemporary reinforcement learning, “REINFORCE” most often denotes an episodic, Monte Carlo policy-gradient algorithm. More broadly, the name is also used for the underlying score-function gradient estimator, which can optimize expectations involving sampled discrete or continuous variables outside a sequential decision problem.\n\nThe estimator is closely related to likelihood-ratio methods in simulation and mathematical optimization. Its distinguishing feature is that it differentiates the probability of a sampled outcome rather than differentiating through the outcome itself."
    },
    {
      "heading": "Mathematical setting",
      "text": "Consider a finite-horizon Markov decision process. At time \\(t\\), an agent observes state \\(s_t\\), samples action \\(a_t\\) from a parameterized policy \\(\\pi_\\theta(a_t\\mid s_t)\\), and receives reward \\(r_t\\). The environment then produces the next state.\n\nFor an episode with \\(T\\) actions, define the discounted return from time \\(t\\) as\n\n\\[G_t=\\sum_{k=t}^{T-1}\\gamma^{k-t}r_k,\\]\n\nwhere \\(0\\leq\\gamma\\leq1\\) is the discount factor. A standard objective is\n\n\\[J(\\theta)= \\mathbb{E}_{\\tau\\sim p_\\theta} \\left[ \\sum_{t=0}^{T-1}\\gamma^t r_t \\right],\\]\n\nwhere \\(\\tau\\) denotes a complete trajectory and \\(p_\\theta\\) is its probability distribution under the policy.\n\nREINFORCE estimates the gradient of this expected value from sampled episodes. It does not require a model of the environment’s transition probabilities or derivatives of its reward function."
    },
    {
      "heading": "The score-function identity",
      "text": "For a sampled variable \\(x\\) and a function \\(f(x)\\) that does not explicitly depend on \\(\\theta\\), the central identity is\n\n\\[\\nabla_\\theta\\mathbb{E}_{x\\sim p_\\theta}[f(x)] = \\mathbb{E}_{x\\sim p_\\theta} \\left[ f(x)\\nabla_\\theta\\log p_\\theta(x) \\right].\\]\n\nThis follows from\n\n\\[\\nabla_\\theta p_\\theta(x) = p_\\theta(x)\\nabla_\\theta\\log p_\\theta(x),\\]\n\nunder conditions allowing differentiation and integration to be interchanged. Standard applications assume differentiable probabilities, suitable integrability, and support that does not change with the parameters. If \\(f\\) also depends explicitly on \\(\\theta\\), its direct derivative contributes an additional term.\n\nFor a trajectory, assuming the initial-state distribution and environment dynamics do not depend on the policy parameters,\n\n\\[p_\\theta(\\tau)= \\rho_0(s_0) \\prod_{t=0}^{T-1} \\pi_\\theta(a_t\\mid s_t) P(s_{t+1}\\mid s_t,a_t).\\]\n\nConsequently,\n\n\\[\\nabla_\\theta\\log p_\\theta(\\tau) = \\sum_{t=0}^{T-1} \\nabla_\\theta\\log\\pi_\\theta(a_t\\mid s_t).\\]\n\nThe unknown environment probabilities disappear from this derivative. This is what permits learning from interaction without differentiating through the environment.\n\nReward-to-go\n\nA direct application of the identity weights every action’s log-probability gradient by the entire episode return. However, rewards received before an action cannot be caused by that action. Their contributions to its expected gradient are zero, so they can be removed.\n\nFor the discounted objective above, this gives\n\n\\[\\nabla_\\theta J(\\theta) = \\mathbb{E} \\left[ \\sum_{t=0}^{T-1} \\gamma^t G_t \\nabla_\\theta\\log\\pi_\\theta(a_t\\mid s_t) \\right].\\]\n\nUsing reward-to-go generally reduces unnecessary noise compared with weighting every action by the complete return.\n\nThe outer factor \\(\\gamma^t\\) follows from this particular start-state discounted objective. Presentations using undiscounted episodic returns set \\(\\gamma=1\\); formulations based on other objectives or sampling distributions may use different weighting conventions. Omitting the factor is not generally an exact gradient estimator for the objective written above when \\(\\gamma<1\\)."
    },
    {
      "heading": "Algorithm and implementation",
      "text": "A basic batch version proceeds as follows:\n\n- Initialize the policy parameters \\(\\theta\\).\n- Collect complete episodes using the current policy.\n- Compute \\(G_t\\) for every action in each episode.\n- Form the average sampled policy gradient.\n- Update the parameters by gradient ascent.\n- Collect new episodes and repeat.\n\nFor \\(N\\) episodes, a gradient estimate is\n\n\\[\\widehat g= \\frac{1}{N} \\sum_{i=1}^{N} \\sum_{t=0}^{T_i-1} \\gamma^tG_{i,t} \\nabla_\\theta \\log\\pi_\\theta(a_{i,t}\\mid s_{i,t}),\\]\n\nwith update\n\n\\[\\theta\\leftarrow\\theta+\\alpha\\widehat g,\\]\n\nwhere \\(\\alpha\\) is the learning rate.\n\nWhen using software that minimizes a loss function, the same update can be implemented through the sampled surrogate loss\n\n\\[\\mathcal L_{\\text{policy}}(\\theta)= -\\frac{1}{N} \\sum_{i,t} \\gamma^t \\operatorname{stopgrad}(G_{i,t}) \\log\\pi_\\theta(a_{i,t}\\mid s_{i,t}).\\]\n\nHere, \\(\\operatorname{stopgrad}\\) means that the return is treated as a fixed weight during differentiation. Backpropagation computes the derivatives of the log-probabilities through the policy network; it does not differentiate through sampled actions or environment transitions. The surrogate supplies the desired gradient at the policy that generated the samples, rather than directly evaluating the expected-return objective.\n\nFor discrete actions, the policy often uses a categorical distribution. For continuous actions, it may output the parameters of a Gaussian distribution. In both cases, REINFORCE requires a tractable log-probability for the sampled action."
    },
    {
      "heading": "Baselines and variance reduction",
      "text": "REINFORCE estimates can have high variance, because returns depend on many random actions and environmental events. A standard modification subtracts a baseline:\n\n\\[\\widehat g= \\sum_t \\gamma^t \\bigl(G_t-b(s_t)\\bigr) \\nabla_\\theta\\log\\pi_\\theta(a_t\\mid s_t).\\]\n\nA baseline that does not depend on the current action has zero expected contribution, since\n\n\\[\\mathbb{E}_{a\\sim\\pi_\\theta(\\cdot\\mid s)} \\left[ b(s)\\nabla_\\theta\\log\\pi_\\theta(a\\mid s) \\right] = b(s)\\nabla_\\theta\\sum_a\\pi_\\theta(a\\mid s) = 0.\\]\n\nThe corresponding identity holds for continuous actions with an integral.\n\nA common baseline estimates the state value,\n\n\\[V^\\pi(s)=\\mathbb{E}[G_t\\mid s_t=s].\\]\n\nThen \\(G_t-V^\\pi(s_t)\\) compares the observed outcome with what was expected from that state. The baseline may be learned by regression on sampled returns. It must be treated as fixed in the policy-gradient calculation, even if it has its own trainable parameters.\n\nSubtracting an appropriate baseline can reduce variance without changing the expected gradient. An inaccurate action-independent baseline does not, by itself, introduce bias, but it may provide little variance reduction or even increase variance. Finite-sample dependencies also matter: centering returns using the same batch’s sample mean is not automatically an exactly unbiased implementation.\n\nOther common modifications include larger batches and scaling or normalizing returns or advantage estimates. These can improve numerical behavior, but sample-dependent normalization need not preserve exact unbiasedness. Adding an exploration bonus changes the optimization objective rather than merely reducing estimator variance."
    },
    {
      "heading": "Relationship to other reinforcement learning methods",
      "text": "Actor–critic methods\n\nREINFORCE uses completed sampled returns rather than bootstrapped targets. An actor–critic method combines a policy, the actor, with a learned value estimator, the critic. Many actor–critic algorithms use temporal-difference learning to construct lower-variance policy updates before an episode ends.\n\nREINFORCE with a learned state-value baseline can be described as having an actor and a critic in a broad architectural sense. However, the usual algorithmic distinction is whether the policy update uses full Monte Carlo returns or bootstrapped estimates. Bootstrapping can reduce variance and improve data efficiency, while introducing dependence on value-estimation accuracy.\n\nValue-based methods\n\nQ-learning learns action values and derives a policy from them. REINFORCE instead optimizes an explicitly parameterized stochastic policy.\n\nDirect policy parameterization is convenient for continuous actions and problems requiring randomized behavior. It does not, however, eliminate the difficulties of delayed rewards, exploration, or assigning credit to individual decisions.\n\nModern policy optimization\n\nProximal policy optimization also uses sampled policy-gradient information, but modifies the update objective to discourage excessively large changes and enable controlled reuse of collected data. Basic REINFORCE has no comparable constraint.\n\nREINFORCE is ordinarily on-policy: trajectories should be sampled from the policy whose gradient is being estimated. Reusing trajectories from a different policy without correction generally produces a biased estimate. Importance sampling can correct distribution mismatch under suitable support conditions, but may substantially increase variance."
    },
    {
      "heading": "Applications and limitations",
      "text": "REINFORCE provides a general learning mechanism when a system can sample decisions and evaluate their outcomes, even when those outcomes are not differentiable. Applications include control, discrete sequence generation, and optimization involving stochastic computational choices.\n\nIn natural language processing, a model can sample a sequence and use a sequence-level reward to weight the gradients of token log-probabilities. This connects the estimator to reward-based fine-tuning and some formulations of reinforcement learning from human feedback, although particular training systems often use more elaborate objectives.\n\nThe method’s main limitations are:\n\n- High sampling variance. Long horizons and noisy rewards can require many trajectories for a useful update.\n- Sparse feedback. If rewarding outcomes are rarely sampled, the algorithm receives little information about how to improve.\n- Delayed credit assignment. Reward-to-go does not fully identify which of many earlier decisions caused an outcome.\n- Limited data reuse. Its basic on-policy form discards data after policy updates rather than freely reusing a fixed dataset.\n- Sensitivity to update scale. Reward magnitudes, learning rates, and policy parameterization can strongly affect stability.\n- Local optimization. Even an unbiased gradient estimator does not guarantee a globally optimal policy, particularly with nonlinear function approximation.\n\nThese properties make REINFORCE especially important as a foundational estimator and reference algorithm, while explaining the use of baselines, critics, and constrained update mechanisms in more elaborate policy-gradient systems."
    }
  ],
  "links": [
    "reinforcement-learning",
    "policy-reinforcement-learning",
    "policy-gradient-method",
    "artificial-neural-network",
    "mathematical-optimization",
    "markov-decision-process",
    "probability-distribution",
    "expected-value",
    "learning-rate",
    "loss-function",
    "backpropagation",
    "variance",
    "temporal-difference-learning",
    "q-learning",
    "proximal-policy-optimization",
    "importance-sampling",
    "natural-language-processing",
    "fine-tuning",
    "reinforcement-learning-from-human-feedback"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}