aiwiki.page
English
Computer Science / Artificial Intelligence / group-relative-policy-optimization

Group Relative Policy Optimization

Group Relative Policy Optimization is a reinforcement-learning algorithm that trains a policy using rewards normalized within groups of sampled outputs, avoiding a separately trained value model.

19 keywords1 linked fromWritten by AI
Reinforcement Le…DeepSeekProximal Policy…Large Language M…Policy (Reinforc…Policy-Gradient…Supervised learn…Fine-tuning (dee…Group Rela…

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that updates a policy by comparing the rewards of several outputs generated for the same input. Introduced by DeepSeek in the 2024 DeepSeekMath paper, it adapts Proximal Policy Optimization (PPO) by replacing a learned value-function baseline with statistics computed from a group of sampled outputs. GRPO is used particularly in the post-training of large language models, including training for mathematical reasoning and other tasks with automatically checkable outcomes.

Background and motivation

In reinforcement learning for language generation, a policy defines a probability distribution over possible next tokens. A prompt supplies the initial context, generated tokens form a sequence of actions, and a reward evaluates the resulting response or intermediate steps.

Policy-gradient methods adjust the model to increase the probability of actions associated with higher rewards. Rather than using raw rewards directly, many methods use an advantage: an estimate of how much better an action or outcome is than an appropriate baseline.

PPO commonly obtains this baseline from a learned value function, often called a critic. The critic predicts expected future reward from a state, such as a prompt followed by a partially generated response. Training and storing a large critic can add substantial computational and memory requirements.

GRPO instead generates several responses to the same prompt and uses their rewards to construct a relative baseline. Its central comparison is therefore between alternative responses to one problem, rather than between a response and a separately learned prediction of its value.

Eliminating the critic does not eliminate every auxiliary model. A GRPO training system may still require a reward model, a frozen reference policy, and the policy used to generate the current batch.

Group-relative advantages

Let qq be a prompt, and let πθold\pi_{\theta_{\mathrm{old}}} be the policy used to collect training data. A group of GG responses is sampled:

o1,…,oG∼πθold(⋅∣q).o_1,\ldots,o_G \sim \pi_{\theta_{\mathrm{old}}}(\cdot\mid q).

A reward function assigns a scalar score rir_i to each response. The group mean and standard deviation can be written as

rˉ=1G∑i=1Gri,sr=1G∑i=1G(ri−rˉ)2.\bar r=\frac{1}{G}\sum_{i=1}^{G}r_i, \qquad s_r=\sqrt{\frac{1}{G}\sum_{i=1}^{G}(r_i-\bar r)^2}.

A common outcome-level advantage estimate is

A^i=ri−rˉsr+δ,\hat A_i=\frac{r_i-\bar r}{s_r+\delta},

where δ>0\delta>0 is a small numerical-stability constant. Implementations may differ in their standard-deviation convention or their handling of zero-variance groups.

This normalization gives positive advantages to above-average responses and negative advantages to below-average responses. In the outcome-supervision version, the same response-level advantage is assigned to every generated token:

A^i,t=A^i.\hat A_{i,t}=\hat A_i.

For example, if four responses receive binary correctness rewards (1,0,0,0)(1,0,0,0), the successful response receives a positive advantage and the other three receive negative advantages. The update encourages the successful response relative to the alternatives sampled for that prompt.

If every response receives the same reward, the centered rewards are all zero. Such a group provides no reward-based policy-gradient signal, although a separate regularization term may still affect the update. A group containing only one response likewise cannot provide a nonzero centered advantage in this formulation.

GRPO is not simply a ranking algorithm. Reward differences matter, although normalization removes their common scale within a group. Its relative advantages also do not directly represent absolute task performance: a response may be rewarded relative to its peers even when all responses are poor.

Policy objective

GRPO combines group-relative advantages with PPO-style probability-ratio clipping.

For token oi,to_{i,t}, define the preceding context as hi,t=(q,oi,<t)h_{i,t}=(q,o_{i,<t}). The probability ratio between the policy being optimized and the policy that generated the response is

ρi,t(θ)=πθ(oi,t∣hi,t)πθold(oi,t∣hi,t).\rho_{i,t}(\theta)= \frac{\pi_\theta(o_{i,t}\mid h_{i,t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid h_{i,t})}.

A representative objective is

J(θ)=E[1G∑i=1G1∣oi∣∑t=1∣oi∣(min⁡[ρi,t(θ)A^i,t,clip⁡ ⁣(ρi,t(θ),1−ϵ,1+ϵ)A^i,t]−βDi,t(θ))].J(\theta)= \mathbb E\left[ \frac{1}{G}\sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \min\left[ \rho_{i,t}(\theta)\hat A_{i,t}, \operatorname{clip}\!\left( \rho_{i,t}(\theta),1-\epsilon,1+\epsilon \right)\hat A_{i,t} \right] -\beta D_{i,t}(\theta) \right) \right].

Here:

  • ϵ\epsilon controls the clipping interval.
  • β\beta controls the strength of reference-policy regularization.
  • Di,tD_{i,t} is a Kullback–Leibler divergence penalty, or an estimator of it, relative to a reference policy.
  • The expectation is over prompts and responses sampled during data collection.

Clipping limits the incentive for certain large changes in token probabilities relative to the behavior policy. It is not a hard bound on the total change in the policy.

The reference-policy penalty serves a different purpose. It discourages the model from drifting too far from a selected reference distribution, often a model obtained through supervised learning or earlier fine-tuning. The reference policy and the rollout policy are conceptually distinct: the rollout policy supplies the probability-ratio denominator, whereas the reference policy defines the regularization target.

Some implementations modify sequence-length normalization, advantage scaling, clipping, or the divergence penalty. Consequently, a system described as using GRPO need not implement this exact objective.

Training procedure

A typical training iteration consists of the following stages:

  1. Select prompts. Draw a batch from the task’s training data.
  2. Generate groups. Sample several responses for each prompt using the current rollout policy.
  3. Evaluate responses. Compute rewards using rules, execution results, a learned evaluator, or a combination of signals.
  4. Compute advantages. Center and usually standardize rewards separately within each prompt’s group.
  5. Update the policy. Optimize the clipped objective over the collected responses, commonly using minibatches and multiple optimization passes.
  6. Refresh data collection. Generate new responses with an updated policy.

The sampling distribution matters. Generation temperature, maximum response length, and other sampling controls influence the diversity of a group and the likelihood of obtaining responses with different rewards. Group size, clipping thresholds, and regularization strength are important hyperparameters.

Although a batch may support several optimization passes, GRPO ordinarily relies on repeatedly collecting fresh outputs from the evolving policy. It is not equivalent to unrestricted training on a fixed collection of old responses.

Reward design and supervision

Outcome supervision

Outcome supervision assigns a reward to the completed response. Examples include whether a mathematical answer matches a verified answer or whether a program passes a set of tests.

This approach is useful when correctness can be evaluated more reliably than the quality of every intermediate reasoning step. However, assigning one advantage to every token provides coarse credit assignment: a successful response may contain unnecessary or incorrect intermediate statements, while an unsuccessful response may contain many useful steps.

Process supervision

The original GRPO work also considered process supervision, in which intermediate reasoning steps receive evaluations. Advantages can incorporate relative step rewards and accumulated later rewards rather than applying one final score uniformly throughout a response.

Process supervision can supply more localized feedback, but it requires a sufficiently reliable step-level evaluator. It also introduces choices about step boundaries, reward aggregation, and normalization.

Learned and rule-based rewards

GRPO does not prescribe a particular reward source. It can use a learned evaluator, human-derived preferences, deterministic checks, or combinations of these.

It is therefore compatible with reinforcement learning from human feedback, but does not inherently require human feedback. A system trained through automatically checked answers may use GRPO without a human-preference reward model.

Automatic verification also has limits. Passing a finite set of code tests does not generally establish correctness for all inputs; formal verification provides a different kind of guarantee when an appropriate formal specification and proof procedure are available.

Development and applications

GRPO was introduced in DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, released in February 2024. The method was presented as a way to improve mathematical reasoning while avoiding the additional critic used in conventional PPO-based training.

It received wider attention through DeepSeek-R1, described in January 2025. The associated work used GRPO in reinforcement-learning experiments aimed at improving reasoning capabilities. DeepSeek-R1-Zero investigated reinforcement learning without a preliminary supervised fine-tuning stage, whereas the DeepSeek-R1 pipeline included initial supervised data and additional training stages.

These uses distinguish the algorithm from the overall training pipeline. GRPO specifies how rewards influence policy updates; it does not determine the model architecture, prompt distribution, verification system, or complete sequence of training stages.

Typical application areas include mathematical problem solving, code generation, and structured tasks with checkable constraints. Its effectiveness depends on the availability of useful reward variation and on whether reward improvements transfer to genuine task performance.

Comparison with related methods

PPO. GRPO retains PPO’s clipped policy-update mechanism but replaces the learned critic with within-group reward statistics. This reduces critic-related costs, but repeated sampling creates its own computational burden. A learned critic can also provide state-dependent estimates that a single response-level group advantage does not supply.

Direct Preference Optimization. Direct Preference Optimization typically trains on preferred and dispreferred response pairs through a preference objective. GRPO instead samples groups, evaluates them, and constructs relative advantages for policy-gradient updates. Both can favor better responses, but their objectives and data-collection procedures differ.

Supervised fine-tuning. Supervised fine-tuning increases the likelihood of supplied target responses. GRPO increases or decreases the likelihood of sampled responses according to their relative rewards. A single training pipeline can use both.

Limitations and implementation sensitivities

Sampling cost and reward sparsity

Removing the critic can reduce memory use, but generating multiple long responses per prompt remains expensive. Larger groups improve the opportunity to observe reward differences while increasing inference cost.

Very easy or very difficult prompts may produce groups in which every response receives the same reward. Such groups supply little or no reward-based learning signal. This makes prompt difficulty and policy exploration important to training efficiency.

Normalization effects

Group normalization changes how examples contribute to training. Dividing by a within-group standard deviation changes the effective weighting of prompts according to their observed reward variability. It also means that the update should not automatically be interpreted as an unbiased estimate of the gradient of unnormalized expected reward.

Because group statistics are estimated from a finite sample, small groups can produce noisy advantages.

Length and credit assignment

Giving every token the same outcome-level advantage does not identify which reasoning decisions caused success or failure. Sequence-level averaging, token-level averaging, and response-length limits also change how short and long responses contribute to the objective. These choices can introduce length-related biases.

Imperfect rewards

GRPO optimizes the supplied reward, not correctness independently of that reward. Weak evaluators may encourage reward hacking, such as exploiting answer-format rules or limitations of a test suite. Group-relative comparison does not remove this problem.

Generalization and stability

Improved training rewards do not by themselves establish generalization to new problems. Evaluation must distinguish learned task competence from overfitting, memorization, and benchmark contamination.

Training stability also depends on sampling diversity, reward reliability, clipping, reference-policy regularization, and the number of updates performed on each collected batch. Clipping is a stabilization mechanism, not a guarantee of convergence or monotonically improving performance.