GRPO's group mean is a Monte Carlo baseline that replaces the critic

PPO trains a critic, a second network about the size of the policy, to estimate the value that serves as its baseline. GRPO, introduced with DeepSeekMath [1], drops the critic and gets the baseline by sampling instead.

The baseline is there to cut variance. The policy gradient pushes up each sampled answer's log-probability in proportion to its reward, r ∇θlog⁡πθ(o∣q)r\,\nabla_\theta \log \pi_\theta(o \mid q). If every reward is positive, every answer gets pushed up, and the signal that matters, which answers beat which, rides on a shared offset that only adds noise. Subtracting any baseline bb that doesn't depend on the answer removes the offset without changing the expected gradient, because

Eo∼πθ[ b ∇θlog⁡πθ(o∣q)]=b ∇θ∑oπθ(o∣q)=b ∇θ1=0.\mathbb{E}_{o \sim \pi_\theta}\big[\, b\,\nabla_\theta \log \pi_\theta(o \mid q) \big] = b\,\nabla_\theta \sum_o \pi_\theta(o \mid q) = b\,\nabla_\theta 1 = 0.

This is a control variate, the Monte Carlo trick of subtracting a zero-mean term that is correlated with the noise. The natural choice is the question's expected reward, which is what the critic estimates.

GRPO estimates it directly. For each question it samples a group of GG answers (DeepSeekMath used 64), scores them to get rewards r=(r1,…,rG)\mathbf r = (r_1, \dots, r_G), and takes their mean as the baseline:

A^i=ri−mean⁡(r)std⁡(r).\hat A_i = \frac{r_i - \operatorname{mean}(\mathbf r)}{\operatorname{std}(\mathbf r)}.

Because the mean includes rir_i itself, this baseline does depend on answer ii, but only by a constant factor: ri−mean⁡(r)r_i - \operatorname{mean}(\mathbf r) is exactly G−1G\tfrac{G-1}{G} times the leave-one-out advantage (RLOO), which is unbiased. Same direction, slightly smaller step.

Dividing by the standard deviation is a separate choice, and not a neutral one. It reweights questions: those where the group's rewards barely vary, because they are almost always solved or almost always failed, get scaled up. Liu et al. show that this changes what GRPO optimizes [2].

References

  1. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models [PDF]
    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y. and Guo, D., 2024. arXiv. DOI: 10.48550/ARXIV.2402.03300

  2. Understanding R1-Zero-Like Training: A Critical Perspective [PDF]
    Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S. and Lin, M., 2025. arXiv. DOI: 10.48550/ARXIV.2503.20783