Policy-gradient estimators can have high variance. Baselines reduce variance without changing the mean when they do not depend on the sampled action in a way that breaks the score identity.
Unbiased baselines
The baseline principle is an immediate consequence of policy normalization.
Theorem. For any state-dependent baseline \(b(S_t)\), \[ \mathbb E_{\pi_\theta}\left[\nabla\log\pi_\theta(A_t\mid S_t)b(S_t)\mid S_t\right]=0. \]
Proof. Condition on \(S_t=s\). Then \[ \int \nabla\log\pi_\theta(a\mid s)b(s)\pi_\theta(\mathrm{d}a\mid s) =b(s)\int\nabla\pi_\theta(\mathrm{d}a\mid s) =b(s)\nabla1=0. \]
Optimal scalar baseline
For a single score vector, the variance-minimizing constant baseline has a weighted least-squares form.
Proposition. For an estimator \(g=(G-b)Z\), where \(Z=\nabla\log\pi_\theta(A\mid S)\), the constant \(b\) minimizing \(\mathbb E\|g\|_2^2\) is \[ b^*=\frac{\mathbb E(G\|Z\|_2^2)}{\mathbb E\|Z\|_2^2}. \]
Proof. Expand \(\mathbb E\|(G-b)Z\|^2=\mathbb E\{(G-b)^2\|Z\|^2\}\) as a quadratic function of \(b\). Differentiating and setting the derivative equal to zero gives the displayed ratio.
Reward-to-go
Removing past rewards from a score term lowers variance and keeps unbiasedness.
Proposition. In the likelihood-ratio gradient, the score at time \(t\) may be multiplied only by future return \(\sum_{k\ge t}\gamma^kR_{k+1}\); past reward terms have zero expectation with that score.
Proof. Past rewards are measurable with respect to the history before \(A_t\). Conditional on that history and \(S_t\), the score has mean zero by policy normalization. Therefore products with past rewards integrate to zero.
Control variates
Baselines are a special case of control variates.
Remark. A critic, generalized advantage estimation, and learned value baselines all try to approximate the conditional expectation structure that removes avoidable return noise while preserving the policy-gradient mean.
Why state baselines are unbiased
Lemma. For any integrable \(b(S)\), \[ E_{d^\pi}\{b(S)\nabla_\theta\log\pi_\theta(A\mid S)\}=0. \]
Proof. Conditional on \(S=s\), \[ E_\pi\nabla_\theta\log\pi_\theta(A\mid s) =\int\nabla_\theta\pi_\theta(a\mid s)\,\mathrm{d}a =\nabla_\theta1=0. \] Multiply by \(b(s)\) and average over states.
An action-dependent baseline is generally biased unless its expected score product vanishes or an analytic correction is added. This distinction matters for learned control variates.
The variance-minimizing scalar baseline
Proposition. Conditional on state \(s\), let \(Z=\nabla\log\pi(A\mid s)\) and let \(Y\) be an unbiased action-value target. Among scalar baselines \(b(s)\), the one minimizing \(E[\|Z\|^2(Y-b)^2\mid s]\) is \[ b^*(s) =\frac{E[\|Z\|^2Y\mid s]}{E[\|Z\|^2\mid s]}. \]
Proof. Differentiate the conditional quadratic in \(b\): \[ -2E[\|Z\|^2(Y-b)\mid s]=0. \] Strict convexity holds when \(E(\|Z\|^2\mid s)>0\).
The value baseline \(V^\pi(s)=E(Q^\pi\mid s)\) is optimal only when \(\|Z\|^2\) is conditionally constant or uncorrelated with \(Q^\pi\). It is nevertheless natural, stable, and often close to optimal.
Reward-to-go as Rao--Blackwellization
In a trajectory score estimator, multiplying the score at time \(t\) by rewards received before \(t\) adds zero mean noise because those rewards are measurable before action \(A_t\). Replacing total return by reward-to-go removes these causally irrelevant terms. This is a conditional-expectation variance reduction: it preserves the gradient while reducing second moment.
Generalized advantage estimation
For an approximate value \(V_\phi\), define \[ \delta_t^\phi=R_{t+1}+\gamma V_\phi(S_{t+1})-V_\phi(S_t), \qquad \widehat A_t^{\mathrm{GAE}(\lambda)} =\sum_{l\ge0}(\gamma\lambda)^l\delta_{t+l}^\phi. \] With exact \(V^\pi\), this is an unbiased conditional advantage estimator for \(\lambda=1\); smaller \(\lambda\) introduces bias when \(V_\phi\ne V^\pi\) but shortens the noise horizon. GAE is the policy-gradient analogue of TD(lambda), not a universally unbiased advantage estimate.
Further control variates
Action-dependent control variates can use reparameterization gradients, quadratic critics, or known conditional expectations. In off-policy settings, doubly robust scores combine a value model with importance weighting. The design principle is always the same: subtract a correlated mean-zero term whose expectation is known under the law that defines the gradient.