Policy-gradient methods optimize a differentiable policy directly. The policy-gradient theorem expresses the gradient without differentiating the transition law.
Objective and score identity
The log-derivative trick moves differentiation from the trajectory density to the policy score.
Definition. For a differentiable policy \(\pi_\theta\), define \[ J(\theta)=\mathbb E_{\mu,\pi_\theta}\sum_{t=0}^\infty\gamma^tR_{t+1}. \] The score identity is \(\nabla_\theta p_\theta(x)=p_\theta(x)\nabla_\theta\log p_\theta(x)\) whenever \(p_\theta(x)>0\).
Lemma. For any trajectory functional \(F(\tau)\), \[ \nabla_\theta\mathbb E_\theta F(\tau) =\mathbb E_\theta\{F(\tau)\nabla_\theta\log p_\theta(\tau)\}, \] provided differentiation and integration may be interchanged.
Proof. Write the expectation as \(\int F(\tau)p_\theta(\tau)\mathrm{d}\tau\), differentiate under the integral, and substitute \(\nabla p_\theta=p_\theta\nabla\log p_\theta\).
Policy-gradient theorem
The gradient weights policy scores by action values under the discounted occupancy measure.
Theorem. For a discounted MDP with differentiable policy, \[ \nabla J(\theta)=\frac{1}{1-\gamma} \int d_{\mu,\pi_\theta}(\mathrm{d}s)\int \pi_\theta(\mathrm{d}a\mid s) \nabla\log\pi_\theta(a\mid s)q_{\pi_\theta}(s,a). \]
Proof. The trajectory density factors into the initial law, transition kernels, and policy probabilities. Transition kernels do not depend on \(\theta\), so \[ \nabla\log p_\theta(\tau)=\sum_{t\ge0}\nabla\log\pi_\theta(A_t\mid S_t). \] Apply the score identity to the discounted return. Terms in which a score at time \(t\) is multiplied by rewards before time \(t\) vanish after conditioning on the past, since the conditional expectation of the score is zero. The remaining future rewards give \(q_{\pi_\theta}(S_t,A_t)\). Summing discounted state-action distributions yields the occupancy-measure form.
Compatible baselines
The theorem is unchanged when a state-dependent baseline is subtracted.
Proposition. For any bounded \(b(s)\), \[ \int \pi_\theta(\mathrm{d}a\mid s)\nabla\log\pi_\theta(a\mid s)b(s)=0. \]
Proof. The integrand equals \(b(s)\int\nabla\pi_\theta(\mathrm{d}a\mid s)\). Since \(\int\pi_\theta(\mathrm{d}a\mid s)=1\), differentiating the normalization gives zero.
Advantage form
Subtracting \(v_\pi(s)\) gives the advantage, often reducing variance.
Definition. The advantage function is \[ A_\pi(s,a)=q_\pi(s,a)-v_\pi(s). \] The policy gradient can use \(A_\pi\) in place of \(q_\pi\) because \(v_\pi(s)\) is a valid baseline.
Discounted objective and occupancy derivative
Let \(\pi_\theta(a\mid s)\) be differentiable with parameter-independent dynamics, and define \[ J(\theta)=E_\rho^{\pi_\theta} \sum_{t\ge0}\gamma^tr(S_t,A_t). \] Assume bounded rewards, differentiability under the path integral, and support that does not change with \(\theta\).
Theorem (policy gradient). \[ \nabla_\theta J(\theta) =\frac1{1-\gamma} E_{(S,A)\sim d_\rho^{\pi_\theta}} \left[ Q^{\pi_\theta}(S,A) \nabla_\theta\log\pi_\theta(A\mid S) \right]. \]
Proof. For a finite trajectory, the likelihood-ratio identity gives \[ \nabla J =E\left[ \sum_{t\ge0}\gamma^tr_t \sum_{k=0}^t\nabla\log\pi_\theta(A_k\mid S_k) \right]. \] Interchange sums. Rewards before time \(k\) have conditional expectation independent of \(A_k\), and their score contribution vanishes. The remaining return from \(k\) has conditional mean \(Q^\pi(S_k,A_k)\). Thus \[ \nabla J =\sum_{k\ge0}\gamma^k E[Q^\pi(S_k,A_k)\nabla\log\pi(A_k\mid S_k)], \] which is the occupancy form.
Corollary (advantage form). For any state-only baseline \(b(s)\), \[ \nabla J =\frac1{1-\gamma}E_{d^\pi} \{[Q^\pi(S,A)-b(S)]\nabla\log\pi(A\mid S)\}. \] Taking \(b=V^\pi\) gives the advantage form.
Compatible approximation and natural gradient
Let the policy score be \(\phi_\theta(s,a)=\nabla_\theta\log\pi_\theta(a\mid s)\), and define the discounted Fisher matrix \[ F(\theta)=E_{d^\pi}\{\phi_\theta\phi_\theta^\top\}. \] If \(w\) is the least-squares projection of \(A^\pi\) onto \(w^\top\phi_\theta\), its normal equations are \[ Fw=E_{d^\pi}(\phi_\theta A^\pi) =(1-\gamma)\nabla J. \] Hence \(w=(1-\gamma)F^{-1}\nabla J\): compatible advantage regression directly estimates the natural-gradient direction.
Deterministic policy gradient
Theorem. For a differentiable deterministic policy \(a=\mu_\theta(s)\), under regularity, \[ \nabla_\theta J(\theta) =\frac1{1-\gamma} E_{S\sim d_{\rho,S}^{\mu_\theta}} \left[ \nabla_\theta\mu_\theta(S) \nabla_aQ^{\mu_\theta}(S,a)|_{a=\mu_\theta(S)} \right]. \]
This is obtained by differentiating the Bellman equation and unrolling the resulting resolvent. It avoids action-space likelihood ratios but requires a differentiable critic and can be sensitive to critic extrapolation outside the behavior distribution.
Optimization qualifications
The theorem provides an exact gradient, not global concavity. Policy parametrization creates stationary points, flat directions, and boundary effects. Trust regions and natural gradients control local distributional change; they do not remove nonconvexity. Convergence claims must distinguish exact gradients, Monte Carlo estimates, actor--critic bias, and function approximation.