Actor-critic methods combine policy optimization with value estimation. The actor updates the policy; the critic estimates value or advantage information used by the actor.
Actor and critic
The two time scales correspond to two coupled stochastic approximation problems.
Definition. An actor-critic method maintains policy parameters \(\theta\) and critic parameters \(w\). A typical update is \[ w_{t+1}=w_t+\beta_t\,\widehat{\Delta}_w, \qquad \theta_{t+1}=\theta_t+\alpha_t\,\widehat{\Delta}_\theta, \] where the critic estimates \(v_{\pi_\theta}\), \(q_{\pi_\theta}\), or \(A_{\pi_\theta}\), and the actor uses this estimate in a policy-gradient step.
Remark. The critic is not an objective by itself; it supplies a lower-variance approximate gradient direction for the actor.
TD critic for policy gradient
A one-step TD error can estimate an advantage under the current policy.
Proposition. If \(v=v_\pi\), then \[ \mathbb E\{R_{t+1}+\gamma v(S_{t+1})-v(S_t)\mid S_t=s,A_t=a\}=A_\pi(s,a). \]
Proof. The conditional expectation of \(R_{t+1}+\gamma v_\pi(S_{t+1})\) given \((s,a)\) is \(q_\pi(s,a)\). Subtracting \(v_\pi(s)\) gives \(A_\pi(s,a)\).
Two-time-scale convergence
Classical actor-critic theory often assumes the critic equilibrates faster than the actor.
Theorem. Under standard smoothness, ergodicity, boundedness, and stepsize conditions with \(\alpha_t/\beta_t\to0\), two-time-scale actor-critic tracks an ODE in which the critic is near its fixed point for the current actor.
Proof. The critic recursion evolves on the faster time scale \(\beta_t\) and converges to the stable solution of its projected Bellman equation for quasi-static \(\theta\). The actor recursion sees the critic error as asymptotically negligible and follows the mean policy-gradient ODE. This is the standard two-time-scale stochastic approximation argument.
Bias from approximate critics
A critic can reduce variance but introduce bias if it estimates the wrong quantity.
Remark. Compatible function approximation and advantage centering are designed to make the critic's approximation error orthogonal to policy-score directions, thereby reducing gradient bias.
Coupled recursions
An actor--critic algorithm estimates a value object \(w\) while updating policy parameter \(\theta\): \[ w_{t+1}=w_t+\beta_t H(w_t,\theta_t,Z_{t+1}), \] \[ \theta_{t+1} =\theta_t+\alpha_t \widehat A_t(w_t)\nabla_\theta\log\pi_{\theta_t}(A_t\mid S_t). \] The critic turns delayed returns into a lower-variance approximate advantage; its error can bias the actor.
Two-time-scale principle
Theorem (ODE limit, schematic). Suppose the state-action chain is uniformly ergodic over a compact parameter set, mean fields are Lipschitz, noise has controlled second moments, \[ \sum_t\alpha_t=\sum_t\beta_t=\infty,\quad \sum_t(\alpha_t^2+\beta_t^2)<\infty,\quad \frac{\alpha_t}{\beta_t}\to0, \] and for each fixed \(\theta\), the critic ODE \(\dot w=\bar H(w,\theta)\) has a globally stable equilibrium \(w^*(\theta)\). Then the critic tracks \(w^*(\theta_t)\), while actor limit points follow \[ \dot\theta=\bar G(\theta,w^*(\theta)) \] after projection to the parameter set.
This theorem is an application of two-time-scale stochastic approximation. The condition \(\alpha_t/\beta_t\to0\) means the critic sees the actor as nearly fixed, while the actor sees the critic as nearly equilibrated. It is not enough that both step sizes merely decrease.
When the actor direction is exact
Proposition (compatible critic). Let \(\phi_\theta=\nabla_\theta\log\pi_\theta\), and choose \(w\) to minimize \[ E_{d^\pi}\{A^\pi(S,A)-w^\top\phi_\theta(S,A)\}^2. \] Then \[ E_{d^\pi}\{\phi_\theta\,w^\top\phi_\theta\} =E_{d^\pi}(\phi_\theta A^\pi) =(1-\gamma)\nabla J(\theta). \] Thus replacing advantage by the fitted compatible critic preserves the exact policy gradient.
An arbitrary value network does not satisfy this orthogonality identity. Approximation error may introduce a nonvanishing actor bias even with infinite data.
Bias from bootstrapping and shared data
Actor and critic use the same evolving Markov trajectory. Valid analysis must control temporal dependence, nonstationarity as \(\theta_t\) changes, bootstrapping bias, and estimation error in \(w_t\). Experience replay changes the sampling distribution and usually makes updates off-policy; importance weights or emphatic corrections may be required for the theorem targeted by the algorithm.
Average-reward actor--critic
For an ergodic continuing task with average reward \(\bar r(\theta)\), the differential value solves a Poisson equation \[ h^\pi=r_\pi-\bar r(\pi)\mathbf1+P_\pi h^\pi, \] with a normalization such as \(d_\pi^\top h^\pi=0\). Policy gradients use the differential action value and stationary occupancy. This is not obtained by substituting \(\gamma=1\) into discounted formulas because values are defined only up to constants and mixing controls the effective horizon.