Entropy regularization changes hard maximization into soft maximization. It encourages exploration, smooths policy improvement, and connects reinforcement learning with convex duality.
Regularized objective
The policy is rewarded for both external reward and entropy.
Definition. For temperature \(\alpha>0\), the entropy-regularized return includes \[ \sum_{t\ge0}\gamma^t\{R_{t+1}+\alpha H(\pi(\cdot\mid S_t))\}, \] where \(H(\pi(\cdot\mid s))=-\int\pi(\mathrm{d}a\mid s)\log\pi(a\mid s)\) in the discrete or density case.
Remark. Larger \(\alpha\) favors more random policies; as \(\alpha\downarrow0\), the ordinary control objective is recovered.
Soft Bellman operator
The maximum is replaced by log-sum-exp or its continuous analogue.
Definition. For finite actions, the soft optimality operator is \[ (T_\alpha v)(s)=\alpha\log\sum_a\exp\left(\frac{r(s,a)+\gamma\sum_{s'}P(s'\mid s,a)v(s')}{\alpha}\right). \]
Proposition. The soft Bellman operator is a \(\gamma\)-contraction in sup norm.
Proof. The log-sum-exp map is 1-Lipschitz with respect to the sup norm: shifting all action scores by at most \(\varepsilon\) shifts the log-sum-exp by at most \(\varepsilon\). The only dependence on \(v\) is through \(\gamma Pv\), whose sup-norm change is at most \(\gamma\|u-v\|_\infty\).
Soft greedy policy
The soft maximizer is a Gibbs distribution over action values.
Theorem. For action scores \(q(s,a)\), the solution of \[ \max_{\pi(\cdot\mid s)}\left\{\sum_a\pi(a\mid s)q(s,a)+\alpha H(\pi(\cdot\mid s))\right\} \] is \[ \pi_\alpha(a\mid s)=\frac{\exp(q(s,a)/\alpha)}{\sum_b\exp(q(s,b)/\alpha)}. \]
Proof. Use a Lagrange multiplier for the constraint \(\sum_a\pi(a\mid s)=1\). The first-order condition is \(q(s,a)-\alpha(1+\log\pi(a\mid s))-\lambda=0\), hence \(\pi(a\mid s)\propto e^{q(s,a)/\alpha}\).
KL control
Entropy regularization is equivalent to KL regularization relative to a reference policy in many formulations.
Remark. Replacing entropy by \(-\operatorname{KL}(\pi(\cdot\mid s)\|\rho(\cdot\mid s))\) gives soft policies tilted from the reference \(\rho\). This is the form used in trust-region and mirror-descent views of policy optimization.
Convex duality behind the soft maximum
Theorem (Gibbs variational formula). For a finite action set, reference distribution \(\mu(\cdot\mid s)>0\), and \(\tau>0\), \[ \tau\log\sum_a\mu(a\mid s)e^{q(a)/\tau} =\max_{\pi(\cdot\mid s)} \left\{\sum_a\pi(a\mid s)q(a) -\tau D_{\mathrm{KL}}(\pi\|\mu)\right\}. \] The unique maximizer is \[ \pi_q(a\mid s) =\frac{\mu(a\mid s)e^{q(a)/\tau}} {\sum_b\mu(b\mid s)e^{q(b)/\tau}}. \]
Proof. Let \(Z=\sum_a\mu_ae^{q_a/\tau}\) and \(\pi_q=\mu e^{q/\tau}/Z\). Direct rearrangement gives \[ \sum_a\pi_aq_a-\tau D(\pi\|\mu) =\tau\log Z-\tau D(\pi\|\pi_q). \] Nonnegativity of KL proves both claims.
Soft Bellman equations
The KL-regularized optimality operator is \[ (T_\tau V)(s) =\tau\log\int \exp\!\left\{\frac{r(s,a)+\gamma P_aV(s)}{\tau}\right\} \mu(\mathrm{d}a\mid s). \]
Proposition. \(T_\tau\) is monotone, translation equivariant, \[ T_\tau(V+c)=T_\tau V+\gamma c, \] and a \(\gamma\)-contraction in sup norm. It therefore has a unique soft value \(V_\tau^*\).
Proof. If \(\|V-W\|_\infty\le c\), every action score differs by at most \(\gamma c\). Monotonicity and translation equivariance of log-sum-exp place \(T_\tau V-T_\tau W\) between \(-\gamma c\) and \(\gamma c\).
Approximation to ordinary control
For uniform \(\mu\) on \(m\) actions, \[ \max_aq_a-\tau\log m \le\tau\log\left(\frac1m\sum_ae^{q_a/\tau}\right) \le\max_aq_a. \] Consequently, \[ \|V^*-V_\tau^*\|_\infty \le\frac{\tau\log m}{1-\gamma}. \] As \(\tau\downarrow0\), the soft policy concentrates on greedy actions; as \(\tau\) grows, it remains close to the reference policy.
Maximum entropy and KL control
With a uniform reference, subtracting \(\tau D(\pi\|\mu)\) is equivalent up to a constant to adding policy entropy. A nonuniform \(\mu\) encodes a prior controller, producing relative-entropy control. State-dependent temperatures and constraints lead to convex dual multipliers rather than an arbitrary exploration bonus.
Soft actor--critic objective
Soft actor--critic uses a critic for \[ Q^\pi_\tau(s,a) =r(s,a)+\gamma E[V^\pi_\tau(S')\mid s,a], \] \[ V^\pi_\tau(s) =E_{A\sim\pi}\{Q^\pi_\tau(s,A) -\tau\log[\pi(A\mid s)/\mu(A\mid s)]\}. \] Policy improvement minimizes KL to the Gibbs law induced by \(Q\). Function approximation and replay make practical SAC approximate; the exact tabular monotonicity theorem requires accurate evaluation and unrestricted policy updates.