Differential entropy is useful but delicate: unlike discrete entropy, it is not invariant under smooth changes of coordinates. Mutual information and relative entropy remain invariant quantities.
Differential entropy
Definition. If \(X\) has density \(f\) on \(\mathbb R^d\), its differential entropy is \[ h(X)=-\int f(x)\log f(x)\,\mathrm{d}x, \] when the integral is well-defined.
Change of variables
Proposition. If \(Y=g(X)\) for a smooth bijection \(g\), then \[ h(Y)=h(X)+\mathbb E\log |\det Dg(X)|. \] Thus differential entropy depends on the coordinate scale.
Invariant information
Theorem. Relative entropy and mutual information are invariant under one-to-one measurable transformations. In particular, \(I(X;Y)\) is the robust continuous-alphabet analogue of dependence information.
Proof by Jacobian cancellation
Proof. Under a smooth bijection, densities acquire reciprocal Jacobian factors. In ratios such as \(f/g\), or \(f_{XY}/(f_Xf_Y)\), these factors cancel. Differential entropy alone has no ratio and keeps the Jacobian term.
Differential entropy
Definition. If \(X\) has density \(f\) on \(\mathbb R^d\) and \(f\log f\) is integrable, \[ h(X)=-\int_{\mathbb R^d}f(x)\log f(x)\,\mathrm{d}x. \]
Unlike discrete entropy, differential entropy can be negative and depends on coordinates. Relative entropy and mutual information retain invariant operational meaning.
Change of variables
Theorem. If \(Y=g(X)\) for a differentiable bijection with nonsingular Jacobian, then \[ h(Y)=h(X)+ \mathbb E\log|\det J_g(X)|. \]
Proof. The transformed density is \(f_Y(g(x))=f_X(x)/|\det J_g(x)|\). Substitute into \(-\mathbb E\log f_Y(Y)\).
Corollary. For invertible matrix \(A\) and vector \(b\), \[ h(AX+b)=h(X)+\log|\det A|. \] In one dimension, \(h(aX+b)=h(X)+\log|a|\).
Negative entropy
Example. If \(X\sim\operatorname{Unif}[0,a]\), then \[ h(X)=\log a. \] This is negative when \(a<1\), showing why differential entropy is not an absolute count of possibilities.
Quantization limit
Theorem. Let \(X_\Delta\) be uniform scalar quantization of a sufficiently regular density with bin width \(\Delta\). Then \[ H(X_\Delta)+\log\Delta\to h(X) \quad\text{as }\Delta\downarrow0. \]
Proof outline. Bin probabilities satisfy \(p_k\approx f(x_k)\Delta\). Therefore \[ -\sum_kp_k\log p_k \approx-\sum_kf(x_k)\Delta\log f(x_k)-\log\Delta, \] and the first sum converges to the entropy integral.
Relative entropy is coordinate invariant
Proposition. For a measurable bijection \(g\), \[ D(P_X\|Q_X)=D(P_{g(X)}\|Q_{g(X)}). \]
Proof. Both transformed densities acquire the same inverse-Jacobian factor, which cancels in their likelihood ratio. Integrate after change of variables.
Mutual information for continuous variables
Theorem. \[ I(X;Y)=D(P_{XY}\|P_XP_Y) =h(X)+h(Y)-h(X,Y) \] when all displayed differential entropies exist without indeterminate \(\infty-\infty\). The KL definition remains valid more generally.
If \(Y=X\) for a non-atomic continuous variable, then \(I(X;Y)=+\infty\): the joint law lies on the diagonal and is singular relative to \(P_XP_Y\). Yet \(h(X)\) may be finite. This is another reason to define information through KL first.
Conditional differential entropy
Definition. \[ h(X\mid Y)= -\int p(x,y)\log p(x\mid y)\, \mathrm{d}x\,\mathrm{d}y, \] when integrable. Unlike the discrete case, \(h(X\mid Y)\) can be negative, but \[ I(X;Y)=h(X)-h(X\mid Y)\ge0. \]
Gaussian entropy
Proposition. If \(X\sim N_d(\mu,\Sigma)\) with \(\Sigma\) positive definite, \[ h(X)=\frac12\log\{(2\pi e)^d\det\Sigma\}. \]
Proof. \[ -\mathbb E\log f(X) =\frac d2\log(2\pi)+\frac12\log\det\Sigma+ \frac12\mathbb E[(X-\mu)^\top\Sigma^{-1}(X-\mu)]. \] The quadratic expectation is \(d\).
Entropy under support constraints
Theorem. Among densities supported on a measurable set \(A\) of finite volume, the uniform density maximizes differential entropy, with maximum \(\log\operatorname{vol}(A)\).
Proof. Let \(U_A\) be uniform on \(A\). Then \[ D(P\|U_A)=\log\operatorname{vol}(A)-h(P)\ge0. \]
Measure-theoretic caution
Densities depend on a reference measure. The invariant object is relative entropy \(\int\log(\mathrm{d}P/\mathrm{d}Q)\,\mathrm{d}P\). Differential entropy is the negative KL integrand relative to Lebesgue measure without a normalizable Lebesgue probability, so it inherits coordinate dependence.