Information geometry begins by treating a statistical model as a smooth space of probability laws. Parameters are coordinates; the intrinsic object is the family of distributions and the tensors that survive reparametrization.
Regular statistical models
A statistical manifold is a model with enough smoothness to differentiate probability laws and compare nearby distributions.
Definition. Let \((\mathcal X,\mathcal A,\mu)\) be a measured space. A \(d\)-dimensional regular statistical model is a family \[ \mathcal P=\{p_\theta:\theta\in\Theta\subset\mathbb R^d\} \] of positive densities such that \(\Theta\) is open, \(\theta\mapsto p_\theta(x)\) is smooth, and differentiation under \(\int p_\theta\,\mathrm{d}\mu=1\) is justified.
Remark. The positivity assumption avoids boundary singularities and lets every model point have the same support. Many important models with boundaries need separate treatment.
Charts and identifiability
The parameter is a chart only when different parameter values give different distributions.
Proposition. If \(\theta\mapsto p_\theta\) is injective and has full-rank differential, then \(\Theta\) is a coordinate chart for the image \(\mathcal P\). If \(\phi=\phi(\theta)\) is a smooth reparametrization, the same distribution is described by \(p_{\theta(\phi)}\).
Proof. Injectivity identifies each point of \(\mathcal P\) with one parameter value. Full rank gives a smooth immersed image, and a smooth inverse on the image gives a chart. A reparametrization composes the chart with a diffeomorphism, so it changes coordinates without changing the point \(p\in\mathcal P\).
Curves and tangent vectors
Tangent vectors are infinitesimal changes of distributions, not merely arrows in parameter space.
Definition. A smooth curve \(t\mapsto p_{\theta(t)}\) through \(p_\theta\) has tangent represented by \[ \dot p_0=\dot\theta^i\partial_i p_\theta, \qquad \dot\ell_0=\dot\theta^i\partial_i\log p_\theta. \] The second representation is the score representation of the tangent vector.
Proposition. Every tangent score \(u=\dot\theta^i\partial_i\ell_\theta\) satisfies \(\mathbb E_\theta u=0\).
Proof. Differentiate the normalization identity along the curve: \[ 0=\frac{\mathrm{d}}{\mathrm{d}t}\int p_{\theta(t)}\,\mathrm{d}\mu\bigg|_{t=0} =\int \dot p_0\,\mathrm{d}\mu =\int \dot\ell_0\,p_\theta\,\mathrm{d}\mu. \]
Intrinsic viewpoint
A construction is geometric only if it transforms tensorially under changes of chart.
Remark. This principle is why information geometry emphasizes the Fisher metric, affine connections, duality, and divergences rather than formulas tied to one convenient parametrization.
Dominated models and regular parametrizations
Let \((\mathcal X,\mathcal A,\mu)\) be a measure space and let \(\mathcal P=\{P_\theta:\theta\in\Theta\}\) be dominated by \(\mu\), with densities \(p_\theta\). A finite-dimensional manifold structure is not supplied merely by writing a parameter \(\theta\): the map \(\theta\mapsto P_\theta\) must be locally injective and sufficiently regular in a topology on probability measures.
Definition (regular local model). A \(d\)-dimensional parametrization at \(\theta_0\in\operatorname{int}\Theta\) is quadratic-mean differentiable if there is \(\dot\ell_{\theta_0}\in L_2(P_{\theta_0})^d\) such that \[ \int\left[ \sqrt{p_{\theta_0+h}}-\sqrt{p_{\theta_0}} -\frac12h^\top\dot\ell_{\theta_0}\sqrt{p_{\theta_0}} \right]^2\,\mathrm{d}\mu=o(\|h\|^2). \] It is regular when the coordinates are locally identifiable and \(I_{\theta_0}=P_{\theta_0}(\dot\ell_{\theta_0}\dot\ell_{\theta_0}^{\top})\) is nonsingular.
Proposition. Quadratic-mean differentiability forces \(P_{\theta_0}\dot\ell_{\theta_0}=0\), and the score is unique as an element of \(L_2(P_{\theta_0})^d\).
Proof. Write \(s_\theta=\sqrt{p_\theta}\). Because \(\|s_{\theta_0+h}\|_2^2=\|s_{\theta_0}\|_2^2=1\), expansion of the squared norm gives \[ 0=h^\top\int \dot\ell_{\theta_0}p_{\theta_0}\,\mathrm{d}\mu+o(\|h\|). \] Taking \(h=te_j\) and letting \(t\to0\) proves centering. If \(a\) and \(b\) both satisfy the quadratic-mean expansion, then \(\|h^\top(a-b)s_{\theta_0}\|_2=o(\|h\|)\). Again use \(h=te_j\) to obtain \(a_j=b_j\) in \(L_2(P_{\theta_0})\) for every \(j\).
Curves, velocities, and equivalent representations
A smooth curve \(t\mapsto P_t\) through \(P=P_0\) has velocity represented by its score \(g\in L_2^0(P)\), where \[ \sqrt{p_t}=\sqrt p+\frac t2g\sqrt p+o(t)\quad\text{in }L_2(\mu). \] Two parametrized curves represent the same tangent vector precisely when their scores agree \(P\)-almost surely. This quotient is essential: velocities of density coordinates are not intrinsic, whereas scores are.
Example (a smooth submodel). If \(P\) has density \(p\) and \(g\in L_\infty(P)\) satisfies \(Pg=0\), then \[ p_t(x)=\frac{e^{tg(x)}p(x)}{P e^{tg}} \] defines a curve with score \(g\). Truncation and approximation show that bounded scores are dense in \(L_2^0(P)\); hence the nonparametric tangent space of an unrestricted dominated model is \(L_2^0(P)\).
Immersions, embeddings, and singular models
The density map may be viewed through the square-root embedding \[ \iota:P\longmapsto 2\sqrt p\in L_2(\mu). \] Its image lies on the sphere of radius \(2\), and \(\mathrm D\iota_P[g]=g\sqrt p\). Pulling back the ambient \(L_2\) inner product gives the Fisher metric. Thus a regular statistical model is an immersed submanifold of a Hilbert sphere; it is embedded only when its topology agrees with the subspace topology and the parametrization has no self-identifications.
Counterexample (mixture singularity). For \[ p_{\alpha,\mu_1,\mu_2} =\alpha\phi(\,\cdot-\mu_1)+(1-\alpha)\phi(\,\cdot-\mu_2), \] the points with \(\mu_1=\mu_2\) are not regular: changing \(\alpha\) leaves the density unchanged and the Fisher information loses rank. Label switching also identifies \((\alpha,\mu_1,\mu_2)\) with \((1-\alpha,\mu_2,\mu_1)\). The resulting space is stratified rather than a single smooth manifold. Ordinary differential-geometric arguments must therefore be restricted to regular strata.
The local manifold picture used throughout this note assumes regularity. Its statistical asymptotic meaning is developed in local asymptotic normality, while infinite-dimensional tangent constructions are treated in the semiparametric pages of statistical inference.