Model-based reinforcement learning estimates or constructs a transition model and then plans in it. Its central question is not merely whether the model predicts observations well, but whether its errors remain small under the state-action distributions induced by planning.
Plug-in planning and simulation bounds
Let \(\widehat M=(\widehat r,\widehat P)\) approximate \(M=(r,P)\). If rewards lie in \([0,R_{\max}]\) and \[ \|\widehat r-r\|_\infty\le\varepsilon_r,\qquad \sup_{s,a}\|\widehat P(\cdot\mid s,a)-P(\cdot\mid s,a)\|_{\mathrm{TV}} \le\varepsilon_P, \] then every policy satisfies \[ \|V_M^\pi-V_{\widehat M}^\pi\|_\infty \le\frac{\varepsilon_r}{1-\gamma} +\frac{2\gamma R_{\max}\varepsilon_P}{(1-\gamma)^2}. \]
Theorem (plug-in policy loss). If \(\widehat\pi\) is optimal in \(\widehat M\), then \[ \|V_M^*-V_M^{\widehat\pi}\|_\infty \le \frac{2\varepsilon_r}{1-\gamma} +\frac{4\gamma R_{\max}\varepsilon_P}{(1-\gamma)^2}. \]
Proof. Let the uniform fixed-policy simulation bound be \(\Delta\). Then \[ V_M^*-V_M^{\widehat\pi} \le (V_M^{\pi^*}-V_{\widehat M}^{\pi^*}) +(V_{\widehat M}^{\pi^*}-V_{\widehat M}^{\widehat\pi}) +(V_{\widehat M}^{\widehat\pi}-V_M^{\widehat\pi}). \] The middle term is nonpositive by optimality of \(\widehat\pi\), and the other two are bounded by \(\Delta\).
Finite-sample model estimation
For a finite generative model, \(n(s,a)\) independent next-state samples give, with high probability, \[ \|\widehat P(\cdot\mid s,a)-P(\cdot\mid s,a)\|_1 =O\!\left(\sqrt{\frac{S+\log(SA/\delta)}{n(s,a)}}\right). \] Substitution into a uniform simulation lemma is valid but often pessimistic: it pays for all directions of transition error and two effective-horizon factors. Variance-aware analyses control \((\widehat P-P)V\) directly and can reach sharper minimax sample complexity.
Optimism and confidence sets
Rather than plan only in \(\widehat M\), optimistic methods construct a set \(\mathcal M_k\) containing the true MDP and select \[ (M_k,\pi_k)\in \arg\max_{M\in\mathcal M_k,\pi}J_M(\pi). \] On the confidence event, optimism converts regret into accumulated confidence widths along visited state-action pairs. Posterior sampling instead draws \(M_k\) from a posterior; conditionally under a correct Bayesian model, the sampled and true MDP have the same posterior law, yielding a posterior-sampling regret identity.
Planning with learned latent dynamics
In high-dimensional observations, a model may learn latent state \(Z_t=e(O_{\le t})\), transition \(p_\phi(z_{t+1}\mid z_t,a_t)\), and reward. Prediction loss alone does not ensure control sufficiency. A representation is value sufficient when rewards and transition expectations of all relevant value functions are preserved. Bisimulation metrics formalize this: \[ d(s,t) =\max_a\left\{ |r(s,a)-r(t,a)| +\gamma W_d(P(\cdot\mid s,a),P(\cdot\mid t,a)) \right\}. \] States at small bisimulation distance have close optimal values.
Compounding error and model-predictive control
Open-loop rollout error compounds with horizon because a model is evaluated on states generated by itself. Model-predictive control repeatedly replans over a shorter horizon using the newest observation, reducing but not eliminating distribution shift. Ensembles, uncertainty penalties, and short synthetic rollouts aim to prevent planning from exploiting model errors. Their validity depends on calibrated uncertainty under the planner's induced distribution.
Identifiability versus usefulness
A transition model need not identify latent physical mechanisms to support optimal prediction or control; observationally equivalent models can have the same value for all policies in the data-supported class. Conversely, a model with excellent one-step likelihood can be useless for control if small errors occur in rare but decision-critical transitions. Evaluation must therefore include policy value, calibration, and coverage, not prediction loss alone.