Optimal Value Inference for Reinforcement Learning

Summary

The paper studies offline statistical inference for the optimal average reward J* of a finite state-action Markov decision process, using trajectories collected under a possibly time-varying behavior policy. Its core move is to avoid post-hoc softmax smoothing of an estimated Q*: the authors instead replace the maximization inside the Bellman optimality recursion by a softmax, obtaining a self-induced Bellman equation whose fixed point (Q_beta, J_beta) serves as both smoothing target and key nuisance. They build a Neyman-orthogonal score around a normalized adjoint weight lambda_beta, handling the singular linearization induced by value anchoring, and prove sqrt(NH)-rate asymptotic normality for fixed and diverging horizons with time-varying behavior policies via a triangular-array martingale central limit theorem. Nuisances are estimated by a regularized minimax procedure (finite-sample rate C*sqrt((|S||A| log n + log(1/delta))/n) for the Bellman pair) and a plug-in ERM for the adjoint weight; only second-order product rates are required, so no consistent optimal-policy estimator is needed. Semiparametric efficiency is established for the smoothed target at fixed horizon, for J* under a unique optimum, and at ties the limit matches the efficient influence function of the uniform policy over optimal actions. The bias condition sqrt(NH)e^{-beta*Gamma}->0 involves the unknown optimality gap Gamma, and a practical joint choice of beta with the nuisance rates is not spelled out in the provided text. The experimental section (synthetic data, bike repositioning, LLM agentic tool use) is not part of the excerpt.