Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence

Summary

The paper develops model-free reinforcement learning algorithms for continuous-time stochastic control problems in which the drift coefficient and running reward are unknown while the diffusion coefficient and terminal reward are known. Working in the entropy-regularized exploratory framework of Wang–Zariphopoulou–Zhou, the authors avoid learning the value function directly (whose derivatives are numerically fragile) and instead learn a coupled pair: the spatial gradient of the value function, represented via Bismut–Elworthy–Li formulae along a control-independent reference diffusion, and the Gibbs exponent (essentially the q-function), estimated from one-step state responses and reward signals. For sufficiently small horizons they prove the discretized loss controls the error up to O(√Δt) and that ε-minimizers yield convergent value, gradient, and policy approximations (Theorems 3.9–3.10); general horizons are handled by backward induction over short segments, with constants that grow exponentially in T/δ (Remark 4.3). Numerically, SGD-trained networks achieve 1–10% relative errors up to dimension 100, roughly matching the model-based deep BSDE benchmark of Han–Jentzen–E at 2–3× its runtime below d=50, while a naive policy iteration diverges (Example 5.3). A restricted one-dimensional controlled-diffusion case is treated via a Hessian representation. Main limitations: an environment that returns trajectories from arbitrary off-trajectory states (Assumption 3.3), consistency rate only O(√Δt), and no approximation-error theory for the neural-network implementation.