Semi-gradient temporal-difference learning is a core algorithmic component of reinforcement learning, but bootstrapping with function approximation generally induces a non-conservative vector field and can lead to instability.
We analyze this effect for two-layer neural networks in the mean-field regime and identify a simple stability mechanism in which entropy-induced relaxation dominates the sensitivity of the moving bootstrap target.
Under this condition, we prove exponential convergence to a unique fixed point and quantify its bias relative to the unregularized Bellman solution.
We also show that the same mechanism extends to entropy-regularized off-policy control through soft Q-learning.