Diffusion-based policies have recently emerged as a powerful paradigm for offline reinforcement learning, but their reliance on multi-step denoising introduces substantial computational overhead and requires backpropagation through time during training, limiting scalability and practical deployment.
In this work, we propose Instant Flow Q-Learning (InsQL), a single-step generative policy learning framework that retains the expressive capacity of diffusion models while eliminating iterative sampling and BPTT.
InsQL builds on the flow-matching perspective of diffusion policies, but replaces marginal velocity parameterization with average-velocity modeling and introduces a principled compositional mechanism, yielding a theoretically grounded objective that learns average velocity consistently across scales.
This formulation provides a direct noise-to-action mapping and enables single-step action generation.
InsQL further unifies compositional average-velocity modeling with behavior-regularized actor--critic optimization in a single framework, requiring only a single policy network and avoiding distillation, multi-stage procedures, and Jacobian computations during training.
At inference time, actions are generated in a single forward pass and can be refined via value-guided candidate selection to improve performance.
Extensive experiments on standard offline RL benchmarks and real-world robotic manipulation tasks show that InsQL matches or surpasses state-of-the-art diffusion-based methods while substantially reducing both training and inference time.
Overall, these results establish compositional average-velocity modeling as an efficient and scalable alternative to multi-step generative policies for offline RL.