Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories.
Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization.
We introduce Function-Space Policy Optimization (FSPO)
a single-policy framework that combines current privileged adaptation with function-space temporal consolidation.
FSPO stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model.
It further introduces privileged disagreement correction to explicitly suppress locally overvalued actions, reliability-adaptive grounding that strengthens reference supervision when self-generated evidence repeatedly fails, and quality and coverage-aware temporal memory that retains verified behaviors and improves privileged demonstrations.
Results
Across reasoning and tool-use benchmarks, FSPO achieves the highest average accuracy under a 5-hour training budget.
improving over the strongest baseline by approximately 4% points on average
while reducing peak memory from about 70 GB to 56 GB on Olmo3-7B-Instruct.