Reasoning post-training draws on offline demonstrations and policy-generated rollouts, yet conventional supervised fine-tuning (SFT) followed by reinforcement learning (RL) separates their objectives and limits continued reuse of collected responses.
We propose Cross-Behavior Policy Optimization (CBPO), which makes these sources compatible with one regression loss through a fixed-reference target. Because the generating policy does not enter this target, positive and negative offline completions, reference responses, and current and historical rollouts can share a common reward-labeled record format and replay buffer.
As new responses enter the buffer, optimization retains the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic.
Restricting the available sources gives offline, on-policy, and off-policy variants, while CBPO-Unified combines them.
Performance Results
Across two model scales and four mathematical reasoning benchmarks, CBPO delivers leading avg@32 and pass@32 performance across benchmarks and data regimes.
On Qwen3-1.7B, CBPO-Unified improves benchmark-averaged avg@32 and pass@32 over SFT DAPO by 2.33 and 4.44 percentage points, respectively.
In the replay ablation, retaining historical rollouts improves these metrics by 1.87 and 3.72 percentage points over the no-history configuration.