Composed Supervision for Self-Improvement
Although self-improvement is becoming increasingly important as tasks become more complex, models struggle to generate reliable supervision for harder, out-of-distribution instances.
Exploiting Compositional Structure
One way to address this challenge is to exploit compositional structure: models often solve simpler subproblems more reliably than full problems, so their subproblem predictions can be composed into supervision for harder ones.
Systematic Study and Failure Modes
We systematically study when this kind of composed supervision supports self-improvement and when it fails. Our controlled experiments identify three failure modes:
- unreliable subproblem predictions
- errors introduced by composition
- limited supervision coverage
Results on Realistic Tasks
Beyond controlled settings, these findings carry over to more realistic coding and reasoning tasks: composed labels are more accurate than direct self-generated labels and improve performance even on compositions longer than those seen during training, while the failure modes remain visible.
Combining with Reinforcement Learning
Further, although these failure modes often keep compositional self-improvement from fully succeeding on its own, a short SFT warmup with compositional self-improvement can bootstrap reinforcement learning on tasks that require composing skills the model already has, particularly when direct exploration rarely succeeds.
Conclusion
Together, these findings characterize when compositional self-improvement can provide sufficiently reliable supervision to expand model capabilities.