Distillation of reasoning chains from Large Language Models (LLMs) into their smaller siblings is a standard training approach.
Frequently, researchers do not have full access to a teacher model and its internal representations, which limits the resulting performance.
We propose a recursive offline distillation method combining two components: entropy-based sampling for selective distillation and answer mixing.
Specifically, for three open models (3-4B), at each epoch, we sample reasoning chains weighted by entropy difference with an auxiliary medium model (70-72B) and, next, supplement thinking traces with single-token ground truth answers.
Our approach consistently outperforms naive distillation with comparable data budgets ($57.15 \pm 1.01$ vs 51.61 on average) and offers a much steeper learning curve.
Additionally, we present a comparative analysis of methods to generate and normalize distillation traces.