首页 > AI前沿 > Beyond Distribution Matching: Selective Margin Preservation for Supervised Fine-Tuning

Beyond Distribution Matching: Selective Margin Preservation for Supervised Fine-Tuning

arXiv自然语言 2026-10-14 15:09 4 阅读 查看原文

Supervised fine-tuning (SFT) changes logit margins among non-target tokens at teacher-forced prefixes.

Matching a local reference distribution keeps those probabilities and also penalizes margin expansions.

We ask whether a soft auxiliary can preserve selected local non-target margin relations without freezing that distribution.

We introduce Selective Rank-Preserving SFT (SRP-SFT), a soft margin target: a hinge on reference-scaled lower bounds for pairs from the reference's top-$K$ non-target tokens, applied at fixed teacher-forced prefixes.

The hinge stops once the requested bound is met.

We derive the KL cost of violating a margin, characterize the feasible region and its conditional utility, and give a finite-step condition for reducing aggregate violations.

In one-epoch experiments on Qwen3-4B-Base and UltraFeedback, this target raises MATH-500 and code-generation scores over CE and LP-SFT at all three training seeds.

At seed 1234, MATH-500 pass@1 rises from 49.56% to 54.56% and the four-metric code mean rises from 65.91% to 69.50% relative to LP-SFT.

Ranking controls, held-out cache diagnostics, sensitivity sweeps, and six additional backbones show where these score changes appear.