LoRA fine-tuning adapts small language models (SLMs) to heterogeneous instruction data within a low-rank update subspace, making it vulnerable to three structural problems:
- conflicting gradients that cancel
- static data selection that cannot track evolving learning dynamics
- subspace saturation that causes later updates to overwrite useful directions
We argue that effective adaptation therefore requires controlling which data-induced gradients enter the LoRA subspace and when.
We propose GRADE (GRadient-Aligned Data-centric rEcipe), a data-centric framework combining two mechanisms:
- a state-aware selector that continually admits samples aligned with the evolving multi-task gradient field
- a self-calibrating step-level gate that rejects updates likely to cause destructive overwrite near saturation
Across three current-generation backbones and a heterogeneous seven-dataset instruction pool, GRADE outperforms strong data-selection and PEFT-stabilization baselines in accuracy and robustness.
It is the only method to improve consistently over standard LoRA on every architecture, while producing more coherent gradient trajectories and less destructive overwrite.
These results show that successful SLM adaptation depends not only on which data are selected, but also on which gradients are allowed to enter and persist in the constrained update subspace.