Low-rank adaptation (LoRA) has attracted significant attention in the parameter-efficient fine-tuning (PEFT) landscape. However, its design suffers from two fundamental limitations.
- First, its invariance manifold induces flat directions in the loss landscape, creating an optimization bottleneck for adaptive optimizers.
- Second, its effective rank scales only linearly with the parameter budget.
Scalable Morpho Adaptation (SaMA)
In this paper, we propose Scalable Morpho Adaptation (SaMA), a PEFT method grounded in a principled generalization of the Kronecker product. By relaxing the shared-block constraint in the perfect-shuffle factorization, SaMA spans from low to full rank and achieves a practical rank that scales quadratically with the parameter budget, up to full-rank.
Invariance Manifold and Optimization
Furthermore, we show that its invariance manifold collapses to a diagonal group, making it structurally more optimizer-friendly.
Empirical Results
Empirically, SaMA consistently outperforms strong PEFT baselines on commonsense and arithmetic reasoning benchmarks while being more parameter-efficient.