We introduce LoRA-CRAFT (CROSS-layer RANK Adaptation via Frozen TUCKER), abbreviated CRAFT throughout, an extremely parameter-efficient fine-tuning (PEFT) method that applies Tucker tensor decomposition to pre-trained attention weight matrices stacked across transformer layers and trains only small square adaptation matrices on the resulting frozen Tucker factors.
Existing tensor-based PEFT methods decompose GRADIENT UPDATES: LoTR applies Tucker decomposition with shared factor matrices, while SuperLoRA groups and reshapes $ΔW$ across layers before applying Tucker decomposition. Separately, methods such as PiSSA apply SVD to PRE-TRAINED WEIGHTS but operate independently per layer.
CRAFT bridges these two lines of work: it performs full Tucker decomposition via Higher-Order SVD (HOSVD) directly on PRE-TRAINED WEIGHTS organized as cross-layer 3D tensors, freezes all resulting factors, and adapts the model through lightweight trainable transformations applied to each factor matrix.
Experiments on the GLUE benchmark using RoBERTa-base and RoBERTa-large, as well as commonsense reasoning benchmarks using LLaMA2-7B and LLaMA3-8B, demonstrate that CRAFT achieves competitive performance with existing methods while requiring only EXTREMELY LOW TUCKER ADAPTATION PARAMETERS.
On LLaMA3-8B, CRAFT exceeds the average accuracy of LoRA using hundreds of times fewer parameters; on LLaMA2-7B the same holds at a $0.252$M budget.
Our results suggest that CRAFT's efficiency advantage grows with model scale, as the frozen Tucker factors better capture the richer cross-layer structure of larger pre-trained models.