首页 > AI前沿 > Complete-SVD: A Functional Norm Approach to Low-Rank LLM Compression

Complete-SVD: A Functional Norm Approach to Low-Rank LLM Compression

arXiv自然语言 2026-11-19 21:04 9 阅读 查看原文

研究背景与问题

Post-training low-rank compression of neural networks must not only approximate the original weights but also preserve the model's input-output behavior. Recent output-aware methods measure the compression error beyond the compressed layer, at the model output, weighting logit changes by the observed next token or by the model's predicted probabilities.

Complete-SVD方法

We propose Complete-SVD, a method for large language models that minimizes the $L_2$ distance between the logits of the original and compressed models, weighting all logit changes equally.

Linearizing the network around the original weights, together with a Kronecker-factored approximation, reduces this objective to a weighted SVD with a closed-form solution. Its output-side factor is intractable to compute; we estimate it without bias from random probes, at a cost of $K$ backward passes per sequence, independent of vocabulary size and sequence length.

实验结果与对比

At matched settings with uniform ranks and WikiText-2 calibration, Complete-SVD achieves the lowest C4 and PTB perplexity in every setting, and at ratio 0.7 under LEMS rank allocation.

Replacing the $L_2$ distance by the Kullback-Leibler divergence (KL-SVD), with the same estimation framework, lowers calibration perplexity in most settings and raises downstream accuracy, but raises cross-corpus perplexity.

Complete-SVD also improves IFEval instruction-following accuracy over the strongest prior output-aware methods throughout.