首页 > AI前沿 > Wanda-Guided Sparse Perturbations for Efficient Zeroth-Order Optimization

Wanda-Guided Sparse Perturbations for Efficient Zeroth-Order Optimization

arXiv机器学习 2026-11-12 13:25 23 阅读 查看原文

Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models through forward-only evaluations, but high-dimensional random perturbations can limit optimization efficiency. Sparse perturbations reduce the search dimension, making the selection of informative coordinates critical.

We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight--