Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models through forward-only evaluations, but high-dimensional random perturbations can limit optimization efficiency. Sparse perturbations reduce the search dimension, making the selection of informative coordinates critical.
We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), which uses Wanda weight--