The limited supervision available for the few-shot adaptation of vision-language models makes gradient estimation particularly noisy. We find that the gradient magnitudes are severely miscalibrated, while the signs of a small top-magnitude subset remain informative.
This motivates SignSparseSGD and DynSignSparseSGD, two state-free optimizers that use gradient magnitudes only to select where to update and fixed-magnitude signs to decide how to update.
- SignSparseSGD: fixes the support after the first minibatch.
- DynSignSparseSGD: refreshes it at every step.
Across 11 datasets, 3 CLIP backbones, and 5 parameter-levels, the proposed methods are competitive with or outperform parameter-efficient and optimization-based baselines on average, while exhibiting markedly lower sensitivity to long training schedules.
They introduce no task-specific modules and maintain no first- or second-moment optimizer states, while using one shared configuration across datasets, backbones, and parameter-levels.
Our code is included in the supplementary materials and will be released upon publication.