首页 > AI前沿 > ZO-COSMO: Index-Free One-Hop Mixing for Decentralized Zeroth-Order Optimization

ZO-COSMO: Index-Free One-Hop Mixing for Decentralized Zeroth-Order Optimization

arXiv机器学习 2026-09-23 09:05 5 阅读 查看原文

Sparse communication in decentralized zeroth-order learning requires compatible peer-state coordinates.

We characterize this one-hop condition and develop \textsf{ZO-COSMO}, coupling two-query estimation with average-preserving masked consensus using $q$ values per active link.

Global supports serve all-neighbor mixing; matching updates require agreement only within each pair.

We derive a sharp contraction-per-scalar bound within the matching class and convergence guarantees for the core and sparse-momentum updates.

At fixed matching, exact moment identities characterize how shared directions preserve gradient-heterogeneity cancellation and redistribute estimation error and disagreement.

Mechanism experiments cover unequal curvatures, noise, and sparse momentum.

Further tests span $64$ synthetic agents and eight logical Qwen LoRA workers.

At matched payload budgets, Qwen2-7B QNLI gains $3.65$ accuracy points over explicit-index Rand-$k$; edge-local updates gain $3.42$ and $2.53$ points over all-neighbor mixing on eight-worker complete and ring graphs.

A matched-first-step ablation gives a $3.92$-point momentum benefit.

Seed-aware and same-matching controls distinguish encoding, scheduling, and query correlation.