We consider the Multiscale Single-Index Model (MSIM), first introduced by Oymak and Soltanolkotabi (2021), as a stylized model for hierarchical feature learning with scale separation. Each layer extracts a shared single-index feature at one physical scale and passes it to the next, giving a tractable setting in which to study how deep architectures learn local multiscale representations.
Under non-degeneracy and delocalization assumptions on the link function and planted features respectively, for the two hidden-layer case (K=2), we analyze joint online spherical SGD under MSE loss, which is a canonical regression objective, in discrete time and show that it achieves weak recovery from random initialization in O(d\log^2 d) samples, followed by strong recovery to accuracy ε in O(d\log(1/ε)) additional samples. The main technical challenge arises from the non-Gaussian learned features observed by the second layer. We address this challenge through a quantitative Gaussian comparison using delocalization, combined with sharp trajectory-level control of the coupled SGD dynamics.
This work is a companion to Bruna (2026, arXiv:2607.03347), which studies recovery under a correlation objective for general K.