Masked latent prediction methods such as I-JEPA aim to learn predictable features, but what determines which information they retain is less clear. We built a simulator in which every factor behind an image is known, so a probe can measure how readable each factor remains after training.
Experimental Setup
-
We trained small encoders (0.7M parameters) with I-JEPA, SimCLR and BYOL on the same pairs of views and probed a factor that both views share.
-
I-JEPA had a predictor with half the encoder's width and depth, the proportions it uses for ViT-B. Its median score was about 40% of SimCLR's.
Key Findings
Widening the predictor to the encoder's width reversed this ordering at every training seed, against both other methods.
We release the simulator and the trained models.