A language model normally begins training with random word embeddings: whatever 'banana' means must be learned from training corpora.
I implement St. Augustine's picture of word learning, meaning by ostension, for a small masked language model (DeBERTa) trained on 10M words: before training, visually grounded tokens receive embeddings derived from the image regions they label; other tokens start random.
Visual initialization leaves a measurable imprint that lasts until the end of training.
At the same time, the effect remains invisible under most BabyLM benchmarks, which probe abstract grammatical knowledge: visual initialization does not affect performance there.
The only zero-shot exception is object-property knowledge (COMPS), where seeding helps in every configuration.
To follow up on this result, I build a corpus-tailored version of the Visual-Property Swap benchmark, which tests color, material, size, and shape knowledge, with per-item training frequency and seeded status.
Here, vision-seeded models have a persistent, seed-replicated advantage.
Function words and abstract vocabulary also receive strong visual seeds and retain them throughout training, and the training objective draws on them: held-out mask-prediction loss falls for these words in every seed.
However, no benchmark I run registers this.
What evaluation would pick this up remains an open question.