More and more of the text future language models learn from is written by a few of today's models.
If one supplier writes most of a shared corpus, does it pull the models trained on it toward its own writing, or change how fast they drift?
We Retrain Models
We retrain eight open models from their base weights on a shared pool of each other's text for five generations, varying the part written by one model, Phi-2, from an equal share to 90%.
Model Drift and Style
The models drift together toward a style with fewer function words, and none starts repeating itself.
No share of Phi-2 brings the other models closer to its text than the equal share does.
Delay and Departure
We split each ecosystem's separation from the equal-share one into a delay along its route and a departure from that route, both counted beyond the difference between two equal-share runs.
Impact of SmolLM2 and Qwen3-1.7B
When SmolLM2 or Qwen3-1.7B writes half instead, the ecosystem slows less or not at all.
Departure and Share of Phi-2
The departure leans toward Phi-2 more as its share grows, but more than toward every other model only at 90%.
Human Text Impact
Human text filling a quarter or half of the pool slows the models along the same route.