Larger transformers can memorize more data, with a capacity (in bits) that scales roughly linearly with the number of parameters. Counterintuitively, larger models also memorize faster, requiring fewer repetitions of a sequence during training before it is memorized.
Experimental Setup and Results
We reproduce both memorization scaling laws in transformers trained on random binary sequences, showing that rapid memorization can arise even when data contain no semantic or compositional structure.
Explanation of the Effect
Rapid memorization is due to a meta-memorization effect, where models trained to memorize a large set of random sequences learn to memorize novel sequences more quickly.
The rate $r(t)$ at which models memorize grows with training time $t$ as a power-law:
r(t) ~ t^βwith a fixed exponent $β$ but an increasing upper cutoff across model scales.
Meta-memorization enables even one-layer transformers to few-shot memorize a 100-bit sequence.
This effect is partly explained by the formation of structured attention heads that convey position-invariant sequence information to MLPs where memorized sequences are stored.
Simplified Model Analysis
Meta-memorization is recapitulated in a simplified model of a dense associative memory where retrieval is constrained by an attentional bottleneck.
Conclusions and Implications
Our results suggest transformers, through pre-training, acquire reusable mechanisms that enable highly efficient storage and retrieval of memories.