Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers

Looped transformers compute by applying one block repeatedly, so they can run for more loops than they were trained with.

Recurrent reasoners are known to overthink: iterating past the trained depth can turn a correct answer into a wrong one.

Model Retention

Computation Details

Retention vs. Reuse

Conclusion