Neural scaling laws predict how language model performance improves with increased training inputs. While aggregate metrics like validation loss can follow smooth power-law curves, individual downstream tasks exhibit diverse scaling behaviors:
- some improve monotonically
- others plateau
- some even degrade with scale
Limitations of validation-loss-based prediction
We argue that predicting downstream performance from validation loss suffers from two limitations:
- averaging token-level losses obscures signal
- no simple parametric family can capture the full spectrum of scaling behaviors
Neural Scaling Laws (NeuNeu)
To address this, we propose Neural Neural Scaling Laws (NeuNeu), a neural network that frames scaling law prediction as time-series extrapolation. NeuNeu combines temporal context from observed accuracy trajectories with token-level validation losses, learning to predict future performance without the limitations inherent in assuming a specific functional form.
Training and results
Trained entirely on open-source model checkpoints from HuggingFace, NeuNeu achieves 1.99% mean absolute error in predicting model accuracy on 66 downstream tasks -- a 44% reduction compared to logistic scaling laws (3.56% MAE).
Generalization
Furthermore, NeuNeu generalizes zero-shot to unseen model families, architectures, parameter counts, and downstream tasks.
Conclusion
Our work suggests that predicting downstream scaling directly from data outperforms parametric alternatives.