Modern ML research increasingly faces an evaluation bottleneck: building benchmarks and running experiments is costly, slowing iteration.
We ask how much can be inferred about evaluation outcomes before experiments, given only a natural-language description of the task and setup. We call this setting anticipatory evaluation.
EVALCAST: A Literature-Grounded Resource
We introduce EVALCAST, a literature-grounded resource of 2,290 description-performance pairs from 1,519 arXiv papers, including a test set of papers published after the model's knowledge cutoff.
Evaluation Tasks
We evaluate two tasks: predicting absolute performance through regression and comparing benchmark difficulty through pairwise prediction.
Experiments with Frontier LLMs
Experiments with frontier LLMs show that accurate regression remains challenging, while pairwise predictions can recover relative performance ordering under matched settings.
Post-Cutoff and Streaming Evaluations
Comparable performance on post-cutoff and streaming evaluations suggests that the observed signal is not solely attributable to previously reported results.
Overall Conclusion
Overall, EVALCAST provides a resource and evaluation framework for studying anticipatory evaluation and its potential use in prioritizing benchmarks.