Retrieved scientific literature can serve as inspiration for both human and AI scientists.
Inspiration can take different forms: prior work may directly suggest how to address a problem, or surface directions at different levels of abstraction---zooming out to a more general view or zooming in to a concrete realization.
We introduce RATIO (Retrieval Across Typed Ideation Operations)
RATIO is a large-scale benchmark in which relevance is defined by three operations which we name ideation moves:
- Address retrieves potential approaches for stated problems,
- Broaden retrieves more general formulations,
- Specify retrieves concrete instantiations.
RATIO is constructed from millions of full-text scientific papers across CS literature via a general recipe that extends discourse-marker distant supervision---previously used only for classification---to corpus-scale retrieval, combined with extensive LLM and human vetting.
Experiments show that operation-specific fine-tuning substantially boosts retrievers and LLM reranking helps further, but much room for improvement remains.
Researchers blind to condition found the fine-tuned retriever's candidates more useful as inspirations.
RATIO provides a scalable training and evaluation framework
for retrieval components that support literature-grounded ideation, opening up new research avenues in scientific inspiration retrieval.
We release full data, code, models, and annotations.