背景介绍
Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements.
This omission is critical as retrieval oriented geo-analytic tasks cannot be supported reliably without first identifying the necessary datasets and constraints.
GeoFitBench-LLM的引入
GeoFitBench-LLM is introduced as a benchmark for assessing whether LLMs can infer the datasets, the supporting resources and the constraints required by geo-analytic requests in natural language.
The benchmark is structured around two reasoning dimensions, Spatio-Temporal Relational and Semantic Grounding, reflecting long-standing challenges in geospatial information retrieval and related natural language tasks.
评估方法
Ground truth requirements are encoded in a fixed, provider-independent schema for reproducible evaluation of resource identification and attribute-level specification.
实验结果
Experiments on seven models under minimal prompting indicate that identifying the necessary resources is often achievable.
GPT-5.4 Thinking achieves the best result by recovering the complete set of resources in 66.67% of benchmark requests.
Results further indicate that accurate fitness-for-use specification remains more difficult as inferential demands increase.