Runtime and task design
Models receive complete global information and candidate team plans, with reference answers established through execution and exact search.
Results and diagnosis
Cross-model comparison
We evaluate 40 native decision models and compare them with 10 general-purpose language models. The best native decision model achieves 62.50% decision accuracy.
Native decision models remain close to chance in optimality verification. Paired tests reveal that some report better alternatives regardless of whether an optimal plan is already present.
These findings show that team-plan decision accuracy alone can conceal systematic errors in judging whether the supplied options contain an optimal arrangement.
The dataset and evaluation code are publicly available at https://github.com/lililibaohang/TandemBench.