首页 > AI前沿 > MMDM: Measuring Massive Multitask Decision Making

MMDM: Measuring Massive Multitask Decision Making

arXiv自然语言 2026-11-11 00:40 27 阅读 查看原文

System-1 Decision Models directly make structured probabilistic decisions over query-specified options given a state in a single forward pass, without autoregressively generating reasoning trajectories or textual responses. Recent evaluations cover diverse tasks but often rely on existing benchmarks, where prior exposure to test material can obscure generalization. Moreover, real-world decision making often involves long contexts, large option spaces, and complex logical dependencies, challenges that existing benchmarks do not adequately capture together.

Thus, we introduce MMDM (Massive Multitask Decision Making), a benchmark of 5,448 high-quality questions across 16 task categories, combining newly constructed tasks with source-grounded recompositions and rewrites for single-select, multi-select, and soft-label decision making in fresh contexts.

MMDM: Massive Multitask Decision Making

It covers realistic scenarios such as agent state tracking, tool and skill selection, preference judgments, and rubric scoring, with option spaces ranging from 2 to 1,024 and inputs extending to 64,817 tokens.

Evaluation and Findings

With MMDM, we conduct a comprehensive quantitative evaluation of 28 decision models, showing that many perform well on common knowledge yet struggle with interdependent options, constrained decisions, and multi-select tasks.

For example, Perplexity Decider-V1.2 reaches 78.75% Hard Single-select Accuracy on world knowledge, but only 53.95% on option logic and 47.95% on constraint planning, while the best Hard Multi-select Accuracy across all models is 36.81%.
  • Perplexity Decider-V1.2 — Hard Single-select Accuracy on world knowledge: 78.75%
  • Perplexity Decider-V1.2 — Hard Single-select Accuracy on option logic: 53.95%
  • Perplexity Decider-V1.2 — Hard Single-select Accuracy on constraint planning: 47.95%
  • Best Hard Multi-select Accuracy across all models: 36.81%

These results highlight joint reasoning, constraint-aware selection, and complete multi-select recovery as key priorities for decision model development, with MMDM providing a testbed for measuring this progress.