Speech language models have expanded automatic speech recognition (ASR) from transcription to general speech-to-text generation. Yet existing benchmarks still assume a single ground-truth transcript per utterance, an assumption that struggles when models produce readable, formatted, or structured outputs. In this work, we introduce Grounded Generative Speech Recognition (GSR), reformulating speech recognition as generating semantically grounded text instead of a single transcript.
Grounded Reformulation Space and Capabilities
We define a grounded reformulation space that characterizes all valid textual realizations supported by the input speech, and establish a four-level hierarchy of reformulation capabilities:
- Literal transcription
- Canonical reformulation
- Semantic reformulation
- Speech-grounded structuring
GSR-Bench: Benchmark and Evaluation
Based on this formulation, we present GSR-Bench, the first comprehensive benchmark designed to evaluate generative speech recognition systems beyond exact transcript matching. GSR-Bench covers diverse scenarios including:
- Refined meeting transcription
- Cross-dialect translation
- Context-aware speech understanding
- Robustness under silent or noisy acoustic conditions
To address the limitations of word-level evaluation metrics, we develop a multi-dimensional evaluation framework combining speech grounding assessment and LLM-based semantic judgment, whose reliability is validated through human preference studies.
Experiments, Findings, and Failure Modes
Extensive experiments across representative architectures, ranging from conventional ASR systems to end-to-end Speech-LLMs, reveal fundamental trade-offs between transcription fidelity and generative reformulation ability. Our analysis further identifies critical failure modes, including:
- Acoustic hallucination
- Unsupported generation
- Instruction drift
These results provide insights into the design and evaluation of future grounded generative speech systems.