首页 > AI前沿 > When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents

When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents

arXiv自然语言 2026-09-03 15:24 4 阅读 查看原文

Large language models (LLMs) are increasingly deployed as long-horizon conversational agents, motivating growing interest in memory systems.

However, existing benchmarks primarily evaluate memory through QA-style probing rather than in-situ conversational usage.

We introduce LOCOMO-CONV, a conversational memory benchmark derived from LoCoMo with four query styles: dialog, implicit, counterfactual, and composed.

Across five representative memory systems, we evaluate both retrieval recall and end-to-end response quality.

Our experiments show that conversational framing exposes substantial retrieval gaps overlooked by QA benchmarks, especially on implicit and composed queries, which multi-facet query rewriting narrows for raw-turn memory but not abstractive memory.

We further find that strong retrieval does not fully translate into response quality, and that implicit queries exhibit silent grounding, where memory improves contextual grounding without explicitly surfacing the gold fact.

These results point to reasoning-based memory elaboration as a promising direction, and we release auxiliary supportive_memory annotations capturing conversationally useful context beyond the original gold evidence.