首页 > AI前沿 > Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG

Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG

arXiv自然语言 2026-10-27 17:43 5 阅读 查看原文
用户现在需要我把给定的新闻正文排版成语义化HTML片段。首先,先看正文的结构,整个正文是一段长文,首先用

分小节吗?不对,先看内容,第一段是总起,然后接下来的内容需要拆分吗?等下,原文是一个长段落吗?不,原文是一整段,但需要划分小节吗?看内容,首先讲研究的问题,然后方法,结果,结论等。首先,开头第一句是总述,然后“Cattan et al. (2025)...”是背景,接下来“We study whether...”是研究目的,然后“Using the CONFLICTS benchmark...”是方法,“Asked directly...”是直接询问的结果,“The current Gemini 3.8 Flash...”是级联结果,“End to end...”是端到端结果,“Decomposing the taxonomy...”是分类分解结果,“Pre-specified diagnostics...”是诊断结果,“Decision models detect...”是关于假来源的结果,最后是发布链接。 首先,确定小节划分:比如用

划分成几个部分?比如:

研究背景与目标

When the sources retrieved for a query disagree...

Cattan et al. (2025) showed that...

We study whether typed decision models...

研究方法

Using the CONFLICTS benchmark and the original paper's prompt verbatim...

直接询问的性能结果

Asked directly, the decision models reach 50.0--67.0% accuracy; 10 of the 11 match or exceed Gemini 2.5 Flash, the paper's model (61.3%), at 4--39x lower billed cost (2 were free) and 2.6--6.0x lower latency.

The current Gemini 3.8 Flash (70.0%) is significantly more accurate than 8 of them, but sending each decision model's least confident quarter of queries to it gives a cascade statistically tied with it for 8 of the 11, at 25--37% of its cost.

端到端与分类分解结果

End to end, answers generated with the types of 10 of the 11 follow the expected behaviour as often as with the LLM's types.

Decomposing the taxonomy into yes/no questions lowers accuracy for every model, and a repair fitted on one model does not transfer.

诊断与假来源检测

Pre-specified diagnostics show that dates play no measurable role and that sensitivity to source order varies widely between models.

Decision models detect a planted false source less reliably than the LLM as hard labels, but their probabilities still separate planted from clean sources.

We release code, prompts and all model outputs at https://github.com/Parv17k/decision-models-conflict-typing.

等下,需要保留所有信息,不能增删改。首先,原文中的每个句子都属于不同的内容部分,需要正确划分小节。首先,检查每个句子的位置: 1. 第一句:When the sources retrieved for a query disagree... (总述冲突类型的重要性) 2. 第二句:Cattan et al. (2025) showed that... (背景:之前的研究结果) 3. 第三句:We study whether typed decision models... (研究目标) 4. 第四句:Using the CONFLICTS benchmark... (方法) 5. 第五句:Asked directly... (直接询问的性能) 6. 第六句:The current Gemini 3.8 Flash... (级联的性能) 7. 第七句:End to end... (端到端结果) 8. 第八句:Decomposing the taxonomy... (分类分解的结果) 9. 第九句:Pre-specified diagnostics... (诊断结果) 10. 第十句:Decision models detect... (假来源检测) 11. 第十一句:We release code... (发布链接) 接下来,划分小节:

冲突类型与研究背景

When the sources retrieved for a query disagree, the right response depends on why they disagree: a question with several valid answers, a contested topic, stale information and a false source each call for different behaviour.

Cattan et al. (2025) showed that telling a generator the conflict type improves its responses by about 24 points, but that predicting the type is hard even for frontier LLMs.

We study whether typed decision models, which return probabilities over fixed options in a single forward pass instead of generating text, can do this classification fast and cheaply.

研究方法

Using the CONFLICTS benchmark and the original paper's prompt verbatim, we evaluate all 11 decision models served by one commercial inference gateway, from 7 providers, against Gemini models on identical inputs.

性能评估结果

Asked directly, the decision models reach 50.0--67.0% accuracy; 10 of the 11 match or exceed Gemini 2.5 Flash, the paper's model (61.3%), at 4--39x lower billed cost (2 were free) and 2.6--6.0x lower latency.

The current Gemini 3.8 Flash (70.0%) is significantly more accurate than 8 of them, but sending each decision model's least confident quarter of queries to it gives a cascade statistically tied with it for 8 of the 11, at 25--37% of its cost.

End to end, answers generated with the types of 10 of the 11 follow the expected behaviour as often as with the LLM's types.

分类分解与诊断分析

Decomposing the taxonomy into yes/no questions lowers accuracy for every model, and a repair fitted on one model does not transfer.

Pre-specified diagnostics show that dates play no measurable role and that sensitivity to source order varies widely between models.

假来源检测与资源发布

Decision models detect a planted false source less reliably than the LLM as hard labels, but their probabilities still separate planted from clean sources.

We release code, prompts and all model outputs at https://github.com/Parv17k/decision-models-conflict-typing.

这样对吗?需要检查每个句子都包含在内,没有遗漏。现在,确认每个部分的内容正确,没有增删改。然后,确保HTML标签正确,比如

用于小节,

用于段落,链接用,重要内容比如“24 points”、“50.0--67.0%”、“70.0%”等可以用吗?等下,原文中的“about 24 points”、“50.0--67.0% accuracy”、“61.3%”、“70.0%”这些重要数据可以加吗?规则里说重要句子可用,所以比如“Cattan et al. (2025) showed that telling a generator the conflict type improves its responses by about 24 points, but that predicting the type is hard even for frontier LLMs.”这样的。 现在调整:

冲突类型与研究背景

When the sources retrieved for a query disagree, the right response depends on why they disagree: a question with several valid answers, a contested topic, stale information and a false source each call for different behaviour.

Cattan et al. (2025) showed that telling a generator the conflict type improves its responses by about 24 points, but that predicting the type is hard even for frontier LLM