首页 > AI前沿 > RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs

RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs

arXiv自然语言 2026-11-18 13:00 25 阅读 查看原文
用户现在需要我把给定的新闻正文排版成语义化的HTML片段,要遵循给定的规则。首先,原文是一个段落,先看结构,开头是背景,然后讲方法,再讲实验结果,最后是贡献。首先,这个内容可以分小节吗?先想,首先第一段可以讲背景和问题,然后引入RAW方法,然后讲实验和结果,最后讲贡献。不对,原文是一个连贯的段落?不对,原文是一整段,不过可以用h2来划分小节吗?等下,原文内容:第一部分是说部署的LLM每天处理大量对话,拒绝不少请求,但用户侧的拒绝成本没被测量,是盲区。然后说A/B测试拒绝在前沿实验室产品有约束,第三方平台也没法用。然后引入RAW方法,利用LLM的随机性做自然实验。然后讲应用到15个部署的LLM,发现的结果是一半左右的模型,拒绝会导致后续重新参与下降3到17个百分点,其余没有明显影响,其中一个的正估计是因为用户反复重试。最后贡献是测量方法,成本,还有发布的资源。 那怎么划分小节呢?可以用h2来分几个部分?比如第一个h2是“研究背景与问题”,然后下面p,然后h2“研究方法:RAW”,下面p,然后h2“实验结果”,下面p,然后h2“研究贡献”,下面p?不对,等下要保留原文所有信息,不能增删改,所以不能自己加小节标题的内容?不对,小节标题需要基于原文的内容来,不能自己瞎编?不对,规则说长文用h2/h3划分小节,那原文的内容可以拆分成几个逻辑部分,每个部分用h2,下面p。等下,原文的内容是连贯的,我看看: 首先,原文第一部分:“Deployed LLMs field hundreds of millions of conversations a day and refuse a non-trivial fraction of user requests, yet the user-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model: a blind spot for the consumer-facing platforms built on them.” 这部分是背景和问题。 然后:“A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API.” 这部分是现有方法的局限。 然后:“We introduce RAW, a method that uses LLM stochasticity as a natural experiment: at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal.” 这是方法。 然后:“Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect, among them one deployment whose positive point estimate we trace to persistent users retrying the refused request.” 这是实验结果。 最后:“Our contribution is the measurement method, which runs on any deployed model from logs alone for about \$0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.” 这是贡献。 那现在,划分小节的话,比如:

研究背景与现存局限

Deployed LLMs field hundreds of millions of conversations a day and refuse a non-trivial fraction of user requests, yet the user-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model: a blind spot for the consumer-facing platforms built on them. A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API.

研究方法:RAW自然实验

We introduce RAW, a method that uses LLM stochasticity as a natural experiment: at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal.

实验结果

Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect, among them one deployment whose positive point estimate we trace to persistent users retrying the refused request.

研究贡献与发布资源

Our contribution is the measurement method, which runs on any deployed model from logs alone for about \$0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.

不对,等下,小节标题是我自己加的?但规则说不能增删改内容,小节标题算不算增?哦,不对,规则说“保留原文所有信息,不得增删改内容”,那小节标题是不是属于增的?哦,等下,规则说“长文用

/

划分小节”,那如果原文是一个长段落,要划分小节,那小节标题应该是基于原文的内容提炼的,但不能添加原文没有的信息?不对,等下再看原文,原文是一整段,那如果我不划分小节,直接用p包裹?不对,规则说长文用h2/h3划分小节,那这个内容不算特别长,但逻辑上有几个部分,那如果我用h2的话,标题需要是原文中的意思,不能自己瞎编,比如有没有可能原文不需要划分,直接一个p?不对,再看规则,规则说“每个自然段落用

包裹,长文用

/

划分小节”,那原文是一个自然段落吗?是的,但内容较长,逻辑清晰,可以划分小节。不过等下,会不会我加的小节标题属于增了内容?哦,不对,小节标题是对内容的概括,不是新增信息,而且规则要求长文用h2/h3划分,所以是允许的?不对,再仔细看规则4:“保留原文所有信息,不得增删改内容”,也就是说,原文的信息不能少,也不能加新的信息,小节标题是对原文内容的划分,不算新增信息,因为是概括原文的部分内容,对吧? 或者,有没有可能,原文不需要划分小节,直接一个p?但那样可能不符合长文的要求。等下,再想想,原文的内容,有没有需要用列表、引用、代码的地方?这里没有列表、引用、代码,所以主要用h2和p,还有可以加strong来标注重要句子,比如“the user-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model”,还有“we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement”这些可以加strong。 等下,再调整,比如:

研究背景与测量困境

Deployed LLMs field hundreds of millions of conversations a day and refuse a non-trivial fraction of user requests, yet the user-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model: a blind spot for the consumer-facing platforms built on them. A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API.

核心方法:RAW自然实验

We introduce RAW, a method that uses LLM stochasticity as a natural experiment: at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal.

实验结果

Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect, among them one deployment whose positive point estimate we trace to persistent users retrying the refused request.

研究贡献与资源发布

Our contribution is the measurement method, which runs on any deployed model from logs alone for about \$0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.</