研究背景与现存局限
Deployed LLMs field hundreds of millions of conversations a day and refuse a non-trivial fraction of user requests, yet the user-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model: a blind spot for the consumer-facing platforms built on them. A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API.
研究方法:RAW自然实验
We introduce RAW, a method that uses LLM stochasticity as a natural experiment: at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal.
实验结果
Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect, among them one deployment whose positive point estimate we trace to persistent users retrying the refused request.
研究贡献与发布资源
Our contribution is the measurement method, which runs on any deployed model from logs alone for about \$0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.
不对,等下,小节标题是我自己加的?但规则说不能增删改内容,小节标题算不算增?哦,不对,规则说“保留原文所有信息,不得增删改内容”,那小节标题是不是属于增的?哦,等下,规则说“长文用/
划分小节”,那如果原文是一个长段落,要划分小节,那小节标题应该是基于原文的内容提炼的,但不能添加原文没有的信息?不对,等下再看原文,原文是一整段,那如果我不划分小节,直接用p包裹?不对,规则说长文用h2/h3划分小节,那这个内容不算特别长,但逻辑上有几个部分,那如果我用h2的话,标题需要是原文中的意思,不能自己瞎编,比如有没有可能原文不需要划分,直接一个p?不对,再看规则,规则说“每个自然段落用
包裹,长文用
/
划分小节”,那原文是一个自然段落吗?是的,但内容较长,逻辑清晰,可以划分小节。不过等下,会不会我加的小节标题属于增了内容?哦,不对,小节标题是对内容的概括,不是新增信息,而且规则要求长文用h2/h3划分,所以是允许的?不对,再仔细看规则4:“保留原文所有信息,不得增删改内容”,也就是说,原文的信息不能少,也不能加新的信息,小节标题是对原文内容的划分,不算新增信息,因为是概括原文的部分内容,对吧? 或者,有没有可能,原文不需要划分小节,直接一个p?但那样可能不符合长文的要求。等下,再想想,原文的内容,有没有需要用列表、引用、代码的地方?这里没有列表、引用、代码,所以主要用h2和p,还有可以加strong来标注重要句子,比如“the user-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model”,还有“we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement”这些可以加strong。 等下,再调整,比如:
研究背景与测量困境
Deployed LLMs field hundreds of millions of conversations a day and refuse a non-trivial fraction of user requests, yet the user-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model: a blind spot for the consumer-facing platforms built on them. A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API.
核心方法:RAW自然实验
We introduce RAW, a method that uses LLM stochasticity as a natural experiment: at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal.
实验结果
Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect, among them one deployment whose positive point estimate we trace to persistent users retrying the refused request.
研究贡献与资源发布
Our contribution is the measurement method, which runs on any deployed model from logs alone for about \$0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.</