首页 > AI前沿 > Evaluating safeguarding in multi-turn conversations with AI assistants

Evaluating safeguarding in multi-turn conversations with AI assistants

arXiv自然语言 2026-11-19 20:28 13 阅读 查看原文
用户需要我扮演专业的新闻排版工程师,将一段纯文本新闻正文排版成语义化的HTML片段。 规则如下: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出``/``/``等文档级标签。
    
    原文内容:
    Conversational AI assistants are now used for personal and emotional support. Developers report improvements in how new models respond to sensitive conversations, yet users and civil society continue to raise concerns about their behaviour in real-world, repeated interactions that safety evaluations rarely capture. Often, evaluations present signs of distress clearly and in a single message, even though people may reveal their struggles gradually in otherwise ordinary conversations. We worked with three community mental health support workers to test if models provide the same level of safeguarding when cues are shared all at once or gradually over time. We designed nine scenarios in which a user shares cues related to depression, psychosis, or domestic abuse. For each scenario, the support workers established a gold standard response, ranging from signposting support to pausing the everyday task to address safeguarding. We presented five cues either together in one message or dispersed across a conversation. Across more than 7,400 conversations, we evaluated six widely used models from five developers. GPT-5.5, Grok 4.3, DeepSeek V4, and Llama 4 Maverick met the support workers' standards in just 26-59% of conversations when cues were concentrated, and only 3-6% when dispersed. Claude Sonnet 5 and Claude Opus 5 showed a smaller gap, meeting the standards in 68-97% of concentrated conversations and 59-87% of dispersed conversations. Even the best-performing model, Claude Opus 5, fell short of the gold standard in 13% of conversations when cues were dispersed. Safety evaluations centred on explicit, legible concerns can overstate how well safeguards protect people when signs of distress emerge gradually or indirectly through ordinary conversations. For claims of improved safety to be taken seriously, developers must design and assess safeguards around how people actually communicate.
    
    分析原文段落和结构:
    原文是一整段。为了符合“长文用`

    `/`

    `划分小节”以及“每个自然段落用`

    `包裹”,我需要合理划分小节和段落,同时不得增删改内容。 可以划分成以下几个小节(不改变原文文字,只添加标签和换行): 小节1:背景与问题 (Background and Problem) 段落1:Conversational AI assistants are now used for personal and emotional support. Developers report improvements in how new models respond to sensitive conversations, yet users and civil society continue to raise concerns about their behaviour in real-world, repeated interactions that safety evaluations rarely capture. Often, evaluations present signs of distress clearly and in a single message, even though people may reveal their struggles gradually in otherwise ordinary conversations. 小节2:研究方法 (Research Methodology) 段落2:We worked with three community mental health support workers to test if models provide the same level of safeguarding when cues are shared all at once or gradually over time. We designed nine scenarios in which a user shares cues related to depression, psychosis, or domestic abuse. For each scenario, the support workers established a gold standard response, ranging from signposting support to pausing the everyday task to address safeguarding. We presented five cues either together in one message or dispersed across a conversation. 小节3:测试结果 (Test Results) 段落3:Across more than 7,400 conversations, we evaluated six widely used models from five developers. 这里可以把模型列表做成`

    • `吗?原文是:"GPT-5.5, Grok 4.3, DeepSeek V4, and Llama 4 Maverick met the support workers' standards in just 26-59% of conversations when cues were concentrated, and only 3-6% when dispersed. Claude Sonnet 5 and Claude Opus 5 showed a smaller gap, meeting the standards in 68-97% of concentrated conversations and 59-87% of dispersed conversations." 如果改成列表,可能会改变原文的句式结构,增加或减少标点。规则说“不得增删改内容”,最好保持原句,或者只把句子用`

      `包裹。不过规则说“列表内容用`

      • `”,如果原文本身不是列表形式,强行改成列表可能算“改内容”。所以保持段落形式,或者把长句拆分成段落? 原文:"GPT-5.5, Grok 4.3, DeepSeek V4, and Llama 4 Maverick met the support workers' standards in just 26-59% of conversations when cues were concentrated, and only 3-6% when dispersed. Claude Sonnet 5 and Claude Opus 5 showed a smaller gap, meeting the standards in 68-97% of concentrated conversations and 59-87% of dispersed conversations. Even the best-performing model, Claude Opus 5, fell short of the gold standard in 13% of conversations when cues were dispersed." 这些可以作为同一个段落,或者分成几个段落。 小节4:结论与建议 (Conclusion and Recommendations) 段落4:Safety evaluations centred on explicit, legible concerns can overstate how well safeguards protect people when signs of distress emerge gradually or indirectly through ordinary conversations. For claims of improved safety to be taken seriously, developers must design and assess safeguards around how people actually communicate. 检查重要句子,可以用``: 比如:"yet users and civil society continue to raise concerns about their behaviour in real-world, repeated interactions that safety evaluations rarely capture" "For claims of improved safety to be taken seriously, developers must design and assess safeguards around how people actually communicate." 让我们细化HTML结构:

        Background and Problem

        Conversational AI assistants are now used for personal and emotional support. Developers report improvements in how new models respond to sensitive conversations, yet users and civil society continue to raise concerns about their behaviour in real-world, repeated interactions that safety evaluations rarely capture. Often, evaluations present signs of distress clearly and in a single message, even though people may reveal their struggles gradually in otherwise ordinary conversations.

        Research Methodology

        We worked with three community mental health support workers to test if models provide the same level of safeguarding when cues are shared all at once or gradually over time. We designed nine scenarios in which a user shares cues related to depression, psychosis, or domestic abuse. For each scenario, the support workers established a gold standard response, ranging from signposting support to pausing the everyday task to address safeguarding. We presented five cues either together in one message or dispersed across a conversation.

        Test Results

        Across more than 7,400 conversations, we evaluated six widely used models from five developers. GPT-5.5, Grok 4.3, DeepSeek V4, and Llama 4 Maverick met the support workers' standards in just 26-59% of conversations when cues were concentrated, and only 3-6% when dispersed. Claude Sonnet 5 and Claude Opus 5 showed a smaller gap, meeting the standards in 68-97% of concentrated conversations and 59-87% of dispersed conversations. Even the best-performing model, Claude Opus 5, fell short of the gold standard in 13% of conversations when cues were dispersed.

        Conclusion and Recommendations

        Safety evaluations centred on explicit, legible concerns can overstate how well safeguards protect people when signs of distress emerge gradually or indirectly through ordinary conversations. For claims of improved safety to be taken seriously, developers must design and assess safeguards around how people actually communicate.

        检查是否满足所有规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 -> 满足 2. 每个自然段落用`

        `包裹,长文用`

        `/`

        `划分小节。 -> 满足,使用了`

        `划分小节,段落