首页 > AI前沿 > TRIAGE: Reliability- and Load-Conditioned Escalation for Two-Tier LLM Serving with a Semantic Evidence Judge

TRIAGE: Reliability- and Load-Conditioned Escalation for Two-Tier LLM Serving with a Semantic Evidence Judge

arXiv机器学习 2026-11-01 02:25 5 阅读 查看原文
大模型服务 语义证据判断 负载调度
1. 理解需求： * 角色： 专业新闻排版工程师。 * 输入： 一段关于“TRIAGE”（两层大模型服务系统）的纯文本新闻正文。 * 任务： 将其排版成语义化的HTML片段。 * 规则： 1. 仅输出HTML片段本身（禁止Markdown围栏，禁止解释）。 2. 每个自然段落用 `

` 包裹；长文用 `

`/`

` 划分小节（虽然这里只有一段，但我可能需要将长句拆分成段落，或者为了保留原文，就只用一个 `

`。等等，规则要求“每个自然段落用 `

` 包裹”。输入文本其实是一整段长篇内容。我要不要根据逻辑拆分呢？我看一下：“Two-tier LLM serving escalates...” 其实是一整段。不过，仔细看看，它包含背景、方法、结果和结论。为了稳妥且符合“保留原文所有信息，不得增删改内容”的要求，我或许可以把它放在一个 `

` 里，或者用 `

`/`

` 拆分小节，但不修改文字。考虑到它只是一个简短的单段落摘要，就用一个 `

` 包裹整段，但为了迎合规则2，也许我应该把相关的小节拆分出来。让我在不丢失原词的前提下，把它拆分为合乎逻辑的段落。其实，安全一点的做法是直接把整段文本放在一个 `

` 标签里，因为它只是一段新闻片段。或者我也可以用 `

` 来划分主题。为了保险起见，还是不要删减或重构文字。“每个自然段落用 `

` 包裹”——原文是一个自然段落。所以我将只用 `

`。等等，如果我用 `

`/`

`，那就是增加结构。我稍微增加一些标题使其更有新闻感？不行，规则说“不得增删改内容”。增加标题算不算增加内容？通常不算，但把一整段拆成几个 `

` 标签会改变格式。我还是就把它作为一个单独的 `

` 块输出吧。 3. 列表用 `

  • `，引用用 `
    `，代码用 ` 
    `，重要句子用 ``。
        4. 保留所有信息，不增删改内容。
        5. 不要输出文档级标签（如 ``, ``, ``）。

2. 分析输入文本：
    *   “Two-tier LLM serving escalates selected requests from a small model to a large one, but prompt-based routers cannot see whether retrieved evidence supports an answer, and they treat the GPU cost of an escalation as constant. We study TRIAGE, which conditions escalation on a post-generation reliability state and on a load-dependent GPU price, and we report both what held and what did not on one A40 serving Qwen2.5-7B and 32B (AWQ) under VLLM. Escalation cost rose 21.9x from 1 to 10 requests/s but showed no monotone dependence on KV-cache headroom, and a pre-registered claim that the reliability state detects corrupted evidence (AUROC>=0.70) was falsified at 0.571, although any prompt-only router scores exactly 0.500 by construction. Post hoc, a semantic evidence judge raised detection to 0.846 (0.926 on never-served items) and predicted wrong small-model answers at AUROC 0.90-0.96 on RGB and three RouterBench model pairs; the resulting cascade stayed within about 3 pp of GPT-4 at 79.5% lower API cost for GPT-3.5->GPT-4, yet fell below random routing for open-model pairs with 2-3x price ratios. End to end, the 7B->32B pair showed only a +2.88 pp held-out gap, so the allocator rightly declined to escalate and the headline is a null that a small pre-launch gate failed to expose.”

3. 确定结构与格式规则应用：
    *   原文是一个单一的自然段落。为了严格遵守“不得增删改内容”，我会把它放在一个单独的 `
    ` 标签里。等等，我能不能把开头第一句用 `
    ` 或者 `
    ` 提出来作为标题？“Two-tier LLM serving...” 可以作为标题吗？如果当作标题，那么文本就需要进行拆分。提示词说“长文用 `
    `/`
    ` 划分小节”。这篇文本是一段摘要（大约200字）。算不上真正的“长文”。所以只用 `
    ` 是完全没问题的。
    *   重要句子：我应该使用 `` 来突出关键发现，以展示我应用了这条规则。比如哪些是重要句子？
        *   “We study TRIAGE, which conditions escalation on a post-generation reliability state and on a load-dependent GPU price”
        *   “the headline is a null that a small pre-launch gate failed to expose.”
    *   有没有需要 `
    ` 的？“a pre-registered claim that the reliability state detects corrupted evidence (AUROC>=0.70) was falsified at 0.571” —— 这不是直接引用，所以不需要 `
    
           `。 * 有没有需要 ` 
           
    ` 的？没有。
    *   有没有需要 `
    • ` 的？没有。
    *   所以最终输出主要由 `
      ` 组成，并在内部加上一些 `` 标签。

4. 起草HTML：
    ```html
    
      Two-tier LLM serving escalates selected requests from a small model to a large one, but prompt-based routers cannot see whether retrieved evidence supports an answer, and they treat the GPU cost of an escalation as constant. We study TRIAGE, which conditions escalation on a post-generation reliability state and on a load-dependent GPU price, and we report both what held and what did not on one A40 serving Qwen2.5-7B and 32B (AWQ) under VLLM. Escalation cost rose 21.9x from 1 to 10 requests/s but showed no monotone dependence on KV-cache headroom, and a pre-registered claim that the reliability state detects corrupted evidence (AUROC>=0.70) was falsified at 0.571, although any prompt-only router scores exactly 0.500 by construction. Post hoc, a semantic evidence judge raised detection to 0.846 (0.926 on never-served items) and predicted wrong small-model answers at AUROC 0.90-0.96 on RGB and three RouterBench model pairs; the resulting cascade stayed within about 3 pp of GPT-4 at 79.5% lower API cost for GPT-3.5->GPT-4, yet fell below random routing for open-model pairs with 2-3x price ratios. End to end, the 7B->32B pair showed only a +2.88 pp held-out gap, so the allocator rightly declined to escalate and the headline is a null that a small pre-launch gate failed to expose.
      
    ```

5. 检查约束条件：
    *   仅输出HTML片段？是的。
    *   无Markdown围栏？是的。
    *   无解释？是的。
    *   每个自然段落用 `
      ` 包裹？是的。
    *   长文用 `
      `/`
      ` 划分小节？由于原文只是一段，这不是长文，因此跳过该规则。
    *   列表用 `
        
                  
      • `？不适用。 * 引用用 ` 
                   
        
                    `？不适用。 * 代码用 ` 
                    
        `？不适用。
    *   重要句子用 ``？是的，已应用。
    *   保留原文所有信息？是的，未增删改内容。
    *   特殊字符转义：`AUROC>=0.70` -> `AUROC>=0.70`，`GPT-3.5->GPT-4` -> `GPT-3.5->GPT-4`，`7

相关推荐