首页 > AI前沿 > Logical AI OS: A Training-Free Governance Layer Outside the Model for Instruction Following -- Benefits, Costs, Failures, and the Calibration of the Measuring Instrument Itself

Logical AI OS: A Training-Free Governance Layer Outside the Model for Instruction Following -- Benefits, Costs, Failures, and the Calibration of the Measuring Instrument Itself

arXiv自然语言 2026-11-15 11:48 19 阅读 查看原文
用户需要我扮演专业新闻排版工程师,将给定的纯文本新闻正文排版成语义化HTML片段。需要严格遵守所有约束条件: 1. 只输出HTML片段本身,不要Markdown围栏,不要解释。 2. 每个自然段落用

包裹,长文用

/

划分小节。 3. 列表内容用
  • ,引用用
    ,代码用
    ,重要句子可用。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出//等文档级标签。
    
    先分析正文:
    正文主要包含一个长段落,可以分为几个逻辑小节,包括背景与方法、实验设置与结果、探索性发现、成本、失败案例、修正与结论。
    我需要用

    或

    来划分小节,将内容合理分配到

    中,列表部分使用

    • ,对关键数据和结论用。 结构规划:

      研究背景与方法

      Small language models often fail... calibrated.

      (背景和贡献)

      实验设置与结果

      On four 4-bit Qwen-family models... fix;

      (实验配置和第一个结果,可以加粗关键数据)
      • ...
      • ...
      • ...
      • ...
      • ...
      (因为原文用(1)到(5)列举了结果,所以可以转换为
      • 列表,这样符合“列表内容用
        • ”的规则)

          结论与数据公开

          Conclusions are limited to... public.

          详细拆分: 第一段:Small language models often fail explicit user requests (length, number of items, format, required or forbidden words, grammatical person, answer language). We measure a training-free governance layer placed outside the model: it reads the requests with rules, checks the answer mechanically, repairs it when a request is not kept, and hands back the original answer when the repair fails. The loop exists in prior work; our contribution is measuring its benefits, costs and failures, and reporting how the measuring instrument itself was calibrated. 第二段:On four 4-bit Qwen-family models (3B, 7B, 14B, one 7B reasoning model) and 329 Chinese prompts: (1) answers that keep every request the layer read rise significantly (all comparisons p<0.0001), by 34 to 66 of 329 under the corrected checker and 52 to 92 after a length-control fix; a generic redo message brings no significant gain. (2) Exploratory: on 90 repair-triggered prompts, two AI annotators found the layer produced an acceptable answer where the bare model did not more often than the reverse (49:7; 18:12, not significant). (3) Costs: 43-66% more generation time; whole-answer rewriting makes a sizeable share of usable answers unusable. (4) Failures: rule-based reading reaches about 70% and 84% of a two-annotator ceiling on unseen phrasings; unsatisfiable requests make the reasoning model loop; model self-reports are unreliable (18 of 25 "fixed" items were not fixed). (5) Over half of the original checker's "violated" verdicts on structural requests were wrong; after correction the main result holds, and one comparison is downgraded. 第三段:Conclusions are limited to this Chinese material, the Qwen family and sizes up to 14B. The governance implementation is not released; data, answers, verdicts and the checker are public. 排版处理:

          研究背景与治理方法

          Small language models often fail explicit user requests (length, number of items, format, required or forbidden words, grammatical person, answer language). We measure a training-free governance layer placed outside the model: it reads the requests with rules, checks the answer mechanically, repairs it when a request is not kept, and hands back the original answer when the repair fails. The loop exists in prior work; our contribution is measuring its benefits, costs and failures, and reporting how the measuring instrument itself was calibrated.

          实验设置与核心发现

          On four 4-bit Qwen-family models (3B, 7B, 14B, one 7B reasoning model) and 329 Chinese prompts, the study yields the following results:

          • (1) answers that keep every request the layer read rise significantly (all comparisons p<0.0001), by 34 to 66 of 329 under the corrected checker and 52 to 92 after a length-control fix; a generic redo message brings no significant gain.
          • (2) Exploratory: on 90 repair-triggered prompts, two AI annotators found the layer produced an acceptable answer where the bare model did not more often than the reverse (49:7; 18:12, not significant).
          • (3) Costs: 43-66% more generation time; whole-answer rewriting makes a sizeable share of usable answers unusable.
          • (4) Failures: rule-based reading reaches about 70% and 84% of a two-annotator ceiling on unseen phrasings; unsatisfiable requests make the reasoning model loop; model self-reports are unreliable (18 of 25 "fixed" items were not fixed).
          • (5) Over half of the original checker's "violated" verdicts on structural requests were wrong; after correction the main result holds, and one comparison is downgraded.

          结论与数据公开

          Conclusions are limited to this Chinese material, the Qwen family and sizes up to 14B. The governance implementation is not released; data, answers, verdicts and the checker are public.

          检查规则: 1. 只输出HTML片段,无Markdown,无解释。 2. 每个自然段用

          ,长文用

          /

          划分。这里用了

          。 3. 列表用
          • ,重要句子用。 4. 保留原文所有信息,无增删改。注意原文中的`p<0.0001`需要转义为`p<0.0001`,防止被当作HTML标签。 5. 无//标签。 检查转义:`p<0.0001` -> `p<0.0001`。 原文:"p<0.0001",在HTML中`<0`可能会被解析为标签,所以必须转义。 再核对原文信息是否完全一致: 原文第一段:Small language models often fail explicit user requests (length, number of items, format, required or forbidden words, grammatical person, answer language). We measure a training-free governance layer placed outside the model: it reads the requests with rules, checks the answer mechanically, repairs it when a request is not kept, and hands back the original answer when the repair fails. The loop exists in prior work; our contribution is measuring its benefits, costs and failures, and reporting how the measuring instrument itself was calibrated. 完全一致。 原文第二段:On four 4-bit Qwen-family models (3B, 7B, 14B, one 7B reasoning model) and 329 Chinese prompts: (1) answers that keep every request the layer read rise significantly (all comparisons p<0.0001), by 34 to 66 of 329 under the corrected checker and 52 to 92 after a length-control fix; a generic redo message brings no significant gain. (2) Exploratory: on 90 repair-