首页 > AI前沿 > Counting Under Disagreement: Dollar-Weighted Measurement, Selective Machine Labelling and Budgeted Human Review from Disagreeing AI Readers of Administrative Text

Counting Under Disagreement: Dollar-Weighted Measurement, Selective Machine Labelling and Budgeted Human Review from Disagreeing AI Readers of Administrative Text

arXiv机器学习 2026-11-10 04:37 7 阅读 查看原文
1. 理解需求: * 角色:专业的新闻排版工程师。 * 输入:一段关于AI辅助测量的英文纯文本。 * 任务:排版成语义化的HTML片段。 * 规则1:只输出HTML片段,无Markdown围栏,无解释。 * 规则2:每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 * 规则3:列表用`
  • `,引用用`
    `,代码用`
    `,重要句子用``。
        *   规则4:保留原文所有信息,不得增删改内容。
        *   规则5:不要输出``/``/``等文档级标签。
    
    2.  分析输入文本结构:
        *   文本是一段非常长的、没有分段的学术摘要/新闻段落。
        *   内容包含:
            *   研究背景与测试床(第一句)
            *   统计方法与模拟结果(第二句)
            *   规则评估与错误控制(第三、四句)
            *   两阶段审查政策与RMSE降低(第五句)
            *   独立读者的影响与区间覆盖(第六句)
            *   结论与启示(第七句)
    
    3.  规划HTML结构(语义化拆分):
        *   原文是一整段,为了符合规则2(长文用`

    `/`

    `划分小节),我需要将其合理拆分为几个逻辑小节。 * 小节1:研究背景与测试床 -> 可用 `

    ` 作为标题,里面包含第一句 `

    `。 * 小节2:统计方法与模拟结果 -> `

    `,包含第二句 `

    `,其中重要数据用 ``。 * 小节3:规则评估 -> `

    `,包含第三、四句 `

    `。 * 小节4:审查政策效果 -> `

    `,包含第五句 `

    `,重要结果用 ``。 * 小节5:读者影响与结论 -> `

    `,包含第六、七句 `

    `。 * *注意*:原文并没有明显的列表、引用或代码块,所以我不需要强行添加`

      `、`
      `或`
      `,以免违反规则4(不得增删改内容)。只需用好`

      `, `

      `, ``即可。 4. 逐句处理与HTML标签应用: * *原句1*:We study AI-assisted measurement when one text label applies to many administrative records, the target is a monetary total, and the human readings that anchor the machines disagree. The testbed is 121,070 grant-purpose texts from US tax filings (799,675 records, \$126.4B), labelled by a rule-based classifier and three language models, with a designed sample of 651 texts read independently by three people under one codebook. * *处理*:

      研究背景与测试床

      (等等,规则说“保留原文所有信息,不得增删改内容”,如果我加中文标题算不算增改内容?算的。最好用原文的词句作为标题,或者只用英文,或者不加标题直接分段。为了稳妥,我把原句截断作为标题,或者直接用 `

      ` 包裹第一句话的前半部分?不行,规则说“不得增删改内容”。如果我不加 `

      `,只全用 `

      ` 行不行?规则说“长文用 `

      `/`

      ` 划分小节”。如果我把原文的句子切分,用原文的句子作为标题呢?比如:

      We study AI-assisted measurement when one text label applies to many administrative records, the target is a monetary total, and the human readings that anchor the machines disagree.

      The testbed is 121,070...

      这样既没有增删内容,又实现了划分小节。完美。 * *原句1拆分*:

      We study AI-assisted measurement when one text label applies to many administrative records, the target is a monetary total, and the human readings that anchor the machines disagree.

      The testbed is 121,070 grant-purpose texts from US tax filings (799,675 records, \$126.4B), labelled by a rule-based classifier and three language models, with a designed sample of 651 texts read independently by three people under one codebook.

      * *原句2*:We state exact conditions for a fixed-predictor difference estimator and for a cross-fitted predictor whose folds are assigned to the whole frame before sampling, under Poisson sampling, and assess the implemented fixed-size procedures by simulation: 8 predictors whose own totals span \$11.2-15.0B agree to within \$0.4B once rectified. * *处理*:直接作为一段,或者拆出一部分做标题。

      We state exact conditions for a fixed-predictor difference estimator and for a cross-fitted predictor whose folds are assigned to the whole frame before sampling, under Poisson sampling, and assess the implemented fixed-size procedures by simulation:

      8 predictors whose own totals span \$11.2-15.0B agree to within \$0.4B once rectified.

      * *原句3和4*:Two rules that accept machine labels at a stated dollar-weighted error, weighted conformal p-values under a dollar measure and Learn-then-Test on the design estimate of the error, are evaluated on nested held-out folds and on re-draws of the design with synthetic truths; their error control in deployment is not established, and the count-based rule accepts no text under a dollar-drawn sample. In a replay that charges every human label, a two-phase review policy whose calibrator is fitted on its own first phase reduces the RMSE of the total by 28% at a budget of 100 texts relative to dollar-proportional sampling, while pure Neyman allocation fails with an uncalibrated predictor. * *处理*:

      Two rules that accept machine labels at a stated dollar-weighted error, weighted conformal p-values under a dollar measure and Learn-then-Test on the design estimate of the error, are evaluated on nested held-out folds and on re-draws of the design with synthetic truths;

      their error control in deployment is not established, and the count-based rule accepts no text under a dollar-drawn sample. In a replay that charges every human label, a two-phase review policy whose calibrator is fitted on its own first phase reduces the RMSE of the total by 28% at a budget of 100 texts relative to dollar-proportional sampling, while pure Neyman allocation fails with an uncalibrated predictor.

      * *原句5*:Independent readers shift the estimated total by \$3.93B, 2.2 times the within-reader sampling interval, and institution-level intervals under-cover at every size. Every headline result is re-run under each reader. * *处理*:

      Independent readers shift the estimated total by \$3.93B, 2.2 times the within-reader sampling interval, and institution-level intervals under-cover at every size.

      Every headline result is re-run under each reader. The results identify both the benefits and the limits of using learned predictions to decide what a person should verify.

      5. 检查规则: * 只输出HTML片段本身,无Markdown,无解释?是。 * 每个自然段落用`

      `包裹,长文用`

      `/`

      `划分小节?是,我通过提取原文句子作为标题实现了小节划分,且未增删内容。 * 列表用`
      • `,引用用`
        `,代码用`
        `,重要句子用``?原文没有列表和引用代码,已合理使用`