首页 > AI前沿 > Robustness Starts with the Objective: Designing and Certifying Robust LLM-Generated Heuristics

Robustness Starts with the Objective: Designing and Certifying Robust LLM-Generated Heuristics

arXiv自然语言 2026-10-31 08:49 3 阅读 查看原文
思考过程:1. 分析请求: * 角色:专业的新闻排版工程师。 * 任务:将纯文本新闻正文排版成语义化的HTML片段。 * 规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
            4.  保留原文所有信息,不得增删改内容。
            5.  不要输出``/``/``等文档级标签。
        *   输入文本:
            LLMs are increasingly used to design heuristics for computer systems. However, most synthesis frameworks evaluate candidates by their average performance, which can mask severe regressions on specific workloads. Even adding adversarial training inputs doesn't eliminate these regressions, and percentile objectives (e.g., optimizing for the 95th percentile) overlook how severe regressions can be beyond the selected threshold. We present TAILOR, a framework that guides LLM synthesis using the Conditional Value at Risk (CVaR) of the cost difference between each candidate and an existing baseline on the same workload. This paired CVaR measures the average difference over the worst fraction of workloads. We show that a negative paired CVaR guarantees improvements in both average performance and tail optimality gap relative to the unknown optimum, without ever computing that optimum. We find that TAILOR achieves the best mean--tail trade-off among the synthesis objectives we evaluate across three case studies: Azure GPU batch admission, Azure VM packing, and server placement. Tested across seven diverse optimization benchmarks, TAILOR produces heuristics that are certified with 99% confidence to improve both average performance and tail optimality gaps over the baseline in six of them. On these six benchmarks, TAILOR also outperforms known robust, human-designed alternatives in the tail while achieving better or comparable average-performance improvement over the baseline.
    
    2.  分析输入文本结构:
        *   文本是一段连续的学术/技术新闻摘要。
        *   第1句:背景(LLMs设计启发式)。
        *   第2-3句:当前方法的问题(平均性能掩盖回归,对抗训练和百分位目标的局限)。
        *   第4-5句:提出TAILOR框架及其核心机制(基于CVaR的配对评估)。
        *   第6句:理论保证(负配对CVaR保证平均和尾部性能提升)。
        *   第7-8句:实验结果(三个案例研究,七个基准测试,与人类设计替代方案的比较)。
        *   没有明显的列表、引用或代码块。需要合理划分段落和标题,并对重要概念加粗。
    
    3.  设计HTML结构:
        *   因为是一段连贯的摘要,可以将其分为几个逻辑小节。
        *   `

    `:背景与挑战 (Background and Challenges) -> *等等,规则4说“不得增删改内容”!如果我加了小标题,就属于增加了内容。所以我不能自己编造小标题,除非把现有的句子拆分并作为小标题,但这可能违反“不增删改”。* * 重新审视规则4:“保留原文所有信息,不得增删改内容”。这意味着我不能添加原文中没有的文字(比如自定义的`

    `标题文本)。 * 那怎么处理“长文用`

    `/`

    `划分小节”?如果原文没有标题,我可以用`

    `包裹第一句或者核心概念?或者把原文的某些句子提取出来作为标题? * 更好的做法:原文没有标题,那就只用`

    `分段。如果必须用`

    /

    `,可以把第一句话或者某个核心短语作为标题,但这样可能被视为“修改/增加”。为了安全起见,我会把自然段落用`

    `包裹,将一些核心句子用``加粗。如果确实需要`

    /

    `,我可以从原文中选取完整的句子或短语作为标题,或者干脆不使用`

    /

    `,因为原文只是一段不长的连续文本(不到400词),用`

    `切分即可。 * 等等,规则2说“长文用`

    `/`

    `划分小节”,如果文本不长,可以不用。但我可以把原文按逻辑拆成几个段落,并用`

    `包裹。 * 让我们把原文拆分成几个自然段: * 段1:LLMs are increasingly used... beyond the selected threshold. (背景和问题) * 段2:We present TAILOR... over the worst fraction of workloads. (提出方法和机制) * 段3:We show that a negative... without ever computing that optimum. (理论保证) * 段4:We find that TAILOR achieves... and server placement. (案例研究结果) * 段5:Tested across seven diverse... over the baseline. (基准测试结果) * 为了体现“重要句子可用``”,我可以将包含核心贡献的句子加粗。 * 例如:“We present TAILOR, a framework that guides LLM synthesis using the Conditional Value at Risk (CVaR) of the cost difference between each candidate and an existing baseline on the same workload.” * “We show that a negative paired CVaR guarantees improvements in both average performance and tail optimality gap relative to the unknown optimum, without ever computing that optimum.” * “Tested across seven diverse optimization benchmarks, TAILOR produces heuristics that are certified with 99% confidence to improve both average performance and tail optimality gaps over the baseline in six of them.” 4. 精确排版:

    LLMs are increasingly used to design heuristics for computer systems. However, most synthesis frameworks evaluate candidates by their average performance, which can mask severe regressions on specific workloads. Even adding adversarial training inputs doesn't eliminate these regressions, and percentile objectives (e.g., optimizing for the 95th percentile) overlook how severe regressions can be beyond the selected threshold.

    We present TAILOR, a framework that guides LLM synthesis using the Conditional Value at Risk (CVaR) of the cost difference between each candidate and an existing baseline on the same workload. This paired CVaR measures the average difference over the worst fraction of workloads. We show that a negative paired CVaR guarantees improvements in both average performance and tail optimality gap relative to the unknown optimum, without ever computing that optimum.

    We find that TAILOR achieves the best mean--tail trade-off among the synthesis objectives we evaluate across three case studies: Azure GPU batch admission, Azure VM packing, and server placement. Tested across seven diverse optimization benchmarks, TAILOR produces heuristics that are certified with 99% confidence to improve both average performance and tail optimality gaps over the baseline in six of them. On these six benchmarks, TAILOR also outperforms known robust, human-designed alternatives in the tail while achieving better or comparable average-performance improvement over the baseline.

    检查规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 -> 满足。 2. 每个自然段落用`

    `包裹,长文用`

    `/`

    `划分小节。 -> 文本不长,没有现成的小标题,强行加`

    `会违反规则4(增删内容)。所以只用`

    `。如果非要加`

    `,可以把第一句变成标题?不行,会破坏句子结构。所以只用`

    `是最佳选择,或者把第一段作为引言,但原文没有标题。遵守“不增删改”最高优先级。 3. 列表、引用、代码、加粗。 -> 使用了``。 4. 保留原文所有信息,不得增删改内容。 -> 逐字核对,没有增删改。 5. 不要输出文档级标签。 -> 满足。 5. 最后复核文本内容: 原文: LLMs are increasingly used to design heuristics for computer