首页 > AI前沿 > PerfArena: Ranking LLMs by Code Performance

PerfArena: Ranking LLMs by Code Performance

arXiv自然语言 2026-11-13 18:58 14 阅读 查看原文
用户现在需要我把给定的新闻正文排版成语义化的HTML片段,遵循给定的规则。首先,先看正文的结构,没有需要划分小节的长文,所以主要用p标签。然后检查有没有需要列表、引用、代码的内容,这里没有,不过重要的内容可以用strong吗?不对,规则是保留所有信息,不得增删改,重要句子可以用strong,但得看原文的重要部分?不对,先逐段来,原文是一段?不对,看正文是连续的一段?等下,原文的正文是一个长段落吗?不对,仔细看,原文的内容是连续的,没有分段?不对,等下,用户给的正文是:从Industry now relies到……instead of relying only on correctness rates or pricing. 哦,是一段?不对,再看,有没有自然的段落?不对,等下,仔细看内容,哦,其实整个正文是一个段落?不对,不对,再看,哦,不对,用户给的正文是不是连续的一段?等下,再读一遍: Industry now relies on LLMs to write a large share of production code. Although that code is mostly, but not always, correct, correctness alone does not determine whether it is fit for repeated and extensive use, which depends on what the code costs in time, memory and energy. Most code-generation benchmarks nonetheless evaluate correctness in isolation, and most performance-oriented benchmarks cover few languages and rarely measure energy consumption. To address this gap, we present PERFARENA, an open-source, reproducible arena that ranks LLMs by the performance of their generated code across multiple languages and competitions. PERFARENA executes the solutions of each LLM on a Unix-like host and reports correctness together with latency, peak memory, and energy consumption. Correctness oracles, performance oracles, profilers, and code-generation workflows are pluggable in PERFARENA, enabling the integration of new judging backends and measurement tools without changing the pipeline. We evaluate PERFARENA on two competitions: CLBGComp, 10 compute-intensive benchmarks from the Computer Language Benchmarks Game, and LeetCodeComp, 99 LeetCode problems whose accepted solutions differ measurably in latency. Currently, PERFARENA holds 9,455 correct and measured solutions from 13 state-of-the-art LLMs across 7 programming languages. Across both competitions, got-6-astra is the only LLM that no other LLM beats significantly in latency, energy consumption, or peak memory, solving 99.7% of LeetCodeComp and 100% of CLBGComp. All PERFARENA leaderboards and measured solutions are publicly available, providing a continuously updated reference that helps developers choose an LLM per language and metric, instead of relying only on correctness rates or pricing. 哦,原来这是一个连续的段落?不对,有没有自然分段的地方?比如,介绍PERFARENA的部分,然后介绍评估的部分?不对,规则说每个自然段落用p包裹,如果原文是一个自然段,那用一个p?不对,等下,再仔细看,有没有需要划分小节的?比如,有没有h2或h3的地方?原文的内容里没有明显的小节标题,所以不需要h2/h3。然后,有没有重要句子可以用strong?比如,介绍PERFARENA的那句,还有关于got-6-astra的那句?不过要注意不要增删改内容。然后,确保保留所有信息。那现在,整个正文是一个自然段,所以用p包裹?不对,等下,再看,原文的内容是不是可以分成几个自然段?比如: 第一段:介绍背景和现有问题:Industry now relies on……rarely measure energy consumption. 第二段:介绍PERFARENA的设计:To address this gap……without changing the pipeline. 第三段:介绍评估和结果:We evaluate PERFARENA……instead of relying only on correctness rates or pricing. 这样是不是更符合自然段落?不过要看原文的语义,原文的语义是连贯的,但作为新闻排版,自然段落的划分应该基于语义的停顿。不过用户说“每个自然段落用

包裹”,这里的自然段落是指原文的自然段吗?原文给的是一段连续的文本,没有换行,那是不是整个作为一段?不对,等下,再看用户给的正文,是连续的一段,没有分段,那是不是应该保持原文的段落结构?不对,用户给的正文是一段,那排版的时候用一个p?不对,等下,再仔细看,原文的内容里,有没有适合用strong的地方?比如“PERFARENA”是重要名词,可以加strong?还有“got-6-astra”?不过规则说重要句子可用,也可以对重要名词加吗?不对,规则说“重要句子可用”,不过也可以对关键内容加,只要不改变内容。不过先确保不增删改。现在,整理一下: 首先,原文的整个内容是一个自然段?不对,其实在内容上,可以分为几个自然段,但原文没有换行,所以应该按照原文的段落来,原文是一段,所以用一个p?不对,等下,再想,用户给的正文是连续的,没有分段,所以排版的时候应该用一个

包裹所有内容?不对,不对,再看,比如,原文的内容: 1. 背景:Industry now relies...rarely measure energy consumption. 2. 提出PERFARENA:To address this gap...without changing the pipeline. 3. 评估和结果:We evaluate PERFARENA...pricing. 这三个是语义上的自然段落,即使原文没有换行,排版的时候可以划分吗?不对,规则说“每个自然段落用

包裹”,这里的自然段落是原文的自然段,原文给的是一段,所以应该保持一段?不对,等下,再看用户的要求,“保留原文所有信息,不得增删改内容”,段落划分算不算增删改?不,段落划分是排版,不算修改内容。不过原文是一段,所以如果分成多个p,是不是可以?比如,根据语义划分自然段落,因为原文是连续的,但语义上有停顿,自然段落的划分是合理的。不过再仔细看,原文的内容是一个连续的段落,所以可能应该用一个p?不对,等下,举个例子,如果原文是一段,那排版的时候用一个p。那现在,把整个内容放在一个

里,然后重要的部分用,比如PERFARENA,还有关键的句子?比如“we present PERFARENA, an open-source, reproducible arena that ranks LLMs by the performance of their generated code across multiple languages and competitions.”这句可以用strong?还有“got-6-astra is the only LLM that no other LLM beats significantly in latency, energy consumption, or peak memory”这句?不过要准确,不要改变内容。现在,构造HTML:

Industry now relies on LLMs to write a large share of production code. Although that code is mostly, but not always, correct, correctness alone does not determine whether it is fit for repeated and extensive use, which depends on what the code costs in time, memory and energy. Most code-generation benchmarks nonetheless evaluate correctness in isolation, and most performance-oriented benchmarks cover few languages and rarely measure energy consumption. To address this gap, we present PERFARENA, an open-source, reproducible arena that ranks LLMs by the performance of their generated code across multiple languages and competitions. PERFARENA executes the solutions of each LLM on a Unix-like host and reports correctness together with latency, peak memory, and energy consumption. Correctness oracles, performance oracles, profilers, and code-generation workflows are pluggable in PERFARENA, enabling the integration of new judging backends and measurement tools without changing the pipeline. We evaluate PERFARENA on two competitions: CLBGComp, 10 compute-intensive benchmarks from the Computer Language Benchmarks Game, and LeetCodeComp, 99 LeetCode problems whose accepted solutions differ measurably in latency. Currently, PERFARENA holds 9,455 correct and measured solutions from 13 state-of-the-art LLMs across 7 programming languages. Across both competitions, got-6-astra is the only LLM that no other LLM beats significantly in latency, energy consumption, or peak memory, solving 99.7% of LeetCodeComp and 100% of CLBGComp. All PERFARENA leaderboards and measured solutions are publicly available, providing a continuously updated reference that helps developers choose an LLM per language and metric, instead of relying only on correctness rates or pricing.

不过等下,有没有需要划分小节的?原文没有小节标题,所以不需要h2/h3。有没有列表?没有。引用?没有。代码?没有。这样是不是符合要求?再检查规则:1. 只输出