首页 > AI前沿 > User-Schedulable Approximations for Quantized Tensor Programs

User-Schedulable Approximations for Quantized Tensor Programs

arXiv机器学习 2026-11-13 05:25 11 阅读 查看原文
用户需要我把给定的英文新闻正文排版成语义化的HTML片段。需要遵循规则:只输出HTML片段本身,不要Markdown围栏、不要解释;每个自然段落用

包裹,长文用

/

划分小节;列表内容用
  • ,引用用
    ,代码用
    ,重要句子可用;保留原文所有信息,不得增删改内容;不要输出//等文档级标签。
    
    正文是关于量化让语言模型在边缘设备上运行,但导致张量内核数量激增。在GGML中,内核是手动编写的,54个权重-激活对中只有8个有分块矩阵乘法内核。他们扩展了Halide的算法和调度分离到值表示。每个近似是一个编解码器,由可重用单元组成。一个新的指令approximate_by应用编解码器到程序的一个阶段,返回编码器和解码器作为普通阶段,让Halide调度。可选契约,如往返律,通过属性测试检查;该律暴露了GGML的q8_1中不一致的块和。一个算法和一个调度库为Arm CPU和Apple GPU上GGML和KleidiAI的31种格式生成347个内核,覆盖38个GGML没有内核的权重-激活对,代码比GGML和KleidiAI少3.7倍。在Apple M3 Max上,内核在91.0%有内核的情况下达到GGML特定格式内核速度的至少95%,中位数加速1.13倍,在GGML回退到分块点积时更快。算上Apple的BLAS库,在CPU情况下仍有86.9%达到该阈值。移植到x86使用Intel的AMX增加了793行代码,在83.5%的情况下与GGML相当。
    
    现在,考虑如何划分小节。这是一篇技术性文章,可以划分为几个小节。原文是一个长段落,但用户要求用

    /

    划分小节。我可以将其分成几个部分:背景与问题、方法、实现与结果等。但需要注意不能增删改内容,只能重新组织结构,用标题分隔。 检查原文是否有自然段落。原文实际上是一个连续的长段落,没有换行。但规则要求每个自然段落用

    包裹,长文用

    /

    划分小节。所以我需要将原文合理分成几个小节,每个小节下用

    包裹内容。 我需要确保不改变内容。原文没有列表、引用、代码,所以不需要使用

      、
      、
      。只有重要句子可以用。我可以给一些关键点加,比如“Quantization lets language models run on edge devices”、“8 of the 54”、“approximate_by”等,但要小心不要过度或改变意思。用户说“重要句子可用”,所以我可以选择几个关键句子加粗。
      
      结构设计:
      - 小节1:背景与问题(Quantization lets language models run on edge devices... only 8 of the 54 weight-activation pairs it runs on our Arm CPU have a tiled matrix-multiplication kernel.)
      - 小节2:方法(We extend Halide's separation of algorithm and schedule... so both decoders fuse into the register tiles of a matrix product.)
      - 小节3:验证与结果(Optional contracts... in 3.7x less code than GGML and KleidiAI.)
      - 小节4:性能(On an Apple M3 Max... matches GGML in 83.5% of its cases.)
      
      但需要确保不增删内容,只是用标题分隔。标题本身是新的,但规则允许用

      /

      划分小节,所以标题是结构的一部分,不是内容增删。但标题不能添加原文没有的信息,只能是概括性的。为了安全,标题要简洁,不添加新信息。 或者,我可以不用太多小节,只分两三个。让我看看原文内容,其实可以分成三段:第一段介绍问题,第二段介绍方法,第三段介绍结果。但原文是一个长句群,我需要用

      包裹每个自然段落,但原文没有自然段落分隔。所以我必须根据内容逻辑来分。 我决定分成三个

      小节: 1. 问题与现状 2. 方法与指令 3. 实现与性能 但标题要中性,不添加新内容。比如:

      Background and Challenge

      Quantization lets... kernel.

      Proposed Approach

      We extend Halide's... matrix product.

      Validation and Results

      Optional contracts... KleidiAI.

      On an Apple M3 Max... cases.

      注意原文最后一句“A port to x86 with Intel's AMX adds 793 lines of code and matches GGML in 83.5% of its cases.”可以放在第三个小节里,或者单独作为一段。我可以把它放在最后一个

      里,和前面性能部分合并,但原文是连续句子,我可以分开。为了清晰,可以将其作为第三个

      。 但需要确保不拆分句子导致信息丢失。可以保持句子完整。原文在“in 3.7x less code than GGML and KleidiAI.”后面是句号,然后“On an Apple M3 Max...”是新句子。所以可以在“KleidiAI.”后断开,用两个

      。 现在,关于:我可以给关键短语加粗,比如“Quantization lets language models run on edge devices”中的“Quantization”,或者整个句子。但用户说“重要句子可用”,所以我可以给整个关键句子加粗,但要注意不要破坏句子结构。例如:

      Quantization lets language models run on edge devices, but it sharply increases the number of tensor kernels a library must maintain: every weight format...

      但这样只加粗了前半句,后半句没加粗。或者只加粗部分。我觉得可以加粗一些核心概念,比如“approximate_by”、“round-trip law”等。但为了简洁,也许不加粗也可以,规则说“可用”,不是必须。为了体现排版,我适当加粗几个重要句子。 让我仔细看原文,确保没有遗漏。原文: "Quantization lets language models run on edge devices, but it sharply increases the number of tensor kernels a library must maintain: every weight format must be paired with every activation format on every instruction set and GPU, with an encoder and a decoder that agree on the stored bytes. In GGML, the tensor library used by llama.cpp, these kernels are written by hand, and only 8 of the 54 weight-activation pairs it runs on our Arm CPU have a tiled matrix-multiplication kernel. We extend Halide's separation of algorithm and schedule to the representation of values. Each approximation is a codec, an encoder paired with a decoder, composed from reusable units: quantizers, integer representations, bit packers and layouts. A new directive, approximate_by, applies a codec to one stage of a program and returns the encoder and decoder as ordinary stages that Halide can schedule, so both decoders fuse into the register tiles of a matrix product. Optional contracts, such as a round-trip law, are checked by property tests; the law exposed an inconsistent block sum in GGML's q8_1. One algorithm and one schedule library generate 347 kernels for 31 of GGML's and KleidiAI's formats on Arm CPUs and Apple GPUs, covering 38 weight-activation pairs for which GGML has no kernel, in 3.7x less code than GGML and KleidiAI. On an Apple M3 Max, our kernels reach at least 95% of the speed of GGML's format-specific kernels in 91.0% of the cases that have one, with a median speedup of 1.13x, and are faster still where GGML falls back to blocked dot products. Counting Apple's BLAS library, which is usually faster for larger f32 products, we still reach that threshold in 86.9% of CPU cases. A port to x86 with Intel's AMX adds 793 lines of code and matches GGML in 83.5% of its cases." 我可以这样划分:

      Background

      Quantization lets language models run on edge devices, but it sharply increases the number of tensor kernels a library must maintain: every weight format must be paired with every activation format on every instruction set and GPU, with an encoder and a decoder that agree on the stored bytes. In GGML, the tensor library used