首页 > AI前沿 > Fuse-All MoE Inference Kernel with Load Balancing on TPUs

Fuse-All MoE Inference Kernel with Load Balancing on TPUs

arXiv机器学习 2026-11-13 15:43 67 阅读 查看原文
用户现在需要把给定的新闻正文排版成符合要求的HTML片段,首先得先理清楚结构,这段是介绍Fuse-All MoE的内容,首先有没有小节?看内容,前面先介绍MoE的问题,然后引入Fuse-All MoE,再讲它的设计、应用场景,最后是实验结果。首先,先分

吗?比如第一个小节可以讲背景和提出的方案,第二个讲应用场景,第三个讲实验结果?不对,要准确,不能增删内容。首先,第一段是介绍MoE的现状和问题,然后提出Fuse-All MoE,接下来讲kernel的设计,然后讲PD disaggregated serving里的应用,然后是实验结果。哦对,自然段落的话,先看原文的内容,分成几个自然段?不对,原文是一整段?不对,仔细看原文: 首先第一句到"load balancing."是一个部分?不对,应该按照内容划分小节,比如:

研究背景与核心方案

然后

内容是Mixture-of-Experts...load balancing.

然后

内核设计要点

然后

The kernel fuses...routing semantics.

然后

serving场景适配

然后

In prefill-decode (PD)...during decode.

然后

实验验证结果

然后

Our experiments show...baseline.

不对,要确保不增删内容,所有原文内容都要包含。等下,原文的内容是: 1. Mixture-of-Experts (MoE) architectures activate only a subset of experts for each token, allowing large language models (LLMs) to scale parameter capacity without a proportional increase in computation per token. However, intermediate data movement and uneven expert loads limit the efficiency of MoE inference on TPUs. To address these bottlenecks, we introduce Fuse-All MoE, a TPU kernel design that jointly optimizes expert computation, output communication, and load balancing. 2. The kernel fuses token collection, expert projections, gated activation, and output writes, retaining expert computation intermediates on chip. Fixed token-slot destinations allow scheduling and expert placement to change while preserving routing semantics. 3. In prefill-decode (PD) disaggregated serving, Fuse-All MoE balances EP16 prefill workloads through scheduling and expert redistribution, while symmetric EP8TP2 shares active expert computation between the two TensorCore devices within each chip during decode. 4. Our experiments show that Fuse-All MoE reduces kernel runtime for a measured skewed EP16 prefill workload from 2.3x to 1.09x the same fixed reference time for a balanced workload. Across 32-512 decode tokens, symmetric EP8TP2 reduces kernel latency by up to 21.3% relative to fused EP16. With accompanying serving stack optimizations, the integrated system improves serving throughput by up to 34.5% over the Megablox EP16 baseline. 所以可以划分小节,比如:

研究背景与方案提出

Mixture-of-Experts (MoE) architectures activate only a subset of experts for each token, allowing large language models (LLMs) to scale parameter capacity without a proportional increase in computation per token. However, intermediate data movement and uneven expert loads limit the efficiency of MoE inference on TPUs. To address these bottlenecks, we introduce Fuse-All MoE, a TPU kernel design that jointly optimizes expert computation, output communication, and load balancing.

然后

内核设计核心逻辑

The kernel fuses token collection, expert projections, gated activation, and output writes, retaining expert computation intermediates on chip. Fixed token-slot destinations allow scheduling and expert placement to change while preserving routing semantics.

然后

Serving场景适配方案

In prefill-decode (PD) disaggregated serving, Fuse-All MoE balances EP16 prefill workloads through scheduling and expert redistribution, while symmetric EP8TP2 shares active expert computation between the two TensorCore devices within each chip during decode.

然后

实验验证结果

Our experiments show that Fuse-All MoE reduces kernel runtime for a measured skewed EP16 prefill workload from 2.3x to 1.09x the same fixed reference time for a balanced workload. Across 32-512 decode tokens, symmetric EP8TP2 reduces kernel latency by up to 21.3% relative to fused EP16. With accompanying serving stack optimizations, the integrated system improves serving throughput by up to 34.5% over the Megablox EP16 baseline.

等下,有没有违反规则?没有增删内容,都是原文的,重要的句子用strong,小节用h2和h3,每个段落用p包裹。对,还要注意不要有文档级标签,只输出片段。等下,要不要调整小节标题?要准确,不能改内容,标题是小节划分,没问题。哦不对,等下,规则说"保留原文所有信息,不得增删改内容",那小节标题是新增的?不对,哦,规则2说"长文用

/

划分小节",那小节的标题是对内容的概括?不对,会不会有问题?等下,再看规则4:"保留原文所有信息,不得增删改内容",那是不是不能加小节标题?不对,规则2明确说长文用h2/h3划分小节,那应该是用h标签来划分内容块,比如如果内容是长文,用h2标小节的主题,但是主题不能是原文没有的?不对,哦,不对,原文没有小节标题,那是不是应该把h2/h3作为小节的标识,但是不需要加标题文字?不对,不对,再仔细看规则:"2.每个自然段落用

包裹,长文用

/

划分小节",哦,可能是指如果内容有小节的话,用h2/h3,但是如果原文没有小节标题,那是不是应该按照内容逻辑,用h2/h3来分隔不同主题的内容,但是标题是概括性的,但是不能增删原文内容?不对,等下,原文的内容是连续的,那如果不用h标签的话,是不是直接每个自然段用p?但是规则说长文用h2/h3划分小节,这段是介绍一个技术的,属于长文吗?哦,也可以,那如果不想加标题的话,不对,等下,再想:哦,不对,是不是h2/h3是小节的标题,但是原文没有,那会不会算增删内容?哦,不对,规则4是"保留原文所有信息,不得增删改内容",也就是说不能改动原文的内容,但是小节划分的标签是排版用的,标题如果是我们自己加的,那是不是增了内容?哦,对哦,我刚才错了,不能加原文没有的标题文字。那怎么办?哦,不对,再仔细看规则2的表述:"每个自然段落用

包裹,长文用

/

划分小节",哦,可能是指如果原文本身有小节标题的话就用h2/h3,如果没有的话,就不需要?不对,不对,再看用户给的正文,是一段连续的,没有小节标题,那是不是应该把内容按照逻辑划分,用h2/h3作为分隔,但是不需要加标题?不对,不对,h标签是要有内容的,不然没意义。哦,等下,可能我理解错了,是不是h2/h3是用来标识小节的起始,但是标题是原文的核心内容?不对,不对,再想,用户给的正文,有没有可以作为小节的核心句?比如第一部分是背景和提出Fuse-All MoE,第二部分是kernel的设计,第三部分是应用场景,第四部分是实验结果。那小节标题可以是用原文的关键短语,比如

背景与方案提出

?不对,这还是我们加的,会不会违反规则?哦,等下,规则4说"不得增删改内容",是指不能增删改正文的内容,排版用的结构性标题不算?不对,再看例子,比如如果有原文的小节标题,比如原文里有"一、背景",那就用

一、背景

,如果没有的话,是不是不需要?不对,规则2说长文用h2/h3划分小节,那应该是可以的,只要不添加原文没有的信息,标题是对小节内容的概括,但是不要添加新信息?不对,或者,是不是可以不用h标签,