首页 > AI前沿 > Low-Latency LLM-Based Streaming Multi-Talker ASR with Verifiable Rewards

Low-Latency LLM-Based Streaming Multi-Talker ASR with Verifiable Rewards

arXiv自然语言 2026-11-14 11:56 14 阅读 查看原文
用户现在需要把给定的新闻正文排版成语义化HTML片段,按照要求来。首先,先看内容,整体是一段新闻正文,没有小节划分?不对,先理清楚结构:首先开头的段落,先检查有没有需要分的,哦原文是连贯的一段?不对,原文的内容:首先第一句到后面的结果,是不是一个自然段?不对,再仔细看,原文的整个内容是一个连贯的段落?不对,等下,看原文的结构:首先介绍背景,然后问题,然后方法,然后实验结果,然后结论。哦,不对,用户要求每个自然段落用p包裹,长文用h2/h3划分,那这里是不是可以划分小节?不对,先看原文,原文的内容:首先第一部分是背景:LLM-based...latency challenges. 我们的可穿戴场景...each speaker tag must precede its turn. 然后是问题:These sparse tags...delay the text that follows. 然后是方法:We therefore post-train...onset-delay reward. 然后是实验结果:On an eight-direction...12.70% SA-WER. 然后是结论:These results demonstrate...jointly. 不对,不对,用户要求保留原文所有信息,不得增删改,那不能乱加小节标题?哦不对,原文里没有小节标题啊,那是不是整个原文是一个自然段?不对,等下,原文的内容是连续的,没有分段?不对,看用户给的正文: 正文是: LLM-based architectures offer strong generalization and instruction-following capabilities for streaming automatic speech recognition (ASR), but multi-talker use introduces coupled transcription, attribution, and latency challenges. Our wearable setting requires real-time transcription of a conversation between a near-field device wearer and a far-field partner. Because partial hypotheses are rendered on persistent sides of a dialogue UI, speaker roles cannot be permuted and each speaker tag must precede its turn. These sparse tags are weakly optimized by supervised fine-tuning (SFT), while committing to a leading tag can delay the text that follows. We therefore post-train a streaming LLM-based ASR model with reinforcement learning with verifiable rewards (RLVR), combining a fixed-role Speaker-Attributed WER (SA-WER) grader with a turn-level onset-delay reward. On an eight-direction multi-talker cohort, SA-WER RL reduces SA-WER from $20.51\%$ for SFT to $13.14\%$. Adding the delay objective reduces onset $p50$ by $65\%$ (from $0.918\,\text{s}$ to $0.320\,\text{s}$) and $p90$ by $44\%$ (from $1.389\,\text{s}$ to $0.780\,\text{s}$) relative to SA-WER-only RL, while achieving $12.70\%$ SA-WER. These results demonstrate that fixed-role recognition quality and streaming responsiveness can be improved jointly. 哦,原来整个正文是一个完整的自然段?不对,不对,等下,有没有需要划分的?不对,用户要求如果长文用h2/h3,但是原文没有小节标题,我们不能自己加啊,因为不得增删改内容。那首先,重要的句子可以用strong,比如开头的LLM-based architectures的那句?不对,或者方法部分,结果部分?等下,先按照要求:每个自然段落用p包裹,这里原文是一段?不对,等下,再仔细看,原文的内容是不是可以分成几个自然段?不对,用户给的正文是连续的,没有分段,那是不是整个用一个p?不对,不对,等下,再看规则:保留原文所有信息,不得增删改内容,所以不能自己加分段?不对,不对,原文的标点是连续的,那整个就是一个自然段?不对,等下,再想,比如,有没有需要用strong的地方?比如核心结论?或者关键的数字?比如SA-WER从20.51%降到13.14%,还有延迟的减少,还有最后的结论,可以用strong。然后,有没有代码?没有。列表?没有。引用?没有。那排版的话,首先,整个原文是一个自然段,用p包裹,然后把重要的句子加strong?比如:

LLM-based architectures offer strong generalization and instruction-following capabilities for streaming automatic speech recognition (ASR), but multi-talker use introduces coupled transcription, attribution, and latency challenges. Our wearable setting requires real-time transcription of a conversation between a near-field device wearer and a far-field partner. Because partial hypotheses are rendered on persistent sides of a dialogue UI, speaker roles cannot be permuted and each speaker tag must precede its turn. These sparse tags are weakly optimized by supervised fine-tuning (SFT), while committing to a leading tag can delay the text that follows. We therefore post-train a streaming LLM-based ASR model with reinforcement learning with verifiable rewards (RLVR), combining a fixed-role Speaker-Attributed WER (SA-WER) grader with a turn-level onset-delay reward. On an eight-direction multi-talker cohort, SA-WER RL reduces SA-WER from $20.51\%$ for SFT to $13.14\%$. Adding the delay objective reduces onset $p50$ by $65\%$ (from $0.918\,\text{s}$ to $0.320\,\text{s}$) and $p90$ by $44\%$ (from $1.389\,\text{s}$ to $0.780\,\text{s}$) relative to SA-WER-only RL, while achieving $12.70\%$ SA-WER. These results demonstrate that fixed-role recognition quality and streaming responsiveness can be improved jointly.

不对,等下,有没有需要分小节的?比如,原文没有小节标题,我们不能自己加h2/h3,因为那样会增加内容,违反不得增删改的规则。对,因为h2/h3是标题,原文没有,所以不能加。那整个就是一个p,里面重要的部分用strong。等下,再检查有没有其他需要处理的,比如公式,保留原样就好,用户要求保留原文所有信息,所以$20.51\%$这些直接保留。对。那这样的话,输出就是这个p标签包裹的内容,里面的关键句子加strong。等下,再确认规则:只输出HTML片段,不要其他,对。那现在整理一下: 首先,开头的背景句是重要的,加strong。然后实验结果的核心句子加strong,最后结论加strong。对吗?或者有没有其他重要的?比如方法部分?比如We therefore post-train...那个句子要不要加?也可以?不过不要太多。等下,再想,原文的结构: 1. 背景与问题:LLM-based...delay the text that follows. 2. 方法:We therefore...onset-delay reward. 3. 实验结果:On an eight-direction...12.70% SA-WER. 4. 结论:These results...jointly. 那是不是可以把这些作为不同的段落?不对,原文是连续的,用户给的正文没有分段,所以我们不能自己拆分段落,因为那样会改变原文的分段?不对,等下,用户说“每个自然段落用

包裹”,原文的自然段落就是一个,所以用一个p。然后重要的句子用strong。那现在写出来:

LLM-based architectures offer strong generalization and instruction-following capabilities for streaming automatic speech recognition (ASR), but multi-talker use introduces coupled transcription, attribution, and latency challenges. Our wearable setting requires real-time transcription of a conversation between a near-field device wearer and a far-field partner. Because partial hypotheses are rendered on persistent sides of a dialogue UI, speaker roles cannot be permuted and each speaker tag must precede its turn. These sparse tags are weakly optimized by supervised fine-tuning (SFT), while committing to a leading tag can delay the text that follows. We therefore post-train a streaming LLM-based ASR model with reinforcement learning with verifiable rewards (RLVR), combining a fixed-role Speaker-Attributed WER (SA-WER) grader with a turn-level onset-delay reward. On an eight-direction multi-talker cohort, SA-WER RL reduces SA-WER from $20.51\%$ for SFT to $13.14\%$. Adding the delay objective reduces onset $p50$ by $65\%$ (from $0.918\,\text{s}$ to $0.320\,\text{s}$) and $p90$ by $44\%$ (from $1.389\,\text{s}$ to