首页 > AI前沿 > Beyond Word Error Rate: A Failure Taxonomy of Streaming Dictation ASR

Beyond Word Error Rate: A Failure Taxonomy of Streaming Dictation ASR

arXiv自然语言 2026-11-12 23:09 20 阅读 查看原文
用户现在需要把给定的新闻正文排版成语义化HTML片段,首先先理清楚内容的结构。首先整体是一个关于ASR系统评估的内容,有没有需要分小节的?看一下内容,首先开头讲传统比较方式和dictation场景的不同,然后是测试集,然后错误分类,然后是不同系统的表现,最后是说明和资源。不对,先逐段,首先第一部分是开头的对比:Speech recognition那两句,然后是测试集的内容,然后错误分类,然后各个系统的表现,然后最后是说明。等下,先划分,首先有没有合适的h2或者h3?比如首先可以有h2讲评估背景?不对,先别乱加,要保留内容,不增删。首先第一个自然段:前两句是讲传统ASR比较方式和dictation场景的差异,对吧?然后下一部分是测试集的介绍,然后错误分类,然后不同系统的WER表现、漏词、多语言的情况,然后是Whisper的测试,最后是说明和资源。 等下,先逐段来,首先第一个自然段:

Speech recognition (ASR) systems are usually compared by word error rate (WER) on recorded files, after punctuation, capital letters and number format are removed. In dictation, the audio is streamed live and the text goes into the user's message as written, so what matters is which mistakes the user has to fix.

对,这里重要的句子可以加strong。然后接下来是测试集的内容,这部分可以加个h3?比如

测试数据集

?不对,不对,等下,用户说长文用h2/h3划分小节,但是不要增删内容,哦不对,标题是不是新增的?不对不对,等下规则是保留原文所有信息,不得增删改内容,哦,那小节标题如果原文没有的话是不是不能加?哦对哦,规则4说不得增删改内容,那不能自己加标题文字对吧?哦我刚才没注意,那不能随便加h2h3的文本,除非原文有?不对等下看原文有没有可以作为标题的?不对,原文都是正文,那如果不需要分小节的话?不对再看内容,有没有逻辑分段? 哦不对,再仔细看,原文的所有句子,我们可以把逻辑上的部分用小节吗?不对,不行,因为不能加内容,那h2/h3的内容必须是原文里的?不对,不对,再看规则2:“每个自然段落用

包裹,长文用

/

划分小节”,哦,划分小节的话,是结构上的,但是不能加原文没有的文字对吧?那是不是如果原文没有小标题,就不需要硬加?不对不对,等下再看内容,比如错误分类那部分,还有测试集,还有测试结果?不对,不对,原文没有标题,那不能自己编标题,不然就是增加内容了,违反规则4。那那我就按自然段落?不对,原文的段落是怎么分的?原文是一整段?不,用户给的正文是一整段?不对看用户给的正文:从Speech recognition到every transcript are released. 是一整块?不对,我们可以按语义分段对吧?因为用户说每个自然段落用p包裹,那把语义连贯的句子组成段落,对,因为原文是纯文本的一段,我们可以按语义拆分段落,但是不能改内容。 等下重新梳理内容的逻辑层次: 1. 开篇介绍ASR系统的常规评估维度,以及实时听写场景下评估重点的差异:这是第一部分,用p,其中“so what matters is which mistakes the user has to fix”可以加strong,因为是核心结论。 2. 接下来是测试集的构成:这部分内容是“The test set, streamed to eleven ASR systems, comprises 106 scripted English dictations from one speaker (read in a quiet room, with a fan on, and whispered), 20 clips in each of eleven other language and dialect sets, and 400 recordings that join clips in two languages.” 这部分可以作为第二个段落?或者要不要加小节?不对,不能加标题文字,所以不用h标签?不对,等下如果长文的话,划分小节,但是不能加内容,那是不是可以用h2/h3来包裹原文里的核心主题句?不对,不对,h2/h3是标题,要有文本,文本必须是原文的,不能自己加。哦,哦对哦,我刚才搞错了,不能自己写标题,不然就是增内容了。那如果原文没有明确的小标题,那我就不用h2h3?不对,再看用户的规则2是“长文用

/

划分小节”,是可选?还是必须?不对,再看内容,有没有可以作为小节标题的原文句子?比如“Every error was sorted by a fixed rule into four levels: recognition, output formatting, language handling and, for the self-hosted models, audio dropped before the model.” 不对,这是陈述句,不是标题。哦,那可能不需要加h标签?不对,等下再仔细看用户的要求,不要增删改内容,所以绝对不能加原文没有的文字,包括标题。那我就按语义拆分段落,每个段落用p,有需要加strong的地方加,列表的话这里有没有列表?哦原文里的错误分类是四个,哦不对,原文是“four levels: recognition, output formatting, language handling and, for the self-hosted models, audio dropped before the model.” 这个是可以用ul吗?不对,原文是冒号列举,我要不要改成列表?规则3说列表内容用ul li,那如果原文是列举的话,可以转成列表吗?等下规则4是保留原文所有信息,不得增删改,那把冒号的列举改成列表,是不是可以?因为是排版,内容没改。等下看:“Every error was sorted by a fixed rule into four levels: recognition, output formatting, language handling and, for the self-hosted models, audio dropped before the model.” 这里的四个类别,我可以把它拆成列表吗?不对,原文是连续的句子,我如果把这部分改成:Every error was sorted by a fixed rule into four levels:
  • recognition
  • output formatting
  • language handling
  • for the self-hosted models, audio dropped before the model
?不对,这样的话是不是改变了句子结构?哦不对,可能不合适,还是保持句子连贯,不要拆成列表,因为没有明显的列表结构,原文是用冒号列举的,所以不用ul。 然后接下来是各个系统的测试结果,这部分是连续的句子,可以按逻辑分段,比如: 第一段(背景):

Speech recognition (ASR) systems are usually compared by word error rate (WER) on recorded files, after punctuation, capital letters and number format are removed. In dictation, the audio is streamed live and the text goes into the user's message as written, so what matters is which mistakes the user has to fix.

第二段(测试集):

The test set, streamed to eleven ASR systems, comprises 106 scripted English dictations from one speaker (read in a quiet room, with a fan on, and whispered), 20 clips in each of eleven other language and dialect sets, and 400 recordings that join clips in two languages.

第三段(错误分类):

Every error was sorted by a fixed rule into four levels: recognition, output formatting, language handling and, for the self-hosted models, audio dropped before the model.

然后接下来是测试结果部分,这部分有几个不同的维度,可以拆分: 第四段(WER对比与 whispered 漏词情况):

Systems with similar WER failed in different ways. Zoom Scribe and OpenAI's GPT-Live-Transcribe have almost the same WER on quiet English (7.4% and 7.3%), but 34.7% and 8.8% on whispered English. Zoom and both AssemblyAI models left out 26-28% of whispered words, and still 21-26% with their end-of-turn detection relaxed or off; seven other systems left out at most 3%. Qwen3-ASR's 32% came from the speech detection in our own deployment (6% when relaxed). Across 107 EARS speakers, the same four systems lost 7-31% of whispered words, while six others lost under 2%.

第五段(多语言场景表现):

With only English selected, Gemini, Deepgram and our self-hosted Whisper lost the non-English part of a two-language recording in 86-99% of recordings, against 33-46% with both languages selected, while GPT-Live-Transcribe, OpenAI's realtime Whisper and Meta lost it in at most 20%.

第六段(Whisper搭配VAD的表现):

Running Whisper behind a Silero voice activity detector that drops audio judged to be silence raised whispered WER from 10.1% to 51.