首页 > AI前沿 > How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys

How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys

arXiv机器学习 2026-10-08 04:55 6 阅读 查看原文

Speech quality predictors are increasingly used as rewards, yet no agreed measure of their hackability exists.

The usual measurement has two flaws. First, the perturbation reaches the predictor through a processing chain -- here a neural codec -- that shifts the score on its own, which scoring against the raw input charges to the attack. Referencing the unperturbed round trip instead changes measured hackability by up to a factor of four (0.31 to 0.08 for one defence).

Second, one trained attacker is a sample, not a measurement: five attackers differing only in random seed reach success rates from 0.00 to 0.38 against one fixed predictor, so a defence claim needs the worst case over several.

Under this protocol, four published predictors differ widely: NISQA is hacked on 90% of utterances, SSL-MOS on 21%, DNSMOS on 14% and UTMOS on 6%.

We then audit a closed attack-detect-patch loop. It hardens the predictor only in its own attack space, by less than the spread between attackers; a random-perturbation baseline matches it; and it costs up to 0.30 system SRCC out of domain.

Enhancers post-trained against patched predictors hack them far less (PESQ -0.03 versus -0.23).

Code, preregistration and run outputs are released.