首页 > AI前沿 > Omni-JEV: Text and Dual-Stream Speech Decisions

Omni-JEV: Text and Dual-Stream Speech Decisions

arXiv机器学习 2026-11-10 17:15 10 阅读 查看原文

A user utterance does not uniquely determine the next action in full-duplex dialogue: its consequences depend on the assistant's response state and interaction history.

Speech activity, utterance completion, and interruption intent provide related evidence, whose implications for a response require an explicit action policy.

We formulate this task at collected-segment endpoints and introduce Omni-JEV, which maps text or role-indexed user and assistant-reference speech to probabilities over five interaction actions.

An aligned speech-to-speech understanding backbone encodes the acoustic roles separately and shares its candidate readout with text inputs.

Bilingual weak supervision supports cross-entropy fine-tuning followed by reinforcement learning for probability-sensitive decisions (RLCD).

We compare Gaussian-logit and categorical Choice RLCD from the same supervised initialization under matched training exposure, observing different trade-offs between classification and probability quality.

Gaussian Omni-JEV achieves test accuracies of 84.08% for audio and 96.55% for text.

This common action interface supports decisions about when a full-duplex voice assistant should respond, preserve ongoing speech, interrupt, or resume an interrupted response.