首页 > AI前沿 > Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents

Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents

arXiv机器学习 2026-09-23 18:53 6 阅读 查看原文

Agentic video-generation systems close a loop between a generator and a verifier: an LLM plans shots, calls a text-to-video model, and a multimodal judge decides whether the result satisfies the request.

To diagnose where a long workflow fails, recent harnesses deliberately show the judge more than the video—the agent's execution trace, its plan, the narration it synthesized.

We ask whether this auxiliary text moves the judge's verdict on purely \emph{visual} requirements, holding the frames fixed.

On a benchmark of 109 generated two-event clips with manual labels, in which the requested event is either visibly completed or visibly missing, a trace that reports a successful tool call makes three open-weight Qwen-VL judges (7B, 8B, 32B) accept $78$--$90\%$ of the failures, up from $7$--$19\%$ without text, and a contradicting trace makes them reject up to $100\%$ of correct clips; an instruction to ``use only the frames'' does not remove the effect.

Frontier closed judges are essentially unmoved on the same clips, showing that the vulnerability is a property of the judge's learned trust in tool logs rather than of the task.

Plan-derived text carries no clip-specific information, so it can only shift a judge's operating point, and in a repair loop that shift becomes a cap on the true pass rate that no repair policy can exceed; the cap matches simulation to two decimals.

In the loop, contamination is exploited without any adversarial agent: an honest LLM planner that always regenerates ends with a judge pass rate of $1.00$ and a human-labelled pass rate of $0.28$, and a pipeline in which a cheap checker writes its verdict into the trace launders that checker's errors into a stronger final judge ($0.69$ false accepts).