Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents
cs.CR · cs.LG
Quick verdict
声称工具日志污染了开源权重 VLM 判别器（78–90% 的误判）；受控且具说服力，但仅限于 Qwen 模型，无置信区间（CIs），且前沿子集规模较小。
Summary
本文探讨了一个问题：视频生成 Agent 提供给其多模态判别器（multimodal judge）的辅助文本——包括执行轨迹（execution trace）、合成的旁白（narration）以及硬编码字幕（burned-in subtitles）——是否会改变其对纯视觉要求的判定结论。作者在保持帧（frames）不变的情况下，仅对文本进行修改，并在 109 个人工标注的双事件生成片段（clips）上进行测试（其中 50 个片段显示请求的事件存在，59 个片段为近似失败（near-miss failures）），并使用 UCF101 作为对照集。报告成功的执行日志使得开源权重的 Qwen 判别器对失败片段的误接受率从 7–19% 上升到 78–90%；Qwen3-VL-32B 对旁白和字幕具有鲁棒性，但仍然信任工具日志（tool log）；而在 Qwen3 判别器中，冲突日志将正确片段的接受率降低到 0.00–0.08。前沿闭源判别器（GPT-5.4-mini 在所有 109 个片段上进行测试；GPT-5.5、Claude Opus 5 和 DeepSeek-V4 在较小的子集上进行测试）基本保持不变，并且“忽略文本”的指令仍导致 0.41–0.63 的误接受率。在对 36 个标注失败片段进行的模拟修复循环中，一个诚实的 LLM planner 最终判别通过率为 1.00，但真实通过率仅为 0.28；将一个低成本 checker 的判定结果写入轨迹，会使 32B 判别器的误接受率从 0.25 上升到 0.69。所提出的最低权限判别（least-privilege judging）将真实通过率恢复到 0.86–0.92，代价是成本增加约三倍。局限性：仅使用了一个生成器（generator）和一个开源判别器家族，仅由作者进行标注，没有置信区间（confidence intervals），且前沿判别器的子集较小。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A supportive execution log makes open-weight Qwen-VL judges (7B/8B/32B) accept 0.78/0.90/0.83 of generated clips where the requested event is visibly missing, up from 0.17/0.19/0.07 without text (Table 1).
- Frontier closed judges (GPT-5.4-mini, GPT-5.5, Claude Opus 5, DeepSeek-V4) are essentially unmoved on the same clips, so the vulnerability is a learned-trust property, not a property of the task.
- In a simulated repair loop, the round-0 false-accept rate α0 caps the true pass rate of any repair policy at 1 − α0; an honest LLM planner ends at judge pass 1.00 but true pass 0.28 (Tables 3–4).
- Writing a cheap checker's verdict into the trace makes a stronger judge inherit its errors: the 32B judge's round-0 false acceptance rises from 0.25 to 0.69, implying it takes the checker's word 92% of the time.
- Least-privilege judging — routing each requirement only its declared evidence type, with OCR masking of burned-in text — restores true pass rates to 0.86–0.92 at roughly tripled cost and leaves process-level checks intact.
Strengths
Tight causal isolation
Frames are held fixed while exactly one auxiliary stream is varied, in both support and contradict directions (§3, Figure 1, Table 1). The UCF101 control shows the predicted asymmetry: supporting logs barely create false accepts (≤ +0.20; ~+0.05–0.07 for Qwen3, Table 8) while contradicting logs collapse accuracy to roughly chance (§5).
Plan- vs artifact-derived distinction, demonstrated plan-only
§3 separates text that cannot carry clip-specific information (plan, status lines) from checker output that can. The §6 loop's initial trace contains only plan plus status: success, yet 72% of failed clips are accepted at round 0 by the 8B judge — contamination without any false claim or adversarial agent.
Deterministic accounting identity as sanity check
Remedy evaluated on benefits, costs, and blind spots
Unusually complete reproducibility specification
Weaknesses
No uncertainty quantification
Single open-weight family
Author-only labels with exclusion of borderline clips
Log channel conflates text kinds in the headline audit
Closed-loop claims come from a simulation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks one narrow, well-posed question: when a requirement is purely visual (“she drinks from the mug”), does the auxiliary text a video-generation agent hands its multimodal judge — execution trace, synthesized narration, burned-in subtitles — change the verdict if the frames are held fixed? It claims (i) open-weight Qwen-VL judges at 7B/8B/32B are heavily contaminated by tool logs: false accepts on failure clips rise from 0.17/0.19/0.07 to 0.78/0.90/0.83 (Table 1); (ii) frontier closed judges are essentially unmoved on the same clips, so the vulnerability is learned trust in tool logs, not task difficulty; (iii) “ignore the text” prompting fails, leaving 0.41–0.63 false accepts; (iv) in a repair loop the round-0 false-accept rate α0 caps true pass rate at 1 − α0 (Table 4), so honest agents still certify 72% of failures with the 8B judge; (v) a cheap checker's verdict injected into the trace propagates to a stronger judge (32B round-0 0.25 → 0.69, implied inheritance p = 0.92); (vi) least-privilege judging — routing each requirement only its declared evidence type, with OCR masking of burned-in text — restores true pass to 0.86/0.92 at ~3× cost and leaves process checks intact (Tables 3, 5, 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper 'Don't Read the Log' show?
It shows that when a video-generation agent's multimodal judge is also given the agent's execution trace, the trace can override what the frames actually show: open-weight Qwen-VL judges accepted 78–90% of clips where a requested event was visibly missing once a log reported a successful tool call, versus 7–19% without any text.
Which video judges are vulnerable to execution-trace contamination?
In this study, the open-weight Qwen-VL judges at 7B, 8B and 32B parameters. Frontier closed judges (GPT-5.4-mini, GPT-5.5, Claude Opus 5, DeepSeek-V4) barely moved on the same clips. The authors attribute the difference to what each judge learned to trust during training, not to task difficulty — the closed judges' visual baselines were no better.
What is least-privilege judging?
A structural fix proposed in the paper: every requirement declares the evidence type that can establish it (visual, process, audio), and the judge receives only that evidence — frames for visual checks, the execution trace for process checks — with OCR-based masking of text burned into the frames. In the paper's repair-loop simulation it restored true pass rates to 0.86–0.92 at about three times the cost, because failed clips are then actually regenerated.
Does telling the judge to ignore the log fix the problem?
No. A visual-only system prompt reduced false accepts under a supportive log from 0.78/0.90/0.83 to 0.63/0.54/0.41 for the three Qwen judges — still three to six times the no-text baseline — because the instruction competes with the very text it asks the model to ignore. The authors argue a structural constraint (routing evidence by requirement type) is needed instead.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as a text dump in the extraction; Figure 3/4 visual content could not be inspected, though Figure 1's values cross-check against Table 1.
- Code, clips, labels and per-call outputs are promised ('will be released') but were not available at review time, so codeAvailable is false and reproducibility is rated contingent on that release.
- The extracted text appears complete through references and appendices; no truncation detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.