Verify, Don't Trust: Agentic Model Development for Video Discovery Retrieval at Scale
Hao Fu, Baiting Zhu, Minglei Chen, Yinjie Huang, Shuai Ding, Shuai Ding
cs.IR · cs.AI · cs.LG
Quick verdict
Overturns a false -22pp attribution via gated agentic verification (+3.20pp matched, +0.66% GSRR online); credible single-site, replay-only guard validation.
Summary
The paper presents EvoPilot, a human-gated system for long-horizon 'online autoresearch,' in which LLM agents plan and execute model-development experiments on Meta's production Video Deep Dive (VDD) retrieval system over hundreds of millions of videos. The core thesis is that a completed run is not valid evidence: only control-treatment pairs whose realized execution matches a human-approved protocol and passes deterministic, artifact-backed verification are admitted (Eq. 3). Over a 37-day campaign spanning seven directions and 20 approved rounds, a primitive autoresearch attempt had wrongly attributed a 22-percentage-point offline hit-rate drop to an interaction head; EvoPilot's gated review traced it to an evaluation defect (control realized K=3000, treatment K=600). After repair, a matched-lineage comparison showed +3.20pp, and the selected model later delivered +0.66% relative GSRR in a 7-day randomized A/B. Post-study replay/mutation tests admitted all 10 clean fixtures and blocked all 10 faults. Notably honest about when each control existed, the paper still rests on a single proprietary system; operational rates come from only 9/20 record-complete rounds; key identity guards have replay-only validation; offline results lack confidence intervals; A/B statistics are unreported; and guardrail outcomes were unavailable.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A completed experiment run is not valid evidence; the unit of scientific evidence is a control-treatment comparison whose realized execution matches a human-approved protocol, verified against terminal artifacts by deterministic checks plus human review.
- EvoPilot's human-gated verification overturned an incorrect -22pp attribution of an offline hit-rate decline to the interaction head by tracing it to an evaluation defect (control realized K_eff=3000 while treatment realized K_eff=600 despite both declaring K=600).
- After repair, a matched-lineage head-only comparison improved offline click hit-rate@3000 by 3.20pp, and the selected model delivered a +0.66% relative GSRR increase in a 7-day randomized online A/B.
- Post-study paired replay/mutation tests admitted all 10 clean fixtures and blocked all 10 planted faults, including the historical funnel mismatch.
- Stored lessons and memory must remain advisory until compiled into executable checks over artifacts; durable state enabled recovery of interrupted rounds and artifact reuse avoiding ~5 GPU-hours of redundant publication.
Strengths
Honest control-evolution accounting
Section 4.6 explicitly refuses to credit a control with preventing the incident that motivated it; Table 3 records each control's introduction round, and the funnel/evaluator-identity gates are plainly labeled post-study, replay-only (Sec 5.2). The headline -22pp catch is correctly attributed to the human gate, not the automated comparator added later.
Rigorous incident forensics with a causal probe
Section 5.3's extreme-K falsification test is genuinely persuasive: pre-repair, declaring K=1 moved the two-tower hit-rate only ~0.5pp (proving K_eff stayed 3000); post-repair, the K=1 reading collapsed to near zero, demonstrating both defect and fix.
Clean evidence taxonomy
Estimand hygiene in the flagship result
Implementable protocol specification
Weaknesses
Single proprietary deployment, no concurrent baseline
Flagship guards validated only on curated fixtures
Thin inferential reporting
Operational statistics rest on 9/20 rounds
Efficiency claims are anecdotal
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This Meta industrial paper reports EvoPilot, a human-gated control layer for LLM-agent-driven model development ('online autoresearch') on a production retrieval system: Video Deep Dive (VDD), a follow-on video discovery experience backed by an hourly refreshed index of hundreds of millions of videos. The central claim (Secs. 1, 4.2) is epistemic: a completed workflow is not valid evidence; the unit of evidence is a control-treatment comparison whose realized execution matches a human-approved protocol, admitted only after deterministic, artifact-backed verification plus human review (Eq. 3). Supporting claims: (i) EvoPilot's gated review overturned a false -22pp conclusion from a primitive autoresearch attempt (Sec. 5.3); (ii) ten paired replay/mutation fixtures validate the final guard set (Table 5); (iii) the selected model shipped with a +0.66% relative GSRR gain in a 7-day randomized A/B (Sec. 5.5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EvoPilot?
EvoPilot is a human-gated system from Meta researchers for LLM-agent-driven ML model development on production systems. LLM agents plan and run experiments, but a human-approved protocol plus deterministic checks over final artifacts decide which control-treatment comparisons count as valid evidence; agent self-reports and memory alone never do.
What was the -22pp bug described in the paper?
An early 'primitive autoresearch' attempt attributed a 22-percentage-point offline hit-rate drop to an interaction head. Both arms declared output depth K=600, but an evaluation defect meant the control actually produced depth 3000 while the treatment produced 600, making the arms incomparable. EvoPilot's review caught the mismatch; after repair, a matched comparison showed a +3.20pp improvement.
What were the main results of the EvoPilot study?
Over a 37-day campaign (7 research directions, 20 approved rounds) on Meta's Video Deep Dive video-discovery retrieval: the repaired head-only comparison improved offline click hit-rate@3000 by 3.20 percentage points; a 7-day randomized online A/B showed a +0.66% relative increase in GSRR; replay/mutation tests blocked all 10 planted faults and admitted all 10 clean cases; artifact reuse avoided about 5 GPU-hours.
Is EvoPilot code or data publicly available?
No. The system, the Video Deep Dive retrieval stack, the evaluation data, and the experiment ledgers are internal to Meta. The paper describes the protocol and controls in implementable detail but releases no code, prompts, skill text, or logs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is available only as caption text; the loop diagram could not be visually verified.
- The metadata abstract is truncated mid-sentence; the embedded full text (including Appendix A and Tables 7-8) appears complete and was used instead.
- All operational numbers (97.2h wall-clock, 38 incidents, 6,995 ledger events) are author-reported internal records that cannot be externally verified.
- Equations were extracted as plain text; notation was reconstructed from context.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.