FailBench: How Reliable are VLMs at Judging Robot Task Success?
Zaruhi Navasardyan, Tatul Danielyan, Hrant Davtyan
cs.RO · cs.AI
Quick verdict
Credible cross-source VLM robot-failure benchmark: best model 0.77 BA over 14 sources; fine-tuned detectors hurt transfer; success-bias partly qualitative.
Summary
FailBench assembles 2,197 robot manipulation attempts from 14 public sources (12 real-world, 2 simulated) into a single benchmark for judging whether a VLM can tell success from failure. The authors reuse the outcome labels assigned by the original data providers rather than authoring their own, and 75% of failures occurred naturally rather than being staged. Thirteen VLM-based detectors are evaluated with balanced accuracy under each model's intended prompt settings: the best, Gemini 3 Flash, reaches only 0.77 mean balanced accuracy. Fine-tuned failure detectors consistently score below general-purpose models and below their own pretrained base models (Table 7), indicating that specialization hurts transfer. Performance tracks the visual evidence an outcome requires: slices where object motion decides the label approach saturation, while contact-rich assembly collapses (reassemble slice mean 0.52; no model above 0.60). Qualitative trace analysis of one open model suggests a systematic bias toward answering success under ambiguous evidence, with errors carrying longer reasoning (1,868 vs 1,464 characters); the authors themselves flag this analysis as protocol-free. Cropping inputs to a localized outcome-relevant region lifts the top detector from 0.773 to 0.797 macro (exact McNemar p=0.0015) without retraining. Limitations include binary execution-failure scope, vision-only input, tabletop parallel-jaw embodiments, and label spot-checks on only about 25% of samples.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On FailBench (2,197 attempts, 14 sources), the best of 13 VLM detectors reaches only 0.77 mean balanced accuracy, so no current judge is reliable enough to be used as ground truth.
- Purpose-built failure detectors consistently underperform general-purpose VLMs and score below their own pretrained base models, showing fine-tuning for failure detection hurts out-of-source transfer.
- Performance depends on required visual evidence: detection nears saturation when object motion decides the outcome but no model exceeds 0.60 balanced accuracy on contact-rich assembly.
- Models show a systematic bias toward predicting success under ambiguous evidence, and increased reasoning effort does not help since errors carry longer reasoning traces.
- Localizing the outcome-relevant region and cropping input improves the top detector from 0.773 to 0.797 macro balanced accuracy (exact McNemar p=0.0015) without any retraining.
Strengths
Genuinely cross-source design with no self-authored labels
14 independently collected sources under one schema (Section 3, Tables 5-6), each reusing its provider's own outcome labels; per-source available-vs-drawn ledgers (Table 6) and visible imbalances (phail 67F/13S with 61 timeouts; botfails 108/36) rather than hidden rebalancing.
Exemplary score-validity audit against published numbers
Appendix B.1/D reconciles the harness with published results (Guardian 0.741 vs published 0.77 on UR5FAIL; RoboFAC recall-reweighted 0.811 vs published 0.806) and shows the all-failure constant already scores 0.797 on RoboFAC's published split and 0.890 on ViFailback's — turning the class-imbalance critique into arithmetic.
Controlled specialist-vs-base comparison
Statistically anchored intervention with full ablations
Honest, explicit limitation accounting
Weaknesses
Central mechanism claim is not systematically measured
Uneven label provenance and thin verification
No uncertainty on model rankings
Internal numeric inconsistencies
Input-format confound across models
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FailBench isolates one question: given a task instruction and the recording of a completed attempt, did the robot succeed? The benchmark aggregates 2,197 attempts from 14 independently collected public sources (12 real-world, 2 simulated; Fig. 1, Tables 5-6), reuses each source's own outcome labels, and is deliberately majority-organic (75% of failures natural; 1,176 failures vs 1,021 successes). The paper claims: (1) cross-source reliability of VLM judges is low — best macro balanced accuracy 0.77 (Gemini 3 Flash, Table 1); (2) fine-tuned failure detectors transfer worse than general-purpose VLMs and than their own base models (Table 7); (3) performance tracks the visual evidence an outcome requires — near-saturation on motion-decided slices, no model above 0.60 on contact-rich assembly (reassemble slice mean 0.52); (4) a systematic success bias under ambiguous evidence; (5) an input-level localization+crop pipeline gains ~2.4 points without retraining. Scope is explicitly narrow: binary execution failures only, vision-only, no planning failures or progress estimation (Section 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FailBench and what is it for?
FailBench is a benchmark of 2,197 robot manipulation attempts compiled from 14 public datasets (12 real-world, 2 simulated) to test whether vision-language models can correctly judge whether a robot attempt succeeded or failed. It reuses each source dataset's own outcome labels rather than creating new ones, and 75% of its failures happened naturally rather than being deliberately staged.
What is the paper's main result?
The best of 13 evaluated VLM detectors, Gemini 3 Flash, reaches only 0.77 mean balanced accuracy — roughly one wrong verdict in four. Detectors fine-tuned specifically for failure detection score worse than general-purpose VLMs and even below their own base models, and no model exceeds 0.60 balanced accuracy on contact-rich assembly tasks.
How much does the proposed cropping trick improve detection?
Localizing the outcome-relevant region and cropping the input raises the strongest detector's macro balanced accuracy from 0.773 to 0.797 (about +2.4 points, exact McNemar p=0.0015) without retraining. Gains concentrate in wide-workspace scenes (e.g. +10.7 on SimplerEnv), while a few sources regress (e.g. −6.7 on reflect).
Are the benchmark and code publicly available?
The paper states that FailBench and the accompanying evaluation harness are released, with a project page (metric-ai-lab.github.io/failbench), frozen subsets, and per-source sampling ledgers documented in Appendix A, making independent re-evaluation straightforward.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 appear only as captions in the extraction; the underlying plot images are not present (standard for PDF text extraction — all numeric content survives in Tables 1-10).
- The provided text ends after Appendix D's Table 10 (page 19); every section and table referenced in the body (A-D, Tables 1-10) is present, so nothing appears to be cut off mid-content.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.