RoboVAD: A Large Cross-Domain Evaluation Benchmark for Anomaly Detection in Robotic Arm Manipulation Videos
Alexandru-Bogdan Dura, Sebastian Balmus, Radu Tudor Ionescu
cs.CV · cs.AI · cs.LG · cs.RO
Quick verdict
Large robotic-arm VAD benchmark with novel cross-task/cross-anomaly protocols; solid resource, but single-run AUCs and thin annotation validation temper it.
Summary
The authors introduce RoboVAD, a video anomaly detection benchmark of 620,979 frames (5.75 hours) from 1,078 SO-ARM101 teleoperation episodes recorded with two 1080p/30fps cameras, spanning five manipulation tasks (cube pick-and-place variants, stacking, rice pouring, ring insertion) and five frame-annotated anomaly categories (failed grasp, collision, placement failure, object drop, object not dropped), totaling 1,570 localized anomaly events. Its distinguishing feature is a cross-domain evaluation protocol: training uses only cube tasks with a subset of anomaly categories, while held-out tasks (rice pouring, ring insertion) and held-out categories define in-domain, cross-anomaly, cross-task, and combined cross-task+anomaly test settings. Nine methods are benchmarked under micro/macro frame-level AUC, including the authors' FlowJigsaw-SSL/Supervised models operating on six-channel dual-view Farneback optical flow with arrow-of-time, motion-irregularity, and inpainting heads. FlowJigsaw-Supervised reaches 94.6% in-domain micro AUC, but in the combined setting all methods fall below 70%, with FlowJigsaw-SSL best at 67.4%; an ablation shows optical flow dominates RGB inputs. Limitations: single platform and tabletop environment, author-only annotation validation on 50 episodes, no variance or significance reporting, no per-category breakdowns, and several CLIP-based baselines scoring near or below chance, raising baseline-adaptation concerns.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- RoboVAD is the first robotic-manipulation VAD resource with explicit cross-task and cross-anomaly (open-set) protocols, comprising 620,979 frames and 1,570 frame-localized anomaly events over 5 tasks and 5 anomaly categories.
- Methods that perform well in-domain (up to 94.6% micro frame-level AUC) degrade substantially when manipulation tasks, anomaly categories, or both are unseen.
- No evaluated method exceeds 70% micro frame-level AUC in the combined cross-task+anomaly setting, which the authors present as evidence of the benchmark's difficulty.
- Explicit dual-view optical-flow input consistently outperforms RGB for both proposed FlowJigsaw variants across all four evaluation settings (Table III).
- The proposed FlowJigsaw-SSL and FlowJigsaw-Supervised models are competitive with or superior to state-of-the-art VAD competitors on this benchmark.
Strengths
Scale and annotation granularity unmatched in this niche
620,979 frames, 1,078 episodes, and 1,570 frame-level localized anomaly events across two camera views, roughly 3.6x the instance count of the next largest dataset in Table I (Handover, 433), with per-event severity and cross-view visibility labels (Sec. III).
Genuinely new evaluation axis for robot failure detection
Disjoint task and anomaly-category splits yield four protocols (in-domain, cross-anomaly, cross-task, cross-task+anomaly), extending UBnormal-style open-set evaluation with a second domain axis; Table I shows no prior robotic dataset offers either cross setting.
Broad, consistent baselining
Actionable input-representation finding
Reproducibility posture
Weaknesses
No statistical uncertainty behind any reported number
Difficulty narrative rests on non-comparable cross-setting comparisons
Annotation reliability validated minimally and non-independently
No per-category or per-task diagnosis of the claimed difficulty
Baseline adaptation transparency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the RoboVAD dataset and how big is it?
RoboVAD is a public benchmark for detecting manipulation failures in robotic arm videos. It contains 1,078 real teleoperation episodes recorded with two 1080p/30fps cameras, totaling 620,979 frames (5.75 hours). It covers five tasks (single/multiple-cube pick-and-place, cube stacking, rice pouring, ring insertion) and five anomaly categories (failed grasp, collision, placement failure, object drop, object not dropped), with 1,570 anomaly events annotated at frame level. Data and code are released via Zenodo.
How well do current methods perform on RoboVAD?
In the standard in-domain setting the best method reaches 94.6% micro frame-level AUC. Performance drops sharply in cross-domain settings: when both tasks and anomaly types are unseen, every evaluated method scores below 70% micro AUC, with the best at 67.4% (the authors' FlowJigsaw-SSL). An ablation also shows optical-flow inputs substantially outperform RGB frames for the proposed models.
What makes RoboVAD different from existing robot failure datasets?
Prior datasets such as FAILURE, Bot fails, and Bookshelf mostly evaluate on the same tasks and failure types seen during training and often lack temporal (frame-level) annotations. RoboVAD is deliberately organized so training and test sets are disjoint in both manipulation task and anomaly category, enabling open-set, cross-domain evaluation, and it has roughly 3x more anomaly instances (1,570) than earlier benchmarks.
Which robot and setup was used to record RoboVAD?
Episodes were collected on a low-cost SO-ARM101 robot arm controlled through LeRobot's leader-follower teleoperation, in a tabletop scene with two fixed cameras at different heights and viewing angles. Robot joint and gripper states are recorded alongside the video, and all recordings and annotations were produced by the authors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 (task/anomaly example frames and two qualitative score plots) cannot be assessed from the extracted text; qualitative alignment claims are taken at face value.
- Table I arrived flattened in extraction; the column mapping was verified only through internal consistency checks (163,445 + 34,266 + 423,268 = 620,979 frames).
- The public Zenodo release could not be verified without network access; availability is as stated by the authors.
- The per-camera view protocol for baselines is not fully specified in the text, limiting assessment of baseline fairness.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.