VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removal
Haonan Huang, Haonan Huang, Tianrui Qiu, Xianghao Zang, Yinan Du, Zhixiang He, Chi Zhang, Hao Sun, et al.
cs.CV · cs.AI
Quick verdict
VLM evaluator and paired-video benchmark for object removal claims >0.9 human correlation; strong design but provenance and CI-free stats need audit.
Summary
The paper introduces VOR-Bench, a three-part evaluation suite for video object removal (VOR): (1) VORD, a 150-pair dataset (VORD-L: 300) presented as the first VOR benchmark providing both paired edited videos and hand-painted 'graffiti' masks, mixing model-generated, camera-captured, and tool-rendered sources; (2) rMPAF, a pipeline that builds motion-coherent edited pairs by removing objects from first/last frames with GeoRemover and interpolating the middle with a LoRA-tuned Wan-2.1 first-last-frame-to-video model, sharing anchors so non-edited regions stay aligned; and (3) VOR-MDSM, a Qwen3-VL-8B fine-tuned with LoRA on 3,299 human-annotated pairs (15 experts, 5 raters each, ICC(2,5) 0.86–0.89) that scores object removal completeness, mask-background consistency, and logical plausibility on 1–5 scales. Reported video-level correlations with human ratings exceed 0.9 for Pearson/Spearman/Kendall on all dimensions, beating VideoLLaMA3, zero-shot Qwen3-VL, and Gemini-3.1-pro; method-level ranking correlation is 0.976 versus 0.382 for PSNR/SSIM/LPIPS and 0.697 for ReMOVE. On VORD, ROSE, EffectErase, and VOID form the top tier. Main concerns: possible train/test overlap (training uses the same categories and construction pipeline as the test set; the five source models are unnamed), near-perfect correlations without confidence intervals, and unquantified background drift in camera-captured pseudo-pairs. Data release is promised, not yet verified.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- VORD is the first VOR evaluation dataset providing both paired edited videos and hand-painted graffiti masks, with 150 pairs across model-generated, camera-captured, and tool-rendered sources.
- rMPAF produces realistic, motion-coherent paired videos by combining GeoRemover first/last-frame edits with LoRA-tuned Wan-2.1 first-last-frame interpolation under shared anchors.
- VOR-MDSM, a LoRA-tuned Qwen3-VL-8B, achieves r, ρ, τ > 0.9 with human ratings on all three evaluation dimensions, outperforming VideoLLaMA3, zero-shot Qwen3-VL-8B, and Gemini-3.1-pro.
- Model rankings induced by VOR-MDSM match human rankings far better than traditional metrics (Spearman 0.976 vs 0.382) and ReMOVE (0.697).
- On VORD, ROSE, EffectErase, and VOID form the top-performing tier, and mask-background consistency is the weakest dimension across most models.
Strengths
Genuine benchmark gap addressed with a well-designed dataset
VORD (§3.1, Table 1) is the first VOR evaluation set combining paired edited videos with hand-painted first-frame graffiti masks, split 50/50/50 across model-generated, camera-captured, and tool-rendered sources with 14 visual types — directly targeting the 'questionable reference' and mask-realism problems named in §1.
Anchor-shared pairing is a sound answer to missing ground truth
rMPAF (§3.2) generates original/removed videos from identical first/last-frame anchors with the same fine-tuned Wan-2.1-FLF2V decoder, keeping non-edited regions aligned; Table 2's MonST3R validation (RotErr 0.40 vs 21.68/27.80 for Kling/Wan baselines) quantitatively supports camera-motion consistency.
Annotation reliability is reported, not just asserted
Competitive and complete baseline sweep
Practical scale and cost justified by ablation
Weaknesses
Train/test provenance ambiguity threatens the headline numbers
Near-ceiling correlations reported without any uncertainty
Camera-captured 'ground truth' is only an adjacent recording
Thin judge-baseline comparison and small method-level n
Reproducibility currently aspirational
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
VOR-Bench targets evaluation of video object removal (VOR): deleting a masked object from a video with plausible background restoration. The paper argues current practice fails on two axes — questionable reference videos (real footage has no true "removed" ground truth) and metrics (PSNR/SSIM/LPIPS) that diverge from human judgment (Fig. 1: visible artifacts scoring SSIM 0.966/0.972 versus a cleaner 0.931). Three deliverables follow: (1) VORD, 150 paired videos with hand-painted first-frame graffiti masks (VORD-L: 300), split equally across model-generated, camera-captured, and tool-rendered sources (Table 1); (2) rMPAF, a paired-video synthesis framework; (3) VOR-MDSM, a VLM scorer over three dimensions — object removal completeness, mask-background consistency, logical plausibility (§3.3). Headline claim: correlation with human perception exceeds 0.9.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is VOR-Bench?
VOR-Bench is a benchmark for evaluating video object removal (VOR) — the task of deleting an object from a video and plausibly filling the region. It bundles three things: VORD, a dataset of 150 (or 300 in VORD-L) paired original/edited videos with realistic hand-painted 'graffiti' masks; rMPAF, a pipeline for generating motion-coherent paired videos; and VOR-MDSM, a vision-language model that scores removal quality 1–5 on three dimensions: object removal completeness, mask-background consistency, and logical plausibility.
How well does VOR-MDSM align with human judgment?
The paper reports Pearson, Spearman, and Kendall correlations above 0.9 with human ratings on all three scoring dimensions, evaluated over 1,500 videos rated by 15 trained annotators. At the model-ranking level, VOR-MDSM's Spearman correlation with the human ranking of ten removal methods is 0.976, versus 0.382 for PSNR/SSIM/LPIPS and 0.697 for the ReMOVE metric. Note these are point estimates without confidence intervals.
Which video object removal models perform best on VORD?
According to Table 3 of the paper, ROSE ranks first with an average VOR-MDSM score of 4.33, followed by EffectErase (4.27) and VOID (4.22); Casper (4.15) and MiniMax-Remover (4.09) form the second tier, while older flow/transformer methods like ProPainter, FuseFormer, and FloED score lowest. Mask-background consistency is the weakest dimension for almost all models, indicating background restoration remains the main open challenge.
Is the VORD dataset or VOR-MDSM code publicly available?
The paper only promises a future release ('We will release VOR-Bench along with its documentation'); at the time of this review no repository, license, or download link had been verified. Training details (Qwen3-VL-8B backbone, LoRA rank 64, 10K steps on 4×A100) are disclosed, but the annotated training corpus and the five unnamed source removal models are not yet inspectable.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (Figs. 2–13) are garbled in text extraction; numeric conclusions rely on tables and inline text only.
- Supplementary Sections 6–8 are only partially extracted; annotation guidelines, detailed rankings, and expert demographics are referenced but not fully present.
- Table 4's Untrained/Trained row grouping is inferred from layout and surrounding text.
- Data/code availability is a promise in the paper; no external check was performed (network access disabled).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.