SAM-V: Geometry-Aware Segment Anything for Multi-View Instance Segmentation
Jiangshan Gong, Yuqun Wu, Qiqian Fu, Yao Xiao, Chuhang Zou, Shenlong Wang, Derek Hoiem
cs.CV · cs.LG
Quick verdict
Strong SAM+VGGT fusion for multi-view segmentation (+5.7 T-mIoU vs PanSt3R; clear ablation), but single-run, small-N benchmarks without variance temper claims.
Summary
SAM-V is a prompt-conditioned multi-view instance segmentation model that couples a frozen SAM (ViT-H) with a frozen VGGT geometry encoder, training only a feature-fusion MLP, a prompt-fusion cross-attention module, and a fine-tuned SAM mask decoder. Sparse point prompts are enriched with VGGT camera tokens and local VGGT features sampled at the prompted pixel, and the decoder attends over all views jointly, predicting per-view masks in one forward pass without offline mask matching or explicit reconstruction. Training is two-stage (Hypersim pretraining, ScanNet++ fine-tuning) with focal plus Dice loss and an IoU-prediction term; four of eight frames per training sample lack the target, explicitly supervising all-zero masks for absent objects. An every-object mode derives prompts from SAM proposals and deduplicates via multi-view NMS. On the IGGT 3D tracking benchmark, SAM-V reports 78.4 T-mIoU / 79.4 O-IoU / 89.2 R@50 on ScanNet++ (58 objects) versus PanSt3R's 72.7/74.8/77.4, and leads all four metrics zero-shot on ScanNet (66 objects); versus SAM2 on Hypersim it lifts diverse-sampling O-IoU from 48.1 to 62.4 while performing comparably under continuous sampling (64.2 vs 63.5, with SAM2 slightly ahead on R@50). Ablations raise O-IoU from 21.3 to 64.2 as camera tokens and local features are added. Limitations: indoor, rigid, static scenes only; roughly 40-50 s/scene versus about 5 s for PanSt3R; single-run results without variance; and the headline ScanNet++ evaluation is in the fine-tuning domain.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Injecting VGGT geometry into SAM at both the dense-feature and sparse-prompt levels (camera tokens + local features) yields cross-view consistent instance masks in a single forward pass, without offline mask matching or explicit 3D reconstruction.
- Geometry-aware prompt fusion is the critical component: ablated O-IoU on Hypersim rises from 21.3 (SAM prompt only) to 35.5 (+camera token) to 64.2 (full) under continuous sampling.
- On the IGGT 3D tracking benchmark, SAM-V surpasses the strongest baseline PanSt3R by 5.7 T-mIoU and 11.8 R@50 on ScanNet++ and leads all four metrics zero-shot on ScanNet.
- The prompt-conditioned model extends to scene-level every-object segmentation via SAM proposals plus multi-view NMS, avoiding the class-vocabulary dependence that costs query-based methods up to 0.12 mean IoU on out-of-vocabulary scenes (Appendix G).
Strengths
Ablation cleanly isolates the proposed mechanism
Table 3 decomposes prompt fusion into SAM-Prompt (O-IoU 21.3 continuous / 23.0 diverse), + camera token (35.5/32.2), + local feature (64.2/62.4), and Fig. 5 illustrates the specific failure each component fixes (view confusion, then drift to nearby instances). The near-elimination of the continuous-vs-diverse gap (21.3 vs 23.0 at base; 64.2 vs 62.4 at full) directly supports the paper's core claim that geometry grounding, not appearance matching, drives consistency.
Absent-object handling is supervised and measured
Training uses 4 positive + 4 negative frames per sample (Sec. 4.1, Appendix A.2), explicitly supervising all-zero masks when the target is absent; Appendix C.4 defines precision/recall so that leaking into frames without the target counts as false positives. This addresses a real failure mode most multi-view baselines ignore, and shows up in R@50 (89.2 vs PanSt3R 77.4 on ScanNet++).
Unusually transparent baseline and timing accounting
Appendix G is a genuinely informative out-of-distribution probe
Zero-shot transfer result
Weaknesses
No uncertainty quantification anywhere
Headline benchmark is in the fine-tuning domain
Latency undercuts the robotics motivation
IGGT baseline re-scored under a modified protocol
PanSt3R evaluated with a demo-default vocabulary on the main benchmark
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
SAM-V: Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SAM-V and what problem does it solve?
SAM-V is a research model for multi-view instance segmentation: given several photos of the same scene and a point click on an object in one image, it segments that same physical object consistently in every image where it appears, including views with large camera movement. It works in a single forward pass by fusing a 2D segmentation model (SAM) with a feed-forward 3D geometry model (VGGT), avoiding the usual two-step approach of segmenting each image separately and then matching masks across views.
How much better is SAM-V than previous methods?
On the IGGT 3D tracking benchmark (indoor scenes), SAM-V reports 78.4 T-mIoU and 89.2 frame-recall@50 on the ScanNet++ split versus 72.7 and 77.4 for the strongest prior method PanSt3R, and it leads on all four reported metrics in a zero-shot test on ScanNet. Against SAM2 on synthetic Hypersim scenes with large viewpoint changes, it improves overall IoU from 48.1 to 62.4. Note these are the authors' single-run numbers without error bars, and SAM-V is about 10x slower per scene than PanSt3R (roughly 49 vs 5 seconds).
Is code and data available for SAM-V?
The abstract states that code and pretrained models are available in the paper's project repository (linked from its arXiv page, arXiv:2609.25490). Evaluation uses public datasets (Hypersim, ScanNet++, ScanNet), so inference-side reproduction is plausible; retraining required 8-14 NVIDIA L40S GPUs per the appendix.
What are SAM-V's main limitations?
Per the authors' own limitations section, it is trained only on indoor datasets (Hypersim, ScanNet++), assumes rigid objects in static scenes, and does not support language/open-vocabulary prompts. The review additionally notes: results are single runs with no reported variance on small object sets (58-66 objects); the headline ScanNet++ evaluation is in the same domain used for fine-tuning; and inference takes roughly 40-50 seconds per scene versus about 5 seconds for PanSt3R, which weakens the real-time robotics motivation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 and A1-A2 are only available as captions in the extracted text; qualitative evidence could not be independently verified from pixels.
- The abstract in the embedded text is truncated mid-sentence (after 'frame-level recall by 12 p'); claims were cross-checked against Sec. 1 and Table 2 instead.
- Some curriculum details are deferred to an unreleased supplement ('Full curriculum schedules, sampling thresholds, and hyperparameters are provided in the supplement'), though Appendix A covers most training specifics.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.