SetPlanner: A Lightweight Plug-in Point-Set Planner for Frozen SAM
Dawei Yan, Yuezhe Yang, Menglan Ruan, Chunfeng Yang, Yudong Zhang
cs.CV
Quick verdict
SetPlanner adds a 1.52M flow-based point-prompt planner to frozen SAM for endoscopy; results are strong and ablated, but SOTA auto-prompting baselines are missing.
Summary
The paper targets fully automatic surgical instrument segmentation with SAM, which natively requires user-supplied point prompts. SetPlanner is a 1.52M-parameter plug-in that generates complete K-point prompt sets with a permutation-aware rectified flow conditioned on frozen SAM encoder features; training targets are rejection-sampled from a 4px-eroded foreground under a Mahalanobis separation constraint, and an optimal noise-to-target assignment (Eq. 2) makes training permutation-invariant. At inference, M=8 candidate point sets are decoded by frozen SAM and reduced to one mask by a medoid consensus readout, while candidate disagreement serves as a ground-truth-free failure score. A 'frozen-pathway' protocol defines a localization gap between a rule floor (best of five zero-training prompting rules, 0.509 Dice on Kvasir) and a GT-derived point reference (0.953); SetPlanner (0.934) recovers 96% of this 44.4-point gap. Across Kvasir-Instrument, Endoscapes, and EndoVis it beats a LoRA-adapted SAM on all six cross-dataset transfer routes (mean Dice 0.695 to 0.839). Main limitations: no head-to-head comparison with published automatic-prompting methods (ProSAM, SeqSAM, AoP-SAM), the LoRA baseline is prompted with random points, LoRA wins in-domain on EndoVis (0.929 vs 0.911), seeds are averaged without variance, and the training-target distribution resembles the GT-derived reference, partially circularizing the recovery claim.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under a frozen-pathway protocol, the localization gap on Kvasir-Instrument is 44.4 Dice points between the best zero-training rule (0.509) and a GT-derived point reference (0.953), and this gap is measurable because the frozen decoder stays coordinate-sensitive (a +64,+64 GT shift costs 60.3 Dice frozen vs 3.8 under LoRA).
- SetPlanner, a 1.52M-parameter permutation-aware conditional-flow planner, recovers 96% of that gap, reaching 0.934 Dice on Kvasir-Instrument with frozen SAM.
- Across three endoscopic datasets, SetPlanner wins all six cross-dataset transfer routes over a LoRA-adapted SAM, raising mean transfer Dice from 0.695 to 0.839.
- Candidate disagreement among M=8 decoded masks ranks low-Dice (Dice<0.7) frames at AUROC 0.969, providing a ground-truth-free failure score.
- Pool quality comes from complete-set flow generation rather than perturbation, capacity, or ensembling, as shown by the jitter, independent-model, and slot ablations in Table 3.
Strengths
Well-posed attribution protocol
The frozen-pathway protocol (Sec 2.1) quantifies a concrete localization gap (44.4 Dice points on Kvasir between rule floor 0.509 and GT-derived reference 0.953), and the (+64,+64) shift experiment (60.3-point drop frozen vs 3.8 under LoRA, Sec 3.1) empirically justifies why freezing makes decoded accuracy attributable to the point source.
Principled generative design with discriminating ablations
Permutation-aware rectified flow over complete point sets with Hungarian-style assignment (Eq. 2) is compared against local jitter (0.934 vs 0.926 Dice, pair IoU 0.951 vs 0.965), eight independent models (0.929 at 8x parameters), and deterministic slots (Table 3), showing the gain comes from complete-set generation rather than capacity or perturbation.
Deployment-oriented extras
Efficiency and interface preservation
Weaknesses
Weak headline baseline
No comparison with published automatic-prompting methods
Selective narrative around in-domain losses
Potential circularity in gap recovery
Thin statistical reporting and leakage risk
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SetPlanner and what problem does it solve?
SetPlanner is a 1.52M-parameter add-on module for the Segment Anything Model (SAM) that automatically generates the point prompts SAM normally requires from a human user. It enables fully automatic segmentation of surgical instruments in endoscopic images without any fine-tuning of SAM itself: a generative flow model proposes complete sets of K points, SAM decodes eight candidate masks, and a consensus rule picks the final mask.
How accurate is SetPlanner compared to fine-tuning SAM?
On the Kvasir-Instrument dataset it reaches 0.934 Dice with frozen SAM, recovering 96% of the gap between trivial prompting rules (0.509) and ground-truth-derived prompts (0.953). Across three endoscopic datasets it beats a LoRA-fine-tuned SAM on all six cross-dataset transfer settings (mean transfer Dice 0.839 vs 0.695), although LoRA wins in-domain on the EndoVis dataset (0.929 vs 0.911).
Does SetPlanner modify or fine-tune SAM?
No. SAM's image encoder, prompt encoder, and mask decoder all stay frozen. Only the small planner (1.52M parameters, about 1% extra compute) is trained, and it outputs points through SAM's native point-prompt interface, so the same planner can in principle be reused across different SAM backbones.
What does the disagreement score in SetPlanner mean?
Because the planner samples eight independent point sets, SAM produces eight candidate masks per frame. The average pairwise IoU overlap among these masks measures how much the automatic point source varies; low agreement (high disagreement) predicts frames where segmentation fails, ranking Dice<0.7 cases with 0.969 AUROC, so unreliable frames can be flagged or deferred without ground truth.
What datasets were used to evaluate SetPlanner?
Three public endoscopic benchmarks: Kvasir-Instrument (472 training / 118 test images), Endoscapes (334/112, video-grouped split), and the EndoVis 2017 robotic instrument challenge (1778 training frames, official 1200-frame test set). All are openly available and de-identified, per the paper's ethics statement.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are available only through their captions in the extracted text; qualitative claims about point-set geometry and candidate diversity could not be visually verified.
- Table 2 was extracted with dense inline formatting; the backbone identity of each block (e.g., the Kvasir (SAM1) rows) was inferred from context and may be misaligned.
- The paper cites several 2026 preprints (e.g., SAM 3, Endoscapes journal version) that could not be cross-checked within this offline review.
- Implementation details such as optimizer, learning rate, LoRA rank, and exact seed protocol are not in the extracted text, limiting reproducibility assessment.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.