Less is More: Encoder-only Audio-Visual Segmentation
Ilpo Viertola, Vladimir Iashin, Sophie Tötterström, Esa Rahtu
cs.CV · cs.AI
Quick verdict
Proposes EASE, a simpler encoder-only AVS model with SotA AVSS at 3x throughput; AVSBench results are strong but efficiency framing favors mid-tier configs.
Summary
The paper proposes EASE, an encoder-only architecture for audio-visual segmentation (AVS) that strips the task-specific Transformer decoder inherited from image segmentation designs in favor of a plain ViT backbone compatible with vision foundation models. EASE adds three components: Gumbel-Softmax-based audio feature enhancement with learned semantic cluster centers, a mid-backbone audio query injection strategy, and a weight-shared bidirectional audio-visual fusion module using Gumbel-Softmax clustering. On AVSBench, the ViT-L 384² configuration reaches 56.5 J / 61.0 F on the AVSS task, surpassing prior methods such as AAVS (48.5 J), while a ViT-B configuration runs at up to 365 FPS — roughly 3× the throughput of prior state-of-the-art models at comparable accuracy — and the best model trains in about 11 GPU-hours on two A100s. Ablations show learned clustering beats linear projection and DDESeg's precomputed feature bank, and clustering-based fusion beats plain cross-attention. Limitations: the largest configuration costs 721 GFLOPs, far above baselines; the headline FPS figure comes from the mid-accuracy configuration, while the highest-accuracy model runs at 86 FPS; some S4/MS3 metrics still trail specialized methods; results are single runs on one dataset without variance reporting.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Removing the task-specific Transformer decoder in favor of a plain ViT encoder yields state-of-the-art AVSS accuracy (56.5 J with ViT-L at 384²) at up to 3× the throughput of prior SotA models at comparable accuracy.
- End-to-end learned Gumbel-Softmax audio clustering outperforms both a linear projection and DDESeg's precomputed audio feature bank (45.8 vs 44.3 and 43.9 J, Table 4).
- The clustering-based bidirectional AV Fusion module outperforms plain cross-attention and CAVP-style bidirectional attention (45.8 vs 44.4 and 43.5 J, Table 5).
- EASE trains in at most 11 GPU-hours (DINOv3 AVSS config) on two A100s, and its plain-ViT design enables adopting vision foundation models that specialized AVS architectures cannot use without re-engineering.
Strengths
Strong speed-accuracy Pareto at mid configurations
Table 1: EASE ViT-B 384² reaches 49.8 J / 54.1 F at 181 FPS vs AAVS 48.5 J / 53.2 F at 69 FPS — ~2.6× throughput at higher accuracy; ViT-B 224² peaks at 365 FPS with 96 GFLOPs.
Ablations isolate each new module's contribution
Table 4: learned clustering 45.8 J beats linear projection (44.3) and DDESeg's precomputed feature bank (43.9); Table 5: clustering-based AVFusion 45.8 J beats CAVP-style bi-way attention (43.5) and fusion without clustering (44.4).
Transparent comparison hygiene
Genuine architectural simplification enabling VFMs
Honest about the low-data regime
Weaknesses
Compute cost of the accuracy SotA is understated
Headline efficiency figures conflate configurations
S4/MS3 results are not uniformly state-of-the-art
MS3 advantage partly from S4 pretraining applied selectively
Single dataset, single runs, no variance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EASE (Encoder-only Audio-Visual Segmentation) targets AVS, focusing on the semantic sub-task AVSS on AVSBench [1,2]. The central claim is architectural: the Transformer decoder stack inherited from Mask Transformer-style image segmentation is largely redundant, and a plain ViT encoder plus lightweight audio machinery suffices. Concretely, the authors claim (i) SotA AVSS accuracy across backbones and input resolutions — ViT-L at 384² reaches 56.5 J / 61.0 F (Tables 1-2); (ii) up to 3× the throughput of prior SotA at comparable accuracy, peaking at 365 FPS for ViT-B 224²; (iii) training in under 11 GPU-hours (DINOv3 AVSS config); and (iv) two novel components — Gumbel-Softmax learned audio clustering and clustering-based bidirectional AV Fusion — plus a mid-backbone audio query injection. Claims are mostly scoped to AVSS, and Sec. 4.2 properly concedes that specialized methods retain advantages on some S4/MS3 metrics.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EASE in audio-visual segmentation?
EASE (Encoder-only Audio-Visual Segmentation) is a 2026 model from Tampere University that segments and classifies sound-emitting objects in video using a plain ViT encoder without a task-specific Transformer decoder. On the AVSBench semantic task it reaches 56.5 Jaccard with a ViT-L backbone at 384×384 input, while remaining much faster to train (about 11 GPU-hours) and run than previous state-of-the-art AVS models.
How fast is EASE compared to previous audio-visual segmentation models?
Its fastest published configuration (ViT-B, 224×224 input) reaches 365 frames per second, roughly 3× the throughput of prior state-of-the-art models at comparable accuracy. The most accurate configuration (ViT-L, 384×384) runs at 86 FPS but costs 721 GFLOPs, more compute than most baselines.
What dataset is EASE evaluated on?
Only AVSBench, the standard audio-visual segmentation benchmark with three subsets (S4, MS3, AVSS) of 10-second videos annotated at one frame per second. No other datasets are reported, so cross-dataset generalization is untested.
Is EASE code and those model weights available?
The paper states that code, model weights, and sample results are available via the project page (ease-avs.notion.site). The evaluation datasets are public, and training the main configuration needs only about 11 GPU-hours on A100-class hardware.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are known only from extracted captions; qualitative segmentation examples and architecture diagrams could not be assessed visually.
- Implementation details (query count Nq, Gumbel-Softmax temperature schedule, exact L1 split points) may live in supplementary material not present in the extracted text.
- Code/weights availability is stated in the paper but the project URL could not be verified offline.
- Baseline numbers are taken from prior publications or author retraining; no independent re-computation was possible from the text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.