Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM
Victor Bercy, Martyna Poreba, Michal Szczepanski, Samia Bouchafa
cs.CV
Quick verdict
Shows G2TM token merging is encoder-driven and decoder-agnostic across ViTs, with strong ablations; lacks head-to-head comparison with competing merging methods.
Summary
The paper extends Graph-Guided Token Merging (G2TM), a single module that merges spatially adjacent ViT tokens via thresholded cosine-similarity graphs, beyond its original Segmenter setting to ask whether token-merging effectiveness depends on the decoder. G2TM is evaluated across three segmentation frameworks (Segmenter, SETR, EoMT) covering linear, convolutional, and Transformer decoders, plus ViT classification on ImageNet-1k, on ADE20K and Cityscapes. For a given backbone, token counts and merged-token sizes are nearly identical across decoders, supporting the claim that G2TM's behavior is encoder-determined. Across configurations it cuts GFLOPs by 22-47% and raises throughput by up to 74% (94% on Cityscapes) with at most a 2-point mIoU loss; classification degrades by no more than 0.7 Top-1. Ablations show optimal hyperparameters depend on pre-training recipe and dataset, not decoder or resolution; DINOv2 features saturate cosine similarity near 1.0, causing an 8.8-point collapse; Proportional Attention helps classification but hurts segmentation. Main limitations: no head-to-head comparison with competing merging methods (ToMe, ALGM, PiToMe), single-run results without variance estimates, and pre-training substitutions that complicate comparison with published baselines. A careful systematic study, but its decoder-agnostic generality is demonstrated only for G2TM itself.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 4 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- G2TM's merging behavior (remaining token counts, merged-token sizes) and accuracy-efficiency trade-off are consistent across Segmenter, SETR, and EoMT decoders for a given backbone size, indicating its effectiveness is a property of the encoder rather than the decoder.
- Optimal G2TM hyperparameters depend on the backbone's pre-training recipe and the target dataset, not on decoder choice or input resolution.
- On ADE20K, G2TM achieves 22-47% GFLOPs reduction and up to 74% throughput increase while keeping mIoU degradation within 2 points for at least one configuration per model; on ImageNet-1k the accuracy cost is at most 0.7 Top-1.
- DINOv2 pre-training saturates adjacent-token cosine similarity (0.99-1.0), rendering the similarity threshold uninformative and causing up to an 8.8-point mIoU collapse, unlike AugReg pre-training.
- Proportional Attention should be task-conditional: it slightly improves classification but consistently harms segmentation, apparently by amplifying attention of merged tokens toward a [CLS] register token.
Strengths
Genuinely systematic cross-decoder design
Table 3 spans three frameworks and five decoder variants on identical ViT backbones; the near-identical token counts and merged-token sizes across SETR-B's Naive/PUP/MLA heads (e.g., 602/625/629 tokens at G2TM@1) give direct, quantifiable evidence that merging behavior is decoupled from the decoder, not just similar accuracy numbers.
Mechanistic pre-training ablation
Table 9 and Figure 6 contrast AugReg vs DINOv2 EoMT: with DINOv2, adjacent-token cosine similarity concentrates in 0.99-1.0, destroying the threshold criterion (mIoU drops to -8.8 at layer 2), and the authors show a deeper insertion (G2TM@3, tau=0.90) recovers 56.7 mIoU — turning a failure case into an actionable diagnostic.
Clean disentangling of dataset vs resolution effects
Honest reporting of failure regimes
Task-conditional analysis of Proportional Attention
Weaknesses
No external method baselines
Single-run results with sub-noise deltas interpreted as findings
Baselines not comparable to published numbers
Efficiency accounting is thin
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is G2TM and how does it speed up Vision Transformers?
G2TM (Graph-Guided Token Merging) is a module inserted once, early (layer 1 or 2) in a Vision Transformer encoder. It represents tokens as a graph where edges connect spatially adjacent patches, weights edges by cosine similarity of token embeddings, cuts edges below a threshold, and merges each resulting connected component into one averaged token. Fewer tokens make all subsequent self-attention cheaper. On ADE20K segmentation it cuts compute by 22-47% and raises throughput by up to 74% while losing at most 2 mIoU points.
Does G2TM require retraining or fine-tuning the model?
Fine-tuning with the module gives the best results: the paper trains with masked self-attention (to keep merged tokens from affecting training) and a curriculum that gradually lowers the similarity threshold. Applying G2TM off-the-shelf to already-trained models works acceptably only for image classification and for ViT-Large segmentation backbones; smaller segmentation models degrade more without fine-tuning.
Why does G2TM perform much worse with DINOv2 pre-trained weights?
DINOv2 features make neighboring tokens almost identical: cosine similarities concentrate between 0.99 and 1.0, so a similarity threshold can no longer distinguish redundant from informative tokens, and merges become essentially uninformed — the paper's worst case is an 8.8 mIoU-point drop on EoMT. Inserting G2TM in deeper layers, where the similarity distribution broadens again, largely recovers performance.
Which decoder architectures does G2TM work with?
The paper tests linear decoders (Segmenter-Linear), convolutional decoders (SETR-Naive/PUP/MLA), and Transformer decoders (Segmenter Mask Transformer, EoMT). Because merging happens in the encoder, behavior is nearly identical across decoders for a given backbone. Convolutional decoders require the token sequence to be restored (unmerged) before their spatial operations, while Transformer decoders can consume the shortened sequence directly.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3, 5, and 6 (similarity distributions and qualitative segmentation maps) are described only via captions in the extracted text; their visual content could not be verified.
- The code link appears as a placeholder ('Codes are available here') in the extracted text; code availability is stated but the repository could not be identified or checked.
- This is arXiv v1 (Sep 2026) and appears not yet peer-reviewed; the paper extends the authors' own VISAPP 2026 G2TM paper, so continuity of claims with that venue could not be cross-checked.
- Table rows render cleanly in the extraction, but a few related-work citations (e.g., refs 18, 19, 26-28, 31-35) appear in lists without accompanying discussion, suggesting minor extraction gaps in Section 2.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.