ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives
Nikos Giakoumoglou, Andreas Floros, Kleanthis-Marios Papadopoulos, Tania Stathaki
cs.CV · cs.AI · cs.LG
Quick verdict
Claims synthetic hard negatives improve contrastive ViT pretraining (ViT-B 77.1% ImageNet linear); broad results, but fine-tuning baselines warrant scrutiny.
Summary
ViTAMINS augments contrastive self-supervised pretraining of vision transformers with synthetic hard negatives. Building on the MoBY framework (momentum encoder, 4096-entry queue), the method ranks queued negatives by similarity to the anchor, takes the top 256, and generates 768 additional negatives on-the-fly via six strategies — interpolation, extrapolation, mixup, noise injection, and gradient- or sign-based perturbations — added to the InfoNCE denominator. Trained 300 epochs on ImageNet without multi-crop, it reaches 73.1% (ViT-S/16) and 77.1% (ViT-B/16) linear top-1 accuracy, exceeding the MoBY baseline (72.3/72.8) and, at ViT-B scale, V-JEPA ViT-L (73.7%). The authors report sharper attention maps, stronger DAVIS-2017 video segmentation than identically trained reproductions (44.3 vs 39.1 J&F for ViT-S), and gains in retrieval, copy detection, COCO detection and ADE20K segmentation, plus k-NN accuracy up to +11.3 points (Swin-T vs the BYOL reproduction). Main concerns: fine-tuning baselines in Table 8 score below their own linear-probe results, inflating ViTAMINS margins; no seeds or confidence intervals accompany margins as small as +0.2; and DINO with its full trick set still leads substantially on DAVIS (61.8). Code is released.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Adding 768 synthetic hard negatives per anchor to a MoBY-style contrastive objective raises ImageNet linear top-1 to 73.1% (ViT-S/16) and 77.1% (ViT-B/16) after 300 epochs without multi-crop (Tables 1-2).
- The resulting features show emergent semantic structure — sharper attention maps (Figures 1, 3) and better DAVIS-2017 segmentation than identically trained negatives-free baselines (44.3 vs 39.1 J&F, Table 5).
- ViT-B/16 ViTAMINS surpasses V-JEPA ViT-L on ImageNet linear evaluation (77.1 vs 73.7, Table 1), implying better resource efficiency.
- Combining all six synthetic-negative strategies beats any individual strategy (Table 9), and SynCo's cooldown — disabling synthetics for the last 100 epochs — is required for stable convergence (Table 11).
Strengths
Unusually broad evaluation under one matched regime
Linear/k-NN on ImageNet (Tables 1-2), ROx/RPar retrieval (Table 3), Copydays (Table 4), DAVIS-2017 (Table 5), COCO and ADE20K (Table 6), plus five transfer datasets (Tables 7-8), all for 300-epoch no-multi-crop training — the same-regime DINO comparison in Sec. 6 is explicit and fair.
Complete method disclosure with exact reductions
All six generation strategies with hyperparameter ranges (Eq. 1), queue size K=4096, top N=256, 768 synthetic negatives, τ=0.2, drop path 0.2/0.0, and explicit reductions to MoBY (Eq. 4) and BYOL (Eq. 5), making the actual delta over prior art unambiguous.
Systematic ablations
Concrete resource-efficiency argument
Weaknesses
Fine-tuning baselines appear misconfigured
No uncertainty quantification
Emergence claim is regime-relative
Mixed baseline provenance and unremarked losses
Essentially SynCo ported to transformers
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ViTAMINS (Sec. 3) modifies MoBY-style contrastive pretraining of ViT-S/B and Swin-T/S by injecting synthetic hard negatives: from a 4096-entry momentum queue it selects the top 256 negatives by cosine similarity to the anchor, then generates 768 synthetic negatives on-the-fly via six embedding-space operations (interpolation, extrapolation, negative-negative mixup, Gaussian noise, gradient- and sign-based perturbations; Eq. 1), all added to the InfoNCE denominator (Eqs. 2-3, τ=0.2). The paper claims (i) higher ImageNet linear and k-NN accuracy than negatives-free baselines (73.1%/77.1% ViT-S/B top-1, 300 epochs, no multi-crop; Tables 1-2), (ii) emergent segmentation structure (Figures 1/3, Table 5), (iii) resource efficiency — ViT-B/16 at 77.1% above V-JEPA ViT-L at 73.7% — and (iv) transfer gains across retrieval, copy detection, COCO, ADE20K and five classification datasets (Tables 3-8). Claims (i) and (iv, linear part) are supported by the tables; (ii) and the abstract's strongest phrasings need qualification.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ViTAMINS in self-supervised learning?
ViTAMINS is a modification of contrastive pretraining for vision transformers (ViT-S/B, Swin-T/S): alongside the usual memory-queue negatives, it generates 768 extra synthetic hard negatives per anchor in embedding space — by interpolating, extrapolating, mixing, noising, or gradient/sign-perturbing the 256 hardest queued negatives — and adds them to the InfoNCE loss. With synthetics disabled it reduces exactly to MoBY.
How accurate is ViTAMINS on ImageNet?
After 300 epochs of pretraining without multi-crop augmentation, linear-probe top-1 accuracy is 73.1% for ViT-S/16 and 77.1% for ViT-B/16 (94.4% top-5); k-NN accuracy reaches 71.0% (ViT-S) and 73.3% (ViT-B). Swin-T and Swin-S variants reach 75.4% and 78.0% linear top-1.
Does ViTAMINS beat DINO?
Under the authors' matched regime (300 epochs, no multi-crop) it beats their DINO reproduction on linear probing, k-NN, retrieval, copy detection, DAVIS video segmentation and dense prediction. However, DINO with its full original recipe (multi-crop, centering, sharpening) still scores far higher on DAVIS-2017 segmentation (61.8 vs 44.3 J&F for ViT-S), so the comparison is regime-dependent.
Is ViTAMINS code and data available?
Yes. The paper links a GitHub repository (github.com/giakoumoglou/vitamins) and discloses full training details: batch size 512, AdamW with base LR 1e-3 and weight decay 0.05, queue size 4096, 256 hardest negatives, 768 synthetic negatives, temperature 0.2, and asymmetric drop path 0.2/0.0. Pretraining uses public ImageNet-1k and ImageNet-100.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Supplementary material is cited for several claims (8/11 and 9/11 linear-probe win rates, the full warmup/cooldown ablation, extra attention visualizations) but is not part of the provided text; those specific claims could not be verified.
- Attention-quality judgments (Figures 1 and 3) rest on images; only captions were extractable, so visual sharpness claims were assessed via quantitative proxies such as Table 5 (DAVIS) instead.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.