Are We Really Doing Few-Shot Learning? A Critical Examination of Pre-Training Assumptions
Alejandro Galan-Cuenca, Marcelo Saval-Calvo, Antonio Javier Gallego
cs.CV · cs.AI · cs.LG
Quick verdict
Shows class-disjoint in-domain pre-training inflates few-shot results by 9.66pp vs OOD; systematic, credible, but limited to 2016-18 metric architectures.
Summary
This paper questions whether standard few-shot learning benchmarks, which pre-train on class-disjoint splits drawn from the same dataset as the target classes, genuinely measure low-data learning. The authors compare no pre-training, in-domain class-disjoint pre-training (ID-PT), supervised out-of-domain pre-training (OOD-PT), and two label-free strategies — clustering-based pseudo-labels (UCKM) and instance-augmentation episodes (UIAug) — across eight public datasets (music notation, historical and handwritten symbols, medical, natural images), three metric-based architectures (Matching, Prototypical, Relation Networks), and 5/20-way, 1/5/10-shot settings, with five bootstrap runs. Both without adaptation: ID-PT improves over no pre-training by 33.41 pp on average versus 23.75 pp for supervised OOD-PT, a 9.66 pp optimistic bias. Support-set adaptation helps OOD-PT far more (7.69 pp) than ID-PT (1.37 pp). With adaptation, UIAug nearly matches supervised OOD-PT (27.71 vs 27.97 pp), suggesting source labels are largely unnecessary. A factor-analysis composite of pre-training-computable domain-similarity descriptors selects sources within a median 1.37 pp of the oracle. Limitations: architectures date from 2016-2018, all images are rescaled to 40x40 (breaking comparability with standard benchmark numbers), the ID-OOD gap averages over heterogeneous OOD sources of varying scale, and no significance tests are reported for the headline gaps.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Class-disjoint in-domain pre-training inflates FSL results: ID-PT gains 33.41 pp over No-PT versus 23.75 pp for supervised OOD-PT (both before adaptation), a 9.66 pp optimistic bias averaged over eight target datasets.
- Episodic adaptation on the support set benefits OOD-pre-trained models far more (7.69 pp average, growing with k) than ID-pre-trained ones (1.37 pp, negative at 1-shot), indicating ID pre-training absorbs the gains adaptation would otherwise provide.
- Source labels are largely unnecessary for OOD pre-training: the augmentation-based UIAug strategy reaches a 27.71 pp mean gain with adaptation versus 27.97 pp for supervised OOD-PT.
- A factor-analysis composite of descriptors computable before pre-training selects OOD sources within a median 1.37 pp of the oracle, outperforming every individual descriptor family (all above 8 pp median gap).
Strengths
Systematic full-factorial design
All source-target combinations across eight public datasets, three architectures, 5/20-way and 1/5/10-shot, 4,000 test episodes, five bootstrap runs with re-drawn class-disjoint splits (Sect. 4.2); code and configs released at github.com/Alejandro-Galan/fsl_critique.
Clear conceptual formalization
Sect. 3.2-3.3 define transfer gain and optimistic bias (Eqs. 5-6) on top of a domain-adaptation-inspired decomposition (Eq. 4); the IDA asymmetry (7.69 pp under OOD-PT vs 1.37 pp under ID-PT, Sect. 5.1.2) empirically supports the framework.
Practically valuable label-free result
Quantified source-selection contribution
Honest protocol disclosure
Weaknesses
Outdated architecture pool, no modern baselines
ID-vs-OOD bias estimate is confounded
Descriptive-only statistics for headline claims
Weak No-PT anchor inflates pre-training narrative
Unified 40x40 resolution limits external validity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether the canonical few-shot learning (FSL) protocol — pre-training on class-disjoint but visually in-domain splits (miniImageNet, Omniglot, CIFAR-FS) — genuinely measures low-data learning. Section 3.2 formalizes three regimes (No-PT, ID-PT, OOD-PT); Section 3.3 defines a risk-based transfer gain and optimistic bias (Eqs. 5-6) via a domain-adaptation-inspired decomposition (Eq. 4). Two practical contributions follow: label-free OOD pre-training (UCKM clustering pseudo-labels and UIAug instance-augmentation episodes, Sect. 3.4) and pre-training-time source selection from domain-similarity descriptors (Sect. 3.5). Central claims: (1) ID-PT beats No-PT by 33.41 pp on average versus 23.75 pp for supervised OOD-PT across eight targets — a 9.66 pp optimistic bias; (2) support-set adaptation (IDA) yields 7.69 pp under OOD-PT but only 1.37 pp (−0.45 at 1-shot) under ID-PT; (3) UIAug (27.71 pp) matches supervised OOD-PT (27.97 pp) with IDA; (4) factor-based source selection reaches a 1.37 pp median oracle gap.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the optimistic bias of standard few-shot learning benchmarks according to this paper?
The authors find that pre-training on class-disjoint classes drawn from the same dataset as the target classes — the standard protocol on miniImageNet, Omniglot, and CIFAR-FS — yields an average accuracy gain of 33.41 percentage points over no pre-training, versus 23.75 points for out-of-domain pre-training. That 9.66-point difference is the optimistic bias attributable to domain overlap, averaged over eight datasets, three architectures, and multiple way-shot settings.
Do you need labeled data for pre-training in few-shot learning?
Mostly no, according to this study. Its augmentation-based UIAug strategy treats each unlabeled source image as its own pseudo-class (augmented views form the support set, originals the query set) and reaches 27.71 pp mean gain with support-set adaptation, nearly matching supervised out-of-domain pre-training at 27.97 pp. A clustering-based alternative (UCKM with 1,000 clusters) performs somewhat worse at 24.98 pp.
How can I choose which dataset to pre-train on for a few-shot task?
The paper proposes computing domain-similarity descriptors before any training — dataset statistics, DINOv2 and few-shot-encoder feature-space overlap, distance-based compactness, and autoencoder reconstruction error — and combining them via factor analysis. This composite score selects sources within a median 1.37 pp of the best possible (oracle) source, whereas any single descriptor family leaves a median gap above 8 pp.
Is the code and data for this paper available?
Yes. The paper states that source code and configuration files to reproduce all experiments are publicly available at github.com/Alejandro-Galan/fsl_critique, and all eight datasets (miniImageNet, Omniglot, CIFAR-FS, OrganAMNIST, Capitan, Egyptian, Greek, TKH) are public. Note that all images are rescaled to 40x40, so absolute numbers are not comparable to standard benchmark reports.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-13 are bar/box plots whose exact values are not recoverable from text extraction; per-dataset numbers cited in this review come from the in-text prose and could not be checked against individual bars.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.