Are Coreset Selection Methods Worth Their Cost?
cs.LG · cs.AI · cs.CV
Quick verdict
Claims no coreset selector wins under matched wall-clock budgets; rigorous audited benchmark supports this, though ImageNet duels are partly single-seed.
Summary
This benchmark paper asks whether one-shot coreset selection for image classification justifies its own selection cost when selection time and downstream training are charged to a single wall-clock budget. The authors standardize training (ResNet-18, one recipe per dataset) across CIFAR-10, Tiny ImageNet, CUB-200, and ImageNet-1K, covering 11 selectors and over 1,500 released runs. In budget grids on CIFAR-10 and Tiny ImageNet, each selector may pick its best (fraction, epochs) operating point within eight wall-clock anchors; none is won by a sophisticated selector - winners are class-balanced random (Uniform), repeated random sampling (RS2), or full-data training. In ImageNet-1K duels at roughly 10 and 18 GPU-hours, full-data training with a compressed schedule is the most accurate point at both budgets (66.17% at 17.33h; 64.77% at 10.24h). A timing audit shows selection cost is dominated (95.3-99.8%) by a fraction-independent full-dataset scan that neither amortizes with smaller fractions nor extrapolates across datasets. Reuse analysis finds selection never breaks even on time against same-size random subsets, so its case rests on accuracy premiums (e.g., Herding +5.1pp on Tiny ImageNet at f=0.1). The paper also documents 9 correctness fixes to DeepCore, including a Herding argmax/argmin inversion worth about 5.9pp. Limitations: ResNet-18-only mainline, single-seed probes for most ImageNet points, image-classification scope, and peak-accuracy reporting without a validation split.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under matched wall-clock budgets, with each selector granted its most favorable (fraction, epochs) operating point, no sophisticated coreset selector wins any of eight anchors on CIFAR-10 or Tiny ImageNet; winners are Uniform, RS2, or full-data training.
- On ImageNet-1K at roughly 10-hour and 18-hour budgets, full-data training with a compressed schedule is the most accurate probed strategy (66.17±0.07 at 17.33h; 64.77±0.13 at 10.24h).
- Selection cost is dominated (95.3-99.8% depending on dataset) by a fraction-independent fixed full-dataset scan, so it cannot be amortized by selecting smaller fractions and does not extrapolate across datasets.
- Against a same-size random-subset baseline, sophisticated selection never breaks even on time at any reuse count K; its justification rests on accuracy premiums that are dataset- and fraction-dependent (Herding +5.1pp on Tiny ImageNet at f=0.1).
- The widely used DeepCore library contains 9 correctness bugs, one of which (Herding argmax instead of argmin) depresses CIFAR-10 accuracy at fraction 0.1 by about 5.9pp (66.36% to 72.24% when fixed).
Strengths
The protocol is maximally generous to selection, making the negative verdict hard to dismiss
Granting each selector its best (fraction, epochs) operating point within eight anchors (Section 3, E1), plus zero selection cost for the three unaudited selectors, stacks the deck in selection's favor - and it still loses everywhere. Robustness includes exact enumeration of all 27 seed assignments, 2,000-draw per-cell Unif(0.9,1.1) timing perturbation, an adversarial ±10% re-pricing that must reach 26% (CIFAR-10) or 57% (Tiny ImageNet) to flip an anchor, and last-epoch re-decision of all 16 anchors (Appendix D, tightest margin 0.05pp).
The cost audit yields a transferable mechanism, not just a leaderboard
Section 4.4 and Figure 4 show the fraction-independent floor (selector training plus feature extraction) is 95.3% (CIFAR-10) to 99.8% (Tiny ImageNet) of audited selection time, with fits like ≈19,518+339f seconds on ImageNet. This explains both why smaller fractions cannot amortize cost and why cost does not transfer across datasets (0.99-1.03 full-epoch equivalents on Tiny ImageNet vs 1.21-1.52 on CUB-200/ImageNet-1K for 40-epoch scorers).
Community-service artifact bundle with quantified fixes to a widely used library
Unusual transparency about the paper's own defects
Mechanistic explanation plus transfer tests beyond the main grid
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper reframes coreset selection evaluation from accuracy at a fixed subset size to accuracy under a shared wall-clock budget that includes selection time. Three claims: (1) an end-to-end protocol charging selection (Tsel) and training (Ttrain) to one audited clock, granting every method its best (fraction, epochs) operating point per budget anchor; (2) a reusable artifact bundle with corrected DeepCore code (9 documented fixes), timing-audit tools, and 1,500+ released runs of indices and raw CSVs; (3) an empirical verdict - no sophisticated selector wins any of eight anchors on CIFAR-10 or Tiny ImageNet, full-data-for-fewer-epochs wins the ImageNet-1K duels, selection cost is a fixed full-dataset scan, and only the reuse regime can justify selection, and only via accuracy. The claims are carefully scoped (fixed-subset select-then-train, image classification), though the abstract's sweep reads broader than the probed set in places.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Are coreset selection methods worth their cost according to this paper?
For one-shot training under a wall-clock budget, no. Across eight budget anchors on each of CIFAR-10 and Tiny ImageNet, every winner was class-balanced random sampling, repeated random sampling (RS2), or full-data training. On ImageNet-1K, training on all data for fewer epochs beat every selection strategy probed at both tested budgets. The paper argues selection only becomes attractive in reuse scenarios, where an accuracy premium (e.g., Herding +5.1pp on Tiny ImageNet at fraction 0.1) can justify the fixed selection cost.
What is the DeepCore Herding bug found in this paper?
The paper documents that the DeepCore implementation of Herding selects the sample farthest from the running surrogate mean (argmax) instead of the nearest one (argmin). Fixing this sign inversion raises CIFAR-10 accuracy at fraction 0.1 from 66.36% to 72.24% (about +5.9pp) under the paper's unified recipe, so accuracy numbers published with the original code may be systematically wrong for this widely used baseline.
Why is coreset selection so expensive?
The timing audit shows that nearly all selection time (95.3-99.8% depending on dataset) is a fixed, fraction-independent full-dataset pass: probe training, feature extraction, or per-sample scoring must touch every training example before any can be discarded. Choosing a smaller fraction shortens downstream training but barely reduces selection cost, and the absolute size of this floor does not transfer between datasets.
What should practitioners do instead of coreset selection?
The authors recommend class-balanced random subsets at the largest affordable fraction under tight budgets, full-data training with a compressed schedule under moderate or large budgets, repeated random sampling where the pipeline allows it, and reserving sophisticated selection for reuse regimes. They advise avoiding GraNd and EL2N at low fractions outright, where drops below random reach tens of points.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-9 appear in the extraction only as dense text fragments; envelope shapes and figure-only values could not be visually verified.
- Some appendix table layouts (Tables 4-7) are garbled by extraction; cited numbers were cross-checked against the prose where possible.
- The released artifact bundle (indices, CSVs, audit tables) is taken on trust from the paper's statements; the release itself was not inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.