Is It Still Worth Training a Classical Model in the Era of LLMs? A Crossover Benchmark on Tabular Data
cs.LG · cs.AI
Quick verdict
Claims small frozen LLMs lose to trained models on tabular data (~6% crossover); a well-replicated, honestly caveated benchmark — largely convincing.
Summary
The paper asks when a frozen LLM stops being competitive with a trained classical model on tabular prediction, quantified as the labeled-data crossover N*: the training-set size where a power-law learning curve error(N)=aN^-b+c drops below the LLM's flat, training-free error. Measurements come from 126 graduate students who each independently evaluated three of 18 business datasets under eight prompting configurations (gpt-4.1-nano/mini × key-value vs natural-language serialization × zero- vs 5-shot), with 16–29 replications per cell; classical curves for six model families come from a companion study with 127 students. Even against an oracle-best LLM configuration, a trained model wins within available labels in 86% of cells (95% CI [76,94]), at a median crossover of ~5.9% of the training set; boosting and random forests are never asymptotically beaten. Few-shot error versus shots follows no power law (median exponent 0.019), and independent re-implementations vary with CV 0.148. A contamination probe shows the LLM depends on exact, recognizable feature values. Limitations: only small cheap models are tested, test sets cap at 100 rows, classification AUROC uses hard labels (probability elicitation adds a median +0.04 AUROC), the classical fits are classroom-produced, and the contamination probe covers six datasets with a headroom confound.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Even against an oracle choice of its best prompt configuration, a trained classical model beats the small frozen LLM using no more than the labels already on hand in 86% of (dataset, model) cells, with the observed crossover at a median ~5.9% of the training set.
- In-context few-shot examples do not behave like training data: error versus shot count follows no power law (median fitted exponent 0.019; only 10% of fits reach R²>0.8).
- Independent re-implementations of the same fixed protocol vary with a coefficient of variation of 0.148 in LLM error, so train-vs-prompt decisions should be reported as ranges.
- A factorial probe shows the frozen LLM's edge is value-dependent — readable-decile binning collapses it toward chance while a model trained on the same bins keeps accuracy — which suggests, without proving, the crossover is a conservative, LLM-favorable estimate.
Strengths
Decision-relevant formalization
The labeled-data crossover N* (Eq. 1: N* = (a/(e_LLM − c))^(1/b)) turns the vague prompt-vs-train debate into one computable boundary, reported both in absolute rows and as a fraction of the available training set (§3–4).
Unprecedented replication depth
126 students independently re-implemented a fixed protocol, yielding 16–29 replications per (dataset, configuration) cell (~3,000 evaluation cells); this enables a direct replication-variance estimate (CV 0.148, §6) and canonical-vs-student concordance (median |log10| ratio 0.118 over 393 cells, Figure 5b) that single-run LLM benchmarks cannot provide.
Conservative headline plus layered robustness
Honest negative and self-limiting results
Clean baseline attribution
Weaknesses
Narrow LLM scope vs. title generality
Hard-label AUROC understates LLM ranking skill
Classical side is classroom-produced from a different cohort
Few-shot sweep has unverified controls
Small test samples and post-hoc oracle selection
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reframes the "prompt vs. train" debate as one measurable quantity: the labeled-data crossover N*, defined as the training-set size where a classical model's power-law learning curve error(N)=aN^(-b)+c crosses a frozen LLM's flat, training-free error (Eq. 1). Scope is explicitly bounded: small production models (gpt-4.1-nano, gpt-4.1-mini) on 18 public business tables (10 classification, 8 regression; 303–53,940 rows), key–value or natural-language serialization, zero- or 5-shot prompting. Headline claims (§1, §4, §9): (1) even against an oracle-best LLM configuration, training wins within available labels in 86% of (dataset, model) cells (95% CI [76,94], dataset-cluster bootstrap); (2) median observed crossover ≈5.9% of the training set; (3) few-shot examples do not scale like training; (4) replication CV 0.148 means decisions should be reported as ranges; (5) the LLM's skill is value-dependent, plausibly making the crossover conservative. The text hedges these claims carefully — oracle framing, "probe, not proof" contamination language, and the cheap-LLM-regime limitation are stated plainly.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLMs beat gradient boosting on tabular data?
According to this paper, for small cheap models (gpt-4.1-nano/mini) on 18 standard business datasets, no: a trained model wins once you have even a small fraction of the available labels — median crossover ~5.9% of the training set — and gradient boosting and random forests were never asymptotically beaten by the frozen LLM; the only cells the LLM survives involve weak baselines like Lasso or Ridge.
How many labeled rows do I need before training beats prompting an LLM?
The paper estimates a labeled-data crossover N*: the training-set size where a classical model's learning curve drops below the frozen LLM's flat error. Across 70 reliable dataset-model cells, the crossover falls at a median ~5.9% of the available training set, and in 40% of cells training already won at the smallest labeled subset evaluated (a few hundred rows).
Is the LLM just memorizing these standard datasets?
The paper's probe shows the LLM depends on exact, recognizable feature values — replacing numbers with readable decile bins collapses it toward chance while a model trained on the same bins keeps its accuracy — but the authors do not claim proof of memorization; a dataset-fame (ubiquity) score observationally did not predict skill (Spearman ρ=−0.13, p=0.60), and the fame contrast was confounded with baseline accuracy.
Does few-shot prompting close the gap between frozen LLMs and trained models?
No, per this study: a 0-to-5-shot jump buys a one-time error reduction of about 0.06, but error versus shot count is essentially flat (median fitted exponent 0.019, 59% of curves effectively flat), so in-context examples do not scale like training data on these tasks.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were available only as lossy extracted text; curve shapes and some axis details (e.g., Figure 1) could not be fully verified visually.
- The companion classical learning-curve study (127 students) is cited but not included in the provided text, so its protocol could not be independently assessed.
- The released repo artifacts (aggregated curves, canonical crossover table) were not inspected in this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.