Beyond Overlap: Estimating the Causal Effect of Benchmark Exposure
cs.CL
Quick verdict
LeakScale causally estimates benchmark-exposure score effects: +7.17–27.31pp in all 4 cells; rigorous, though stylized exposure limits generality.
Summary
The paper introduces LeakScale, an interventional framework that estimates how much controlled exposure to benchmark material changes a benchmark score, as distinct from detecting whether exposure occurred. Each of 2,048 synthetic task families (32 structural shapes across SQLite and deterministic Python domains) pairs a public executable task with a private, non-derivable key-to-value association. Every candidate answer is pre-compiled and executed against hidden fixtures, and a family is released only if exactly one candidate achieves perfect fixture agreement while alternatives score at most 0.75. Models Qwen3-4B-Instruct-2507 and Gemma 4 31B are LoRA-adapted either with (C2, 512 families per domain) or without (C0, 512) the associations, and a difference-in-differences over base/adapted states with 32 seeded generations per family isolates exposure-attributable change from adaptation drift across 262,144 total generations. Exposure raises accuracy in all four model-by-domain cells: +7.17 (Qwen SQL), +12.92 (Gemma SQL), +10.14 (Gemma Python), and +27.31 percentage points (Qwen Python); family-clustered bootstrap CIs exclude zero and the unweighted mean is +14.39 points. Control-condition accuracy drifts — often downward — after adaptation, so naive before-after comparisons would misattribute gains. Limitations: base accuracy sits near the 25% chance level, so the instrument largely measures trained-association recall; exposure is a single high-dose synthetic regimen on two models with one adapter realization per cell; and no public code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Controlled exposure to private benchmark key associations raises four-choice executable accuracy in all four model-by-domain cells, by +7.17 to +27.31 percentage points, with family-clustered bootstrap CIs excluding zero.
- Provenance detection and contamination consequence are distinct estimands: the balanced no-key control shows adaptation drift (−2.34 pp average) that a naive before-after comparison would misattribute to exposure.
- Exposure sensitivity is heterogeneous across model-domain pairs (a nearly fourfold range), so a universal penalty for anything labeled contaminated would be inappropriate.
- Binding tasks to necessary, non-derivable private information with certified executable counterfactuals makes the contamination counterfactual directly measurable.
Strengths
Clean identification strategy
Section 3 combines a necessary, non-derivable private key (§3.1), exhaustive counterfactual certification of all four candidates on hidden fixtures (Eq. 4/6: exactly one candidate at utility 1.0, alternatives ≤0.75), and a deterministically balanced no-key control (16 C0/16 C2 per shape with exact truth-rank and position balance), foreclosing positional or preference confounds.
Verifiable internal numerics
Table 11's integer counts exactly reproduce Table 2's percentages (e.g., Qwen SQL C2 adapted 5,459/16,384 = 33.32%) and the DiD recomputes to the reported +7.17 pp — strong evidence the reported statistics derive from the logged outcomes.
Exemplary reproducibility ledger
Honest scoping and labeled analyses
Thorough crossed robustness
Weaknesses
Instrument operates at chance level, so the headline is near-tautological
No dose-response; single high-dose regimen
Single training realization per cell
DiD assumptions untestable with two time points
No stated public artifact release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper argues that contamination research conflates two questions — *did benchmark material enter training?* (provenance) and *how much did that contact move the score?* (consequence) — and proposes LeakScale, an interventional instrument that measures the second quantity directly. Each synthetic task family pairs a public executable task with a private key→value association absent from, and non-derivable via, all public material (§3.1). Because answering requires the association, exposure becomes a treatable input, and the causal effect is a difference-in-differences (DiD) contrast (Eq. 3): the before-to-after change for exposed families (C2) minus the same change for matched no-key families (C0). Central claims: (i) exposure raises accuracy in all four model×domain cells (+7.17 to +27.31 pp); (ii) naive before-after comparisons misattribute adaptation drift (C0 moves −2.34 pp on average, §5.2); (iii) provenance audits and consequence estimation are complementary. §4.3 scopes claims carefully: controlled post-training exposure on two small models, not web-scale natural contamination.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LeakScale and what does it measure?
LeakScale is an experimental framework that measures how much a model's benchmark accuracy changes when it is deliberately given access to benchmark-specific answer keys during fine-tuning. Unlike contamination detectors that only ask whether training data overlapped a benchmark, LeakScale builds synthetic tasks that cannot be solved without a private key, gives that key to some fine-tuning runs and withholds it from matched controls, and reports the difference — the causal score effect of exposure.
How large is the contamination effect reported in this paper?
Across two models (Qwen3-4B-Instruct-2507 and Gemma 4 31B) and two executable domains (SQLite and deterministic Python), controlled exposure raised four-choice task accuracy by +7.17 to +27.31 percentage points relative to a no-key control group, with all 95% confidence intervals excluding zero and an unweighted mean of +14.39 points, computed over 262,144 total generations.
Does this paper prove that real-world benchmark contamination inflates LLM scores?
Not by itself. The study uses a synthetic, high-dose exposure regimen (a private association repeated 32 times in a small fine-tuning corpus) on two models whose baseline accuracy is near the 25% chance level. The authors explicitly state the results do not estimate effects of naturally occurring web-scale contamination; the paper shows the causal quantity can be measured, not its magnitude in the wild.
Why did the control group's accuracy drop after fine-tuning?
Adapting on the background corpus moved the no-key (C0) group's accuracy down in three of four cells (for example, Qwen Python fell from 25.55% to 19.81%), an average drift of −2.34 points. This adaptation drift is exactly why the paper uses a difference-in-differences design: a naive before-after comparison on exposed tasks would misattribute drift as a contamination effect.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are referenced but are schematic diagrams whose content cannot be verified from the extracted text.
- Several citations are dated 2025–2026 (e.g., Schaeffer et al. 2026; Gemma 4 technical report); their content could not be verified in this review.
- The review covers the full embedded text including Appendices A–F, but no external check of the claimed artifacts or digests was possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.