Can Coding Agents Reproduce Official Statistics? Metadata, Retry Budget and the Limits of Execution Feedback in a Controlled Eurostat Benchmark
cs.LG · cs.AI · cs.CL · cs.SE
Quick verdict
Controlled 360-run benchmark claims metadata and retry budget—not execution feedback—drive LLM Eurostat accuracy; well-designed, rigorous, but small-scale.
Summary
This controlled benchmark asks whether authoritative metadata, execution feedback and retry budgets improve a coding agent's ability to produce statistically correct Eurostat answers. Thirty natural-language tasks spanning seven frozen Eurostat datasets and four difficulty tiers were run under four conditions with Claude Sonnet 5 generating Python: task only (A); task plus a frozen metadata card (B); metadata plus a repair loop driven by sanitized diagnostics with up to three attempts (C); and metadata with the same three-attempt budget but no diagnostics, failed attempts simply redrawn (D). Three replicates gave 360 task-runs scored on six-component exact correctness. Exact correctness rose from 27.8% (A) to 51.1% (B; +23.3 points, Holm-adjusted McNemar p=0.0049) and reached 76.7% under both C and D, whose paired difference was exactly zero (twelve discordant pairs each way; CI -12.2 to +13.3 points; p=1.000). A companion run under an under-specified output contract had understated condition C by 23.4 points and reversed a significance conclusion. Limitations include a single proprietary model tested on two dates, only 30 task clusters, a null that cannot establish equivalence, and harness artifacts such as fifteen static-checker rejections in condition B. The replication package is on Zenodo.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Frozen authoritative metadata cards raise exact correctness from 27.8% to 51.1% (+23.3pp, Holm-adjusted McNemar p=0.0049) and cut wrong-filter outcomes from 39 to 4 out of 90.
- A three-attempt budget raises exact correctness to 76.7%, and sanitized execution diagnostics contribute exactly zero beyond that budget (C vs D: twelve discordant pairs each way, p=1.000, CI -12.2 to +13.3pp).
- An under-specified output contract understated condition C by 23.4pp and reversed a significance conclusion, so evaluator and contract design can dominate measured agent error.
- Even in the best conditions roughly one executable answer in five (20.7-23.3%) was silently wrong, making execution success an unreliable proxy for statistical validity.
Strengths
Budget-matched control isolates feedback from retries
Condition D resends the unchanged request within the same three-attempt budget, making C-vs-D a clean test of diagnostic content: 69/90 vs 69/90 exact, twelve discordant pairs each way, p=1.000. The confound Olausson et al. (2024) flag in self-repair work is here actually resolved by design (Table 1, Section 3.2).
Statistical reporting rare in LLM benchmarks
Six-component exact criterion, Wilson CIs, exact McNemar tests with Holm correction across six contrasts, and task-cluster bootstrap intervals (50,000 resamples, seed 20260831) respecting the 90-pairs-within-30-tasks structure. Table 4 taxonomy rows each sum to 90 and reconcile with Table 2 execution counts.
Evaluator underspecification quantified, not hidden
Strong reproducibility infrastructure
Transparent error taxonomy including harness faults
Weaknesses
Equivalence claim outruns the statistics
Single model, two dates
Harness confound inside the headline retry effect
Small, tier-skewed task set
Between-experiment contract estimate
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper decomposes a claim most agent evaluations leave bundled: when a coding agent improves on a data-retrieval task, was it the information it was given, the number of tries, or the diagnostics it saw? Using 30 natural-language Eurostat tasks (7 datasets, 4 difficulty tiers), with Claude Sonnet 5 generating Python under four regimes — task only (A), plus a frozen metadata card (B), plus a sanitized-diagnostic repair loop with three attempts (C), and the same budget with zero diagnostics (D) — across three replicates (360 task-runs), it claims four things: metadata lifts exact correctness from 27.8% to 51.1% (+23.3pp); the retry budget lifts it to 76.7%; diagnostics contribute exactly 0.0pp beyond that budget (twelve discordant pairs each way); and an under-specified output contract had understated condition C by 23.4pp while reversing a significance test. Scope is deliberately narrow: every task names its dataset, the model is proprietary and singular, and the diagnostics are deliberately thin operational messages.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
In a 360-run controlled benchmark on Eurostat tasks using Claude Sonnet 5, adding a frozen metadata card raised exact correctness from 27.8% to 51.1%, and granting a three-attempt retry budget raised it to 76.7% — but a control group with the same budget and no error diagnostics scored identically (76.7%), so the diagnostics themselves added nothing measurable beyond the retries.
Which model was used and how many runs were performed?
Claude Sonnet 5 via the Anthropic Messages API generated Python for 30 natural-language tasks under 4 conditions in 3 independent replicates — 360 task-runs total, completed 2 September 2026, scored against gold answers derived from frozen Eurostat JSON-stat snapshots from 31 August 2026.
Is the code and data available?
Yes. The replication package is archived on Zenodo: doi 10.5281/zenodo.22253054 is the version used for the paper and 10.5281/zenodo.22253053 always resolves to the latest. It includes MIT-licensed code, CC BY 4.0 data, frozen snapshots, raw records, randomization seeds, and a single evaluation script that regenerates every reported table.
What is a 'silent error' in this paper?
An answer whose generated program runs successfully and emits valid output but is statistically wrong — wrong dataset, filters, result shape, values or unit. Even in the best experimental conditions, 20.7-23.3% of executable outputs were silently wrong, which is why the authors argue a successful execution is not proof of a correct statistic.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 appears in the extracted text only as a caption; its plotted content is fully redundant with Table 2, so no assessment depends on the missing chart image.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.