Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
Makar Ulesov, Vladislav Smirnov, Omar Ibrahim, Arsenii Bobovnikov
cs.CL · cs.AI · cs.CE · cs.SE
Quick verdict
Paired clean-control benchmark: LLM backtest auditors saturate recall but clean-flag rates span 79 points; sound design, items unreleased, no human baseline.
Summary
The paper reframes backtest auditing by LLMs as a calibration problem and introduces a 96-item paired benchmark: 48 clean/flawed pairs across 8 canonical flaw classes (look-ahead, survivorship, data snooping, leakage, unrealistic fills, regime overfit, p-hacked stops), three difficulty strata, and prose and code surfaces. Each clean control reverts one planted methodology edit while deliberately keeping protective constructs (purged splits, realistic costs, slippage) so pattern-matching auditors misfire. A deterministic scorer separately measures flaw recall, clean-control false-positive rate, evidence localization, fix relevance, and utility U(λ), with Wilson 95% CIs, over 1440 audits from four endpoints (DeepSeek V4 Flash primary; GPT-4o mini as budget anchor). DeepSeek reaches 100% closed and clean-aware code recall, but open prompts flag 93.8% of clean code controls; the clean-aware warning drops DeepSeek code FPR from 20.8% (CI 11.7–34.3) to 0.0% (0.0–7.4) at unchanged recall, while GPT-4o mini still flags 79.2% of clean controls under the same prompt. Reviewer persona moves calibration, not recall, and self-reported confidence anti-correlates with calibration. Limitations: 6 items per class, synthetic strategy skeletons, a lexical fix scorer, single temperature-0 calls, an uncompleted human baseline, and no stated code or data release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 16 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Flaw recall alone is a misleading audit metric: three of four tested endpoints tie at 100% code recall while their clean-control false-positive rates span 79 points (0.0% to 79.2%, Table 2).
- A one-line clean-aware warning drops DeepSeek's code false-positive rate from 20.8% (Wilson 95% CI 11.7–34.3) to 0.0% (0.0–7.4) at unchanged 48/48 recall, but GPT-4o mini still flags 38/48 clean controls under the same prompt.
- Open prompts (no taxonomy) over-flag 93.8% of clean code controls while missing 13/48 flawed items, so the flagged pile contains more clean than rigged strategies.
- Residual audit failure concentrates in fix relevance, not detection: clean-aware code all-three specificity is 87.5% despite 100% evidence localization (Sec. 3).
- Reviewer persona and self-reported confidence move calibration the wrong way: adversarial roles account for 11 of 12 false positives and lower-confidence flags are disproportionately on clean controls (Table 3).
Strengths
Paired clean-control design targets the right confound
Sec. 2 and Fig. 1: controls are not unedited copies — they embed protective constructs (purged/embargoed splits, one-way costs, half-spread slippage, non-overlapping labels) so auditors that pattern-match on 'good practice' phrases misfire in both directions. Fig. 1 shows the same open audit correctly citing L18–L19 while falsely accusing L10/L09.
Deterministic, multi-axis scorer with no LLM judge
Sec. 2: evidence credit requires the cited span to hit planted target lines; fix credit requires class-specific keywords; all-three specificity and U(λ)=R̄−λF for λ∈{0.25,1,2} are reported separately; 1440/1440 calls parse. This removes judge-model noise from every headline number.
Statistically explicit headline claim
Operationally meaningful and honest negative results
Scope discipline
Weaknesses
Very small per-class sample and no run-level variance
Human baseline prepared but never run
Training-data contamination unexamined
Headline residual failure sits in the scorer's least reliable channel
No release statement for the benchmark
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This benchmark paper asks a narrow, well-chosen question: can an LLM auditor of quantitative-finance backtests distinguish rigged code from legitimate code, or does it flag anything suspicious-looking? The authors build a 96-item paired benchmark — 48 clean/flawed pairs, 8 flaw classes × 6 items, three difficulty strata, prose and code surfaces — and a deterministic scorer separating flaw recall, clean-control false-positive rate, evidence localization, fix relevance, and utility U(λ)=R̄−λF. Central claims (Abstract, Sec. 3): (1) recall saturates while calibration varies wildly — three of four endpoints tie at 100% code recall yet span 79 FPR points (Table 2); (2) a one-line clean-aware warning drops DeepSeek's code FPR from 20.8% (Wilson 95% CI 11.7–34.3) to 0.0% (0.0–7.4) at unchanged 48/48 recall; (3) the residual failure lives in the fix column, with clean-aware code all-three specificity at 87.5% despite 100% evidence localization. Scope disclaimers are exemplary: the Discussion and Ethics Statement repeatedly state the benchmark 'does not certify trading systems.'
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a clean control in this benchmark?
A methodologically sound twin of a rigged backtest: it keeps the same strategy, dates, code style, labels, and reporting as the flawed item but reverts the single planted flaw. Crucially, clean controls deliberately include good practices (purged and embargoed splits, realistic one-way costs, half-spread slippage, non-overlapping labels), so an auditor that pattern-matches on such phrases gets caught falsely flagging legitimate code.
Why do the authors say recall-only reporting is misleading for LLM auditors?
Because in their experiments three of the four tested models tie at 100% code recall, yet their false-positive rates on clean strategies range from 0.0% (DeepSeek) to 79.2% (GPT-4o mini) under the same clean-aware prompt — a 79-point spread that a recall-only leaderboard would completely hide. False positives translate directly into wasted reviewer threads on legitimate submissions.
Which models were tested and what was the headline result?
DeepSeek V4 Flash (primary auditor), GPT-4o mini (budget anchor), GPT-4.1 mini, and Gemini 2.5 Flash Lite, over 1440 audits at temperature 0. Adding a 'clean-aware' warning to the prompt cut DeepSeek's clean-code false positives from 20.8% to 0.0% with recall unchanged at 100%, while GPT-4o mini still flagged 79.2% of clean controls under the identical prompt.
Does a high score on this benchmark mean an LLM can validate a real trading strategy?
No. The authors state explicitly that all backtests are synthetic, the benchmark isolates one review skill (recognizing planted methodology flaws under matched controls), it does not certify any trading system, and the paper makes no claim about profitability or investment soundness.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–3 are only partially recoverable from the extracted text (axis labels and legend fragments); all quantitative judgments are taken from Tables 1–4 and body text.
- Only one example item pair (Figure 1) is included in the text, so item quality and difficulty calibration across strata cannot be independently judged.
- Several cited works are 2026 arXiv preprints (Wang, Li, Chaudhuri, Ho, Zurawicki et al.) whose positioning claims could not be verified from the provided text.
- The decomposition 1440 = 96 × (6 primary conditions + 3 ladder models × 3 prompts × code surface) is inferred; the paper does not spell out the condition arithmetic.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.