The Imitation Game: When LLMs Learn to Reason Like Programs via Code-Centric Reasoning Data Synthesis
Jinyang Zhang, Weibin Liao, Keqin Bao, Sihang Li, Shaobo Wang, Muyang Ye, Hongxin Ding, Yue Fang, et al.
cs.CL · cs.AI
Quick verdict
Turning Codeforces into narrative execution-grounded reasoning data yields consistent gains, but duplicated baselines and no significance tests weaken rigor.
Summary
MIMIC is a data-synthesis framework that converts 10,008 Codeforces problems with accepted solutions into natural-language reasoning training data. It extracts each problem's mathematical kernel, wraps it in narrative contexts sampled from FineWebEdu (~20 variants per seed, ~200K questions), synthesizes test inputs from the accepted code's control-flow decision points, and produces chain-of-thought supervision by instrumenting the code so every reasoning step is grounded in deterministic execution traces. SFT (~114K samples) and GRPO (~10K prompts) on Qwen3-4B, Qwen3-8B, and DeepSeek-R1-Distill-8B yield consistent gains: GRPO lifts Qwen3-8B's general-reasoning average from 60.2 to 63.8 and math from 66.7 to 69.4, DeepSeek-R1-Distill-8B math from 54.5 to 64.2, and MazeBench from 22.0 to 40.0; raw-Codeforces training instead collapses BBH (78.8 to 44.1 under SFT). The proposed Code-Instrumented Reward (CIR) outperforms binary reward and PRM800K/Math-Shepherd in ablation. Key limitations: no seeds, variance, or significance tests are reported; headline 'Ours (GRPO)' numbers coincide exactly with the Binary-reward ablation row while CIR's better numbers appear only there; several Table 2 baselines (TeaR, CoE) are numerically identical to the paper's own raw-Codeforces ablation, undermining the comparison; and CIR is implemented via a Qwen3-4B judge model despite claims of supervision 'without external reward models'. Code and data are stated to be released.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Wrapping the executable logic of competitive-programming problems in natural narratives removes shortcut memorization while preserving verifiable reasoning structure, yielding transferable gains across general, mathematical, and fine-grained benchmarks.
- Trace-grounded CoT anchored to instrumented, execution-verified accepted code eliminates intermediate-step hallucinations in synthesized reasoning data.
- The Code-Instrumented Reward provides dense process supervision that outperforms binary outcome rewards and PRM800K/Math-Shepherd (65.2 vs 63.8 general avg on Qwen3-8B).
- Training on raw Codeforces statements induces a 'recall illusion' in which models reconstruct memorized templates, degrading general reasoning (BBH 78.8 to 44.1 under Qwen3-8B SFT).
Strengths
Execution-anchored supervision with verification-and-discard
Stage 4 instruments accepted code read-only ([TRACE] probes, App. A.4), executes it on synthesized inputs, verifies the filtered stdout against the gold answer, and discards mismatches. Correctness derives from judge-passed code rather than teacher-LLM output, directly attacking intermediate-step hallucination — the paper's most defensible methodological move.
Persuasive format ablation supporting the recall-illusion thesis
Table 4 and App. E: on raw Codeforces statements Qwen3-8B writes code in 38.7% of cases at 23.8% accuracy, versus 0.7% code rate and 42.2% accuracy in the fused format; Table 12 shows the fused-vs-refined gap widens with difficulty (delta +16.2 at rating >=2000), coherently matching the shortcut-memorization story.
Unusually complete ablation grid
Honest negative controls and candid limitations
Weaknesses
Baseline rows appear to be relabeled ablations
CIR's description contradicts its implementation
No variance or significance reporting
Headline-configuration ambiguity
In-distribution MIMIC benchmark and table inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: MIMIC (arXiv 2609.16076)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the MIMIC paper do in one sentence?
It converts 10,008 Codeforces competitive-programming problems (with their accepted solutions) into natural-language reasoning training data by extracting each problem's math kernel, wrapping it in story-like contexts from web text, generating test inputs from the code's control flow, and writing chain-of-thought solutions grounded in instrumented execution traces, then fine-tunes LLMs on this data with SFT and GRPO.
Does training on MIMIC data hurt coding ability?
The paper reports code benchmarks (HumanEval, MBPP, CruxEval) mostly stay stable or improve in aggregate: for example, Qwen3-8B with GRPO raises its code average from 44.6 to 48.0, and the appendix attributes an MBPP gain of +8.9 points to transfer from reasoning training. Individual code benchmarks fluctuate in both directions across settings, so the aggregate is the meaningful signal.
Is MIMIC's code and dataset publicly available?
The paper states code and data are available at github.com/zjy1298/MIMIC. The appendix prints most synthesis prompts verbatim (kernel extraction, story wrapping, narrative fusion, instrumentation, CoT synthesis), though the test-synthesis prompt is omitted, and the repository contents could not be verified from the paper text alone.
What is the Code-Instrumented Reward (CIR)?
A process reward for GRPO that scores how many of the reference solution's execution checkpoints a response reaches, as reward = k/K. Although advertised as needing no external reward model, the paper's appendix says a Qwen3-4B judge model determines which steps were completed by comparing the response against reference intermediate outputs — an internal contradiction reviewers should press on.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several sections were truncated in extraction (e.g., benchmark details around Table 7, part of the evaluation/appendix text); table cells may contain extraction artifacts.
- Figures (1-3, word clouds, training-dynamics plots) are not analyzable from the text.
- The duplicate-row numeric discrepancies cited as weaknesses could partly be extraction noise, though the 67.0-vs-component-mean mismatch is internal to a single row and likely a source error.
- The GitHub repository's actual contents (code/data release) could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.