PaperCompiler: Faithful Paper-to-Code Generation via Repository-Level Specification Compilation
Yunhao Liu, Hong Phuc Pham, Jaehong Yoon
cs.CL · cs.AI · cs.SE
Quick verdict
PaperCompiler cuts paper-to-code info loss via compiled specs; +13.8% ref-based fidelity vs PaperCoder, but LLM-judge-only eval, no sig tests, no code.
Summary
PaperCompiler is a paper-to-code framework that translates research papers into repository-level implementations by compiling paper-grounded evidence into explicit, machine-readable implementation specifications rather than free-form plans. The pipeline has three phases: Paper Grounding extracts an implementation blueprint plus a reference registry that preserves verbatim source material, tagging each item as paper-supported, inferred, externally delegated, or unresolved; Specification Compilation reconciles evidence into requirements with non-degradation constraints, assigns each requirement an owning file, and emits file-level contracts with producer-consumer dependencies; Constraint-Guided Repository Generation then generates files in topological order under these contracts. On 90 papers from Paper2CodeBench (ICLR/ICML/NeurIPS 2024 subsets) using o3-mini generation and o3-mini-high judging, PaperCompiler improves over PaperCoder: reference-free 4.562→4.777, P2C-Ex 4.535→4.728, and reference-based fidelity 3.647→4.152 (+13.8% relative), the largest gain; high-severity evaluator critiques fall from 13.2% to 6.1%. A ten-paper subset comparison also beats AutoP2C and AutoReproduce at higher token cost (1.71M vs 0.98M tokens). Ablations on nine papers show Reconciliation and Contracting matter most; notably, removing Context Slicing improves reference-free scores while hurting reference-based fidelity. Key limitations: evaluation is entirely LLM-judge-based with no execution-based validation, no statistical significance testing, small ablation samples, and no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Free-form plans and summaries passed between pipeline stages cause paper-specific requirements to be lost, weakened, or reinterpreted during repository generation.
- Compiling paper-grounded evidence into explicit specifications — with provenance tags, non-degradation constraints, per-file ownership, and producer–consumer dependencies — carries requirements through to their implementation locations.
- Under matched settings (MinerU inputs, o3-mini generation, o3-mini-high judging) PaperCompiler beats PaperCoder on 90 Paper2CodeBench papers across all three protocols, with the largest gain in reference-based fidelity (3.647→4.152, +13.8%).
- High-severity evaluator critiques fall from 13.2% to 6.1%, driven mainly by reductions in missing core components (12.3%→6.8%) and evaluation mismatches (13.4%→8.4%).
- Ablations attribute the benefit primarily to Requirement Reconciliation (−0.51 ref-based when removed) and File-Level Contracting (−0.46), with Context Slicing contributing less and only on the reference-based axis.
Strengths
Well-posed core formulation with explicit information-flow semantics
§3 formalizes generation as a controlled transformation (Eqs. 1–4): requirements carry evidence status (paper-supported/inferred/delegated/unresolved), an owner file ω(k) (Eq. 6), producer–consumer artifact flows π/Γ (Eq. 7), and per-file contracts S_i=(I_i,A_i,R_i,H_i,D_i) — a sharper account of cross-stage information loss than the free-form-plan critique alone.
Largest gains on the most discriminative protocol, with per-paper evidence
Reference-based fidelity rises 3.647→4.152 (+13.8%, Table 1 average), and Fig. 4 win ratios (82.1%/92.6%/88.9% ref-based across the ICLR/ICML/NeurIPS subsets) show the improvement is broadly distributed, not driven by outliers.
Unusually candid negative and inconvenient results
Above-average artifact disclosure for a systems paper
Cost accounting included
Weaknesses
Entirely LLM-judged evaluation with no execution or human validation
No statistical significance testing anywhere
Grounding accuracy — the system's core premise — is never measured
Potential train/test contamination undiscussed
Manual critique labeling without reliability measures
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PaperCompiler (arXiv 2609.02272, NTU Singapore) targets faithful translation of ML papers into repository-level code. Its diagnosis: prior paper-to-code agents (PaperCoder, AutoP2C, AutoReproduce) pass free-form plans or summaries between stages, so paper-specific requirements get dropped, weakened, or reinterpreted. The remedy is to treat generation as "specification compilation" (§3, Eqs. 1–4, Algorithm 1): ground evidence with provenance tags (paper-supported / inferred / externally delegated / unresolved), reconcile it into requirements carrying non-degradation and forbidden-substitution constraints, assign each requirement an owning file plus producer–consumer artifact flows, and generate files under file-level contracts in topological order. Central claims: (1) explicit specs carry requirements through to implementation locations; (2) consistent gains over PaperCoder across three protocols on 90 papers, largest under reference-based evaluation (3.647→4.152, +13.8%, Table 1); (3) high-severity critiques halve (13.2%→6.1%, Table 3); (4) ablations attribute the effect mainly to Reconciliation and Contracting (Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PaperCompiler?
PaperCompiler is a framework that turns machine-learning research papers into complete code repositories. Instead of passing free-form plans between pipeline stages, it compiles paper-grounded evidence into explicit implementation specifications — each requirement tagged with its evidence status (paper-supported, inferred, externally delegated, or unresolved), assigned to a responsible file, and protected by non-degradation constraints — then generates code files under those contracts.
How much better is PaperCompiler than PaperCoder?
On 90 papers from Paper2CodeBench (ICLR/ICML/NeurIPS 2024 subsets), using the same o3-mini generator and o3-mini-high judge, PaperCompiler improves over PaperCoder: reference-free score 4.562→4.777 (+4.7%), P2C-Ex 4.535→4.728 (+4.3%), and reference-based fidelity 3.647→4.152 (+13.8% relative) — the largest gain. High-severity evaluator critiques fell from 13.2% to 6.1%. Its token cost is higher: 1.71M vs 0.98M tokens per repository.
Does PaperCompiler actually run or test the generated code?
No. Evaluation in the paper is entirely LLM-judge-based: correctness and fidelity ratings on a 1–5 scale plus a manually labeled critique taxonomy. The authors state in their Limitations that the work does not establish executability or full reproduction of reported experimental results, and that specifications are generation-time guidance rather than correctness guarantees.
Is the PaperCompiler code publicly available?
The reviewed paper text contains no code-release statement. It builds on public components — the Paper2CodeBench benchmark, MinerU document parsing, and OpenAI o3-mini/o3-mini-high APIs — but the PaperCompiler pipeline itself (prompts beyond excerpts, orchestration code, intermediate artifacts) is not stated as released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1–3 inline label text is jumbled by PDF extraction; pipeline understanding here relies on §3 prose and Appendix E prompts, which are intact.
- Table 5 (per-case qualitative scores in Appendix D) has its score cells run together in extraction, so this review avoids citing those specific numbers.
- No truncation observed: the provided text runs complete from the abstract through references, both appendices, and the Appendix F.6 code excerpt.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.