Compile by Training: Turning Natural-Language Specifications into Local Neural Functions
Yuntian Deng, Pengyu Nie, Stuart Shieber
cs.CL · cs.AI · cs.LG
Quick verdict
Turns NL specs into finetuned LoRA functions on a 0.6B interpreter; FuzzyBench-Hard LEM 0.224→0.836, but eval subset is baseline-failure-selected and no CIs
Summary
The paper introduces compile by training, a system that turns a natural-language specification of a text-to-text function into a reusable neural function: teacher LLMs (GPT-5.4-mini and GPT-5.5 in a 2:1 mixture) synthesize input-output examples from the specification, and those examples fine-tune a rank-64 LoRA adapter on a frozen Qwen3-0.6B interpreter, warm-started from an amortized compiler. The packaged artifact runs locally without teacher calls. On FuzzyBench-Hard, a subset selected as the specifications where the fast Program-as-Weights compiler produced no exact matches, mean LLM Exact Match rises from 0.224 to 0.836 at roughly 50.9 s versus 3.5 s compile time. Sweeps report 0.746 to 0.851 from teacher mixing and 0.821 to 0.866 from scaling data to 7,200 pairs. The GPT-5.5 judge reaches 0.977 agreement with 128 author labels (kappa 0.946). Deployed applications include a multi-site website helper with 28 active programs, a 3D avatar passing 43/44 validation instructions, and a live translator that served 100,747 requests in 12 days. Main limitations: evaluation on a baseline-failure-selected subset with no full-benchmark control, no variance or significance reporting, missing few-shot baselines, and judge-teacher circularity.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Natural-language specifications can be compiled into reusable neural functions by teacher-synthesized supervision followed by LoRA specialization of a frozen Qwen3-0.6B interpreter.
- On FuzzyBench-Hard, compile by training raises mean LEM from 0.224 (amortized PAW) to 0.836, at ~50.9 s versus ~3.5 s compile time.
- Supervision choices matter: a 2:1 GPT-5.4-mini/GPT-5.5 mixture lifts LEM from 0.746 to 0.851, and scaling to 7,200 pairs reaches 0.866 (Table 1).
- Compiled functions are practically deployable and composable: 28 active programs serve four live websites, an avatar program passes 43/44 validation instructions, and a live translator served 100,747 requests in 12 days.
Strengths
Large, cost-honest headline gain
Fig. 4 / §5.2: mean LEM on FuzzyBench-Hard rises 0.224→0.836 (+0.612) at 50.9 s vs 3.5 s compile time; the paper states the accuracy/cost tradeoff explicitly rather than hiding compile cost.
Unusual rigor for an LLM-judge metric
Appendix B/E: the GPT-5.5 LEM judge is validated against 128 author labels (accuracy 0.977, κ=0.946, FPR 0.025, FNR 0.023), defaults to INCORRECT when unsure, and the full grader prompt is published.
Interactive-build engineering with measured latency
Deployment evidence is concrete, not hand-wavy
Complete public recipe and prompts
Weaknesses
Evaluation set selected by baseline failure
No uncertainty quantification
Missing no-training baseline
Teacher-judge circularity
Sweep protocol shifts confounds
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a systems/method paper extending Program-as-Weights (PAW; Zhang et al., 2026, arXiv:2607.02512). It replaces PAW's one-shot amortized compiler with a two-stage build: teacher LLMs (GPT-5.4-mini and GPT-5.5 in a 2:1 mixture) synthesize supervised input-output pairs from a user-written natural-language specification (§3.1, Appendix D), then a rank-64/alpha-16 LoRA adapter for a frozen, quantized Qwen3-0.6B interpreter is fine-tuned for 100 steps at batch 48 on 2,400 unique pairs expanded to 6,400 examples, warm-started from the amortized prediction (§3.2, Table 2). The packaged `.paw` artifact (adapter + scaffold + metadata) runs locally without teacher calls. Central claims: (1) the extra compile-time investment lifts mean LLM Exact Match on FuzzyBench-Hard from 0.224 to 0.836 (+0.612, Fig. 4); (2) a minute-scale build (50.9 s vs 3.5 s) remains interactive; (3) compiled functions compose in deployed applications.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'compile by training'?
A technique that turns a natural-language description of a text-processing function into a small, self-contained model. Large teacher models first generate example input-output pairs from the description; those pairs then fine-tune a lightweight LoRA adapter on a frozen 0.6B-parameter language model. The result runs locally like a compiled program, with no further calls to the large teachers.
How accurate is it compared to the fast Program-as-Weights compiler?
On FuzzyBench-Hard, it scores 83.6% semantic accuracy (LLM Exact Match) versus 22.4% for the fast compiler — but this subset consists precisely of tasks where the fast compiler produced no exact matches. Compilation takes about 51 seconds versus 3.5 seconds for the fast compiler.
Do compiled functions still need the large teacher models at run time?
No. Teachers are only used once, at compile time, to generate training data. At run time the function is a small local model (Qwen3-0.6B plus a per-function adapter), so inputs are processed offline without provider API calls.
What are the paper's main limitations?
The evaluation uses only the subset where the baseline failed, with no full-benchmark comparison; results are single point estimates without variance or significance testing; there is no cheaper few-shot baseline without finetuning; the judge model shares a family with the teachers; and the applications are author-built demos rather than systematic user studies.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures appear only as extracted caption/label text (e.g., Fig. 4's bar chart); visual content could not be inspected, though all cited numbers also appear in body text and tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.