Compile, Don't Memorize: A Context Compilation Architecture (CCA) for In-Context Learning
Jinhu Qi, Minda Hu, Wentao Zhang, Weiqiang Jin, Yanyu Chen, Junli Wang, Irwin King
cs.CL
Quick verdict
Context compilation (typed IR + code verifiers) lifts CL-bench pass rates up to +6pp; credible and well-ablated, but single-judge grading tempers confidence.
Summary
The paper argues that read-and-reason prompting fails on rubric-graded in-context learning because one forward pass must extract, plan, generate, and self-verify simultaneously. It proposes the Context Compilation Architecture (CCA): a Compiler converts each long context once into a typed JSON intermediate representation (rules with codeable flags, verbatim exact_terms, output_spec, tools, data profile); CodeGen synthesizes up to three Python verifiers; Reasoner-1 drafts with the IR as a checklist; and a violation-gated Reasoner-2 corrects when ≥2 violations appear. On CL-bench (1,899 tasks, 4 domains) with four open-weight models, CCA beats vanilla prompting and two long-context baselines (ReadAgent-P, Ctx2Skill) on every model's overall score: +6.0pp on Kimi K2.5 (15.4%→21.4%), +5.1pp on GLM-5, +2.7pp on DeepSeek-V3.2 (all McNemar p<0.01), but only +0.5pp non-significant on Qwen3-Next-80B. An ablation attributes the largest single share to IR-as-checklist injection (+2.37pp), not the correction loop. Costs are ~34K tokens/task (~2.9× vanilla), far below Ctx2Skill's ~190K. On LongBench-v2 pure CCA trails vanilla in aggregate (53.88% vs 57.85%), though an adaptive gate reaches 60.24%. Limitations: a single GPT-5.1 judge with moderate cross-judge agreement (κ 0.40–0.61), gains concentrated on rule-dense domains with losses on open-ended EDS tasks, and ablations run on one model only.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Compiling each context once into a typed JSON IR with auto-generated executable verifiers lifts rubric-graded ICL pass rates on every base model tested, up to +6.0pp on Kimi K2.5 (15.4%→21.4%, McNemar p<0.01).
- IR-as-checklist injection (F2) is the single most load-bearing component (+2.37pp of the +6.00pp on Kimi K2.5), exceeding verifier execution (+0.53pp) and correction (+1.28pp) individually.
- CCA beats gist-retrieval (ReadAgent-P) and self-play skill libraries (Ctx2Skill) on every (model, Overall) cell of Table 1 while spending ~5.7× fewer tokens per task than Ctx2Skill.
- The benefit is domain-selective and capacity-bounded: concentrated on rule-dense sub-categories (Legal & Regulatory +20.7 to +27.2pp), reversed on open-ended ones, and non-significant on Qwen3-Next-80B (+0.5pp).
- An adaptive gate on cca_meta.ir_ok converts CCA's LongBench-v2 aggregate loss (53.88% vs Vanilla 57.85%) into a +2.39pp gain (60.24%).
Strengths
Honest boundary and negative results
LongBench-v2 aggregate loss is stated up front (§5.4: pure CCA 53.88% vs Vanilla 57.85%), EDS loses on all four models (Table 1), and the Qwen3 null result is reported with z=0.38, p=0.70 rather than buried; the paper frames CCA as domain-selective (§5.4) and ships a ~30-line adaptive gate.
Best-practice component ablation
Table 3 isolates each F-code on the full 1,899 tasks: removing F2 (IR-as-checklist) costs −3.07pp, correction −1.28pp, F4+F6 −1.81pp; the build-up decomposition (+1.82 scaffold, +2.37 F2, +0.53 F4, +1.28 F6) sums exactly to +6.00pp and the F2×(F4/F6) interaction is derived in Appendix G.4.
Cost transparency unusual for harness papers
Truncation audit appropriate to the grading regime
Strong reproducibility artifacts
Weaknesses
All headline numbers rest on a single LLM judge
Asymmetric tuning between CCA and baselines
Aggregate lift masks narrow scope
Model-capacity moderator rests on one data point
Baseline absolute level unvalidated against published numbers
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets long-context, rubric-graded in-context learning, where one novel context (median 20K characters, max 247K on CL-bench) governs 1–12 downstream questions and a single missed criterion fails the whole task. It claims the read-and-reason paradigm is structurally overloaded, and that compiling each context once into a typed JSON intermediate representation (IR: rules with codeable flags, verbatim exact_terms, workflow, output_spec, available_tools, data_profile), synthesizing executable Python verifiers from it, and gating a correction call on ≥2 violations materially raises pass rates. Three research questions organize the work: effectiveness (RQ1), comparison with gist-retrieval and self-play baselines (RQ2), and boundary conditions across task-structure density and model capacity (RQ3). The novelty claim — first to treat ICL as a compilation problem — is carefully positioned against DSPy, code-as-reasoning, and verifier-synthesis work in §2.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Context Compilation Architecture (CCA)?
CCA is a pipeline that compiles a long, novel context once into a typed JSON intermediate representation (rules with must_do/must_not/conditional slots, verbatim exact terms, output spec, tools, data profile), automatically generates Python verifier code from it, drafts answers with the IR as an explicit checklist, and re-asks the model to fix only the flagged violations when two or more are detected.
How much does CCA improve results on CL-bench?
On CL-bench (1,899 long-context tasks graded against 5–20 rubric criteria each), CCA raised overall pass rates from 15.4% to 21.4% on Kimi K2.5 (+6.0pp), from 16.1% to 21.2% on GLM-5, and from 15.0% to 17.7% on DeepSeek-V3.2 (all p<0.01), while the smallest model, Qwen3-Next-80B, gained only a non-significant +0.5pp.
Does CCA work on every kind of task?
No. Gains concentrate on rule-dense sub-categories (e.g., Legal & Regulatory, +21 to +27pp); CCA loses to vanilla prompting on open-ended simulation sub-categories, on the EDS domain for all four models, and in aggregate on LongBench-v2 (53.88% vs 57.85%), although an adaptive gate that falls back to vanilla reaches 60.24% (+2.39pp).
How expensive is CCA compared to ordinary prompting?
About 34K tokens per task on Kimi K2.5 versus 11.9K for vanilla prompting (~2.9×) and about 1.6× wall-clock, but roughly 5.7× cheaper than the Ctx2Skill baseline (~190K tokens/task). A variant without the correction loop (CCA-V2) keeps about 79% of the gain (+4.72pp) at 22.7K tokens.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction interleaved Table 5 (full result matrix) and Table 6 (sub-category lifts) into Appendix D prose; all values are present and recoverable, but column alignment was reconstructed from surrounding text.
- Figure 2 and Figure 3 content appears only as embedded caption text; plotted point values beyond the labeled annotations are not recoverable, so this review cites only table values.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.