Harness as a Language: A Minimalist Agent Framework With Maximal Expressivity
Zhening Li, Joshua Liu, Mateja Vukelic, Nicole Shen, Supriya Lall, Amitayush Thakur, Alex Zhang, Omar Khattab, et al.
cs.AI
Quick verdict
JAZ's 'invoke' loop (all inputs + history as REPL variables) rivals specialized memory/self-improvement harnesses on 2 benchmarks at lower cost, but margins are thin and untested statistically.
Summary
This MIT CSAIL framework paper presents JAZ, built on a single LLM-based primitive: invoke, a function whose body the model writes at call time. Augmenting any Turing-complete language with invoke (formalized via lambda calculus in Appendix A) yields a loop with two defining properties: the model writes arbitrary executable code including recursive invoke calls (subagents), and everything the model sees — all inputs and the REPL history — is a manipulable variable in the code environment. The authors argue that this minimal loop, given task-agnostic prompting only, replaces capabilities usually engineered as external harnesses. Two evaluations support the thesis: on StuLife's far-recall subset with GPT-5.4 nano, JAZ invoke scores 69.9% versus 61.8% for Letta (MemGPT) at under half the cost (~$18.3 vs ~$42.1); on the full 939 graded tasks the margin is 72.6% vs 70.9% — near the error bars. Across all 417 AppWorld test-challenge tasks it reaches 74.2% TGC versus ACE's 69.9% at lower cost ($20.9 vs $30.6 total; meta-agent $9.9 vs $17.1), with ACE cost-restricted to training on 42 tasks. Strengths: principled derivation, controlled ablations, full per-run reporting. Weaknesses: no significance tests; all ablations are author re-implementations; the Letta baseline required environment modifications and prompt selection; reliance on built-in hooks and elaborate task-agnostic guidance prompts strains the minimalist framing.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- invoke is a novel LLM-based language primitive — a function whose implementation the model supplies at call time — from which a closed agent loop arises via tail recursion, generalizing CodeAct and RLMs.
- The invoke agent loop is distinguished by two properties: the model writes arbitrary executable code containing recursive invoke (subagents), and everything visible to the model — all inputs plus the REPL history — is a variable in the code environment.
- With task-agnostic prompting only, JAZ invoke matches or beats specialized long-horizon harnesses: 69.9% vs Letta's 61.8% on StuLife far-recall (GPT-5.4 nano) at under half the cost ($18.3 vs $42.1).
- With a GPT-5.4 meta-agent and GPT-5.4 nano solver, JAZ invoke achieves 74.2% TGC on AppWorld test-challenge (417 tasks), outperforming ACE (69.9%) at lower cost, without any hand-designed self-improvement workflow.
- Agentic capabilities such as memory and self-improvement can emerge from the expressivity of a sufficiently powerful computational substrate rather than from specialized external harness components (§6).
Strengths
Principled formalization of the agent loop as a language primitive
Definition 1 and Appendix A define invoke over any Turing-complete language (with a lambda-calculus formalization), and derive the closed agent loop as tail-recursive invoke; Properties 1–2 (arbitrary code with sub-invokes; all inputs plus REPL history as REPL variables) cleanly generalize CodeAct and RLMs rather than ad-hoc engineering.
Discipline in controlled comparisons
CodeAct and CodeAct+subagents baselines are re-implementations inside JAZ that remove exactly the two defining properties, with prompts kept byte-identical where possible (§4, Appendix C.1); the task/method decoupling of instructions vs guidance, the no-task-specific-guidance rule, and cache confound controls (fixed cache keys, ≤3 parallel runs) are unusually careful.
Transparent baseline forensics and full data disclosure
Mechanism-level evidence, not only aggregate scores
Weaknesses
No statistical significance treatment anywhere
Baseline and ablation sensitivity to author choices (principal-agent problem)
Cost comparison against ACE is not at parity
The 'no harness' framing is definition-dependent
StuLife is an unpublished, modified benchmark with a custom metric
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: *Harness as a Language: A Minimalist Agent Framework With Maximal Expressivity* (arXiv:2609.26891)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the JAZ agent framework?
JAZ is an open-source LLM agent framework from MIT CSAIL built around a single primitive called invoke: a function whose body the language model writes at runtime each time it is called. The model works in a Python REPL where every input to invoke, and the REPL interaction history itself, are ordinary variables the model can programmatically inspect and pass to recursive sub-invoke calls (subagents).
What does 'invoke' mean in this paper?
It is a language-level construct: augmenting any Turing-complete programming language with invoke creates a language where calling invoke(...) queries an LLM with a serialization of the named inputs, and the code the LLM returns becomes the one-time implementation of that function. Recursive/tail-recursive invoke calls induce a standard agent loop, with subagents as the default.
Does JAZ beat MemGPT/Letta on memory tasks?
On the paper's StuLife long-horizon benchmark (GPT-5.4 nano), JAZ invoke reached 69.9% pass rate on the far-recall subset (207 tasks) versus 61.8% for Letta (the newest MemGPT agent), at roughly $18.3 versus $42.1 per run. On the full 939 graded tasks the margin was smaller: 72.6% vs 70.9%, which lies near the reported error bars. No statistical significance tests were reported.
How does JAZ compare to ACE for self-improving agents?
On all 417 tasks of AppWorld's test-challenge split, JAZ invoke self-improved to 74.2% task-goal completion versus 69.9% for ACE (agentic context engineering) at lower cost (~$20.9 vs ~$30.6 per run; meta-agent $9.9 vs $17.1). Caveat: ACE was cost-restricted to training on only the first 42 tasks before freezing, while JAZ invoke adapted across the whole sequence; results are from 6 replicates with no significance testing.
Is the code and data available?
The paper links two public repositories: the framework at github.com/jaz-lang/jaz and the evaluation code at github.com/jaz-lang/jaz-evals. All prompts appear in the appendices and every replicate result is published in Tables 3–4. Full replication is nonetheless expensive (per-run costs of roughly $18–42 on StuLife) and depends on the proprietary GPT-5.4 model family.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text elides portions of §3 (dynamic scoping and hook mechanics), parts of Appendix C.3 (method details), and the middle of the system-prompt example in E.3; Figure 1–2 content was reconstructed from captions and surrounding text, not analyzed as rendered images.
- All headline numbers were cross-checked against Tables 1–4 (including the per-replicate Tables 3–4) and survived; conclusions are unlikely to hinge on the elided material.
- Replication cost is nontrivial: a single StuLife run costs $18.3–42.1 and the AppWorld CSI protocol spans six replicates; any sandboxed reproduction must be a scaled-down check, not exact re-derivation of headline numbers.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.