Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses
Cen Mia Zhao, Haibo Ruan, Wenjie Chen, Pei-fen Tu, Usman Abbasi, Joel Hesch
cs.AI
Quick verdict
PRISM's failure-routed harness search gives reliable held-out gains; credible protocol, but small run counts and a Telecom scoring caveat temper it.
Summary
The paper frames improving LLM tool agents without retraining as budgeted harness selection: search over prompts plus tool-boundary middleware around a fixed model, evaluated by an optimizer-agnostic protocol reporting mean held-out lift, worst-condition lift, repeatability (RR0), logged cost, and RelLift95(B), a bootstrap estimate of the 5th-percentile held-out gain of the harness selected under budget B using only pre-scorecard information. The protocol is instantiated with BetterHarness, MIPROv2, GEPA, and PRISM, a failure-routed evolutionary optimizer that clusters failures by root cause, routes repairs to prompt, middleware, or joint edit surfaces, and retains a Pareto frontier over gate pass rate and reliability. On BFCL multi-round (600-case pass1) and tau2-Retail/Telecom (74-case pass4), with gpt-5.4-mini as inner and claude-opus-4.7 as outer model, PRISM-MW achieves mean held-out lifts of 14.2, 14.9, and 10.1 percentage points with positive RelLift95 (12.0, 10.1, 7.8), while baselines occasionally peak higher but select brittle harnesses (e.g., BH-PO WorstLift -13.5 on Retail). A one-component-at-a-time ablation attributes the margin chiefly to failure-surface routing and the three-pattern edit constraint. Limitations include small run pools (at most 16 runs per arm), small tau2 scorecards, Telecom PRISM rows scored outside the benchmark-native pass4 path, and uncertainty intervals reported only for the PRISM arms.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Harness optimization around a fixed model (prompts plus pattern-constrained tool-boundary middleware) can be evaluated as a budgeted selection problem whose chosen harness's held-out reliability is measurable via RelLift95(B).
- PRISM-MW achieves the strongest selected-harness profile: mean held-out lifts of 14.2, 14.9, and 10.1 percentage points on BFCL multi-round, tau2-Retail, and tau2-Telecom, with positive RelLift95 (12.0, 10.1, 7.8) on all three.
- The component ablation (Table 3) attributes the margin chiefly to failure-surface routing and the three-pattern edit constraint, not to in-loop gate access or the cross-generation failure matrix.
- Unconstrained middleware editing increases selection risk — BetterHarness falls below its own prompt-only variant on both tau2 tasks — so the edit-pattern constraint, not middleware access alone, makes the surface safely searchable.
Strengths
Reliability-centric evaluation protocol
Section 3/4.5 define WorstLift, RR0, and RelLift95(B), a bootstrap 5th-percentile of held-out lift under pre-scorecard selection. Table 2 makes the value concrete: BH-PO hits MaxLift 23.0 on tau2-Retail yet WorstLift -13.5 and negative RelLift95 (-0.4) — peak discovery and selected reliability cleanly dissociated.
Thorough one-component ablation
Table 3 reverts exactly one attribute at a time (NoRoute, NoGate, NoMatrix, NoConstraint, NoCrossover). Routing and the edit-pattern constraint are the only first-order components on all three tasks; PRISM-NoGate (12.9 pp mean lift on Retail) empirically answers the 'PRISM wins via in-loop gate access' fairness objection.
Engineered fairness across optimizers
Unusual transparency about weaknesses
Reproducibility infrastructure
Weaknesses
Small run pools undermine tail estimates
Telecom scoring-path inconsistency
Cross-seed overlap violates run independence
Baseline provenance and configuration risk
No multiple-comparison handling, wide intervals
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether an automatic search procedure — not a single hand-tuned harness — can be trusted to improve multi-turn LLM tool agents without retraining. The optimized surface is deliberately scoped: system prompts plus tool-boundary middleware limited to three edit patterns (silent argument correction, error blocking, prerequisite blocking; Section 4.3). Contributions: (1) an optimizer-agnostic, budget-aware protocol whose headline statistic is RelLift95(B), the bootstrap-estimated 5th-percentile held-out lift of the harness selected under pre-scorecard information at budget B (Sections 3, 4.5, Figure 3); (2) PRISM, a GEPA-skeleton evolutionary optimizer whose analyst LLM clusters failures by root cause and routes each cluster to a prompt, middleware, or joint mutation slot, retaining a Pareto frontier over gate pass rate and reliability R = 1 − Stuck. Headline claims: PRISM-MW reaches mean held-out lifts of 14.2/14.9/10.1 pp on BFCL multi-round/tau2-Retail/tau2-Telecom with positive RelLift95 (12.0/10.1/7.8), and Table 3 attributes the margin to routing plus the edit-pattern constraint.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RelLift95 in this paper?
RelLift95(B) is a budgeted reliability metric: the estimated 5th-percentile held-out gain of the harness you would end up with if you could afford several optimizer runs under budget B and picked one using only pre-scorecard (gate) information, estimated by bootstrapping the observed optimizer runs. It answers 'how much lift does the unlucky (5th-percentile) pick deliver', not the average.
What is PRISM and how much does it improve LLM agents?
PRISM is a failure-routed evolutionary optimizer that edits prompts and tool-boundary middleware around a fixed LLM without retraining. In the paper it reports mean held-out lifts of 14.2, 14.9, and 10.1 percentage points over the native Deep Agents baseline on BFCL multi-round, tau2-Retail, and tau2-Telecom, with positive lower-tail reliability (RelLift95 of 12.0, 10.1, and 7.8).
Does middleware always help LLM tool agents?
No. The paper's own results show unconstrained middleware editing (BetterHarness) underperforms its prompt-only variant on both tau2 tasks, with negative lower-tail reliability. Gains appear when middleware edits are constrained to three checkable tool-boundary patterns — silent argument correction, error blocking, prerequisite blocking — and routed from root-cause failure clusters.
Is the code and data for this paper available?
The authors state that PRISM, all baseline optimizer wrappers, benchmark runners, and the RelLift95 bootstrap estimator are publicly released at github.com/airbnb/agent-harness-optimizer, with a reproducibility guide and exact commands. The experiments themselves use proprietary API models (gpt-5.4-mini inner, claude-opus-4.7 outer) on public benchmarks (BFCL, tau2-bench), so full replication requires paid API access.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-9 appear in the extraction as captions only; their visual content (the PRISM loop diagram, the concrete prompt/middleware examples in Appendix D) is not part of the text extraction, so those figures could not be assessed directly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.