How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agents
Yukun Zhang, Kemu Xu, Yishen Chen
cs.AI
Quick verdict
Claims prewritten plans lift agent success 7.17 pp over word-matched controls and cheap verifiers curb false passes; headline effect is statistically fragile.
Summary
Using Retail and Airline environments from tau2-bench, this empirical study attributes value to two agent-harness components: planning guidance and terminal release verification. The primary comparison pairs prewritten task-specific plans (Fixed) with word-count-matched shuffled policy text (Sham) across 265 matched cells (four models, 23 tasks): Fixed improves oracle-verified success by 7.17 percentage points (90% task-clustered interval 1.15-13.36), concentrated on higher-complexity tasks. A read-only terminal verifier rejects 61% of oracle-invalid Retail episodes while withholding 17% of correct ones, cutting the false-pass rate from 57.21% to 20.96% for under one cent per episode; under a liability-weighted value framework, a standalone verifier captures nearly all the false-pass benefit of the full stack at a fraction of its cost. Limitations: leave-one-model-out intervals cross zero for two of four deletions, no model-specific effect survives Holm adjustment, the Airline pilot shows no adjusted significance, plan authorship provenance is unrecorded, coverage gaps are substantial, and no public repository is yet available.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Prewritten task-specific plans improve oracle-verified success by 7.17 percentage points over word-count-matched shuffled policy text across 265 matched cells, with gains concentrated in higher-complexity tasks.
- A read-only terminal verifier rejects 61% of oracle-invalid Retail episodes while withholding 17% of correct ones, reducing the false-pass rate from 57.21% to 20.96% at under one cent per episode.
- The configuration ranking reverses with the loss assigned to erroneous acceptance: planning dominates at low liability, verification at high liability.
- A standalone verifier captures nearly all the avoided false-pass benefit of the full planning-plus-verification stack at roughly a twelfth of the incremental cost in the high-risk stratum.
- The Sham-Minimal placebo (+1.89 pp, interval spanning zero) shows the shuffled control does not harm performance, so the Fixed-Sham contrast reflects guidance content.
Strengths
Genuinely controlled planning contrast
The Fixed-Sham design (Sec. 4.3, Appendix D) matches word count and wrapper, and the Sham-Minimal placebo (+1.89 pp, 90% CI [-3.45, 7.09], Eq. 6) shows the shuffled control does not itself depress performance — a rigor most planning-prompt studies lack.
Outcome decomposition separating oracle correctness from acceptance
Eq. 1 and Figure 1 distinguish Z (oracle), S (asserted completion), R (verifier rejection), and A = S(1-R); Table 5's cross-tabulation makes the 61%/17% rejection trade-off and the 57.21%→20.96% false-pass reduction (Eq. 8) directly auditable.
Liability-conditional value framework
Unusually transparent reporting
Honest negative results on capability measurement
Weaknesses
Headline effect is fragile to model composition
Plan provenance admits possible leakage
The cost-fraction headline is stratum-specific
Very narrow task base and contamination risk
Correct-rejection asymmetry underplayed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks how two agent-harness components — planning information and terminal release verification — affect verified success, erroneous acceptance (false pass), and cost in stateful tool-use agents on τ²-bench (Retail and Airline). The headline claims are: (1) prewritten task-specific plans (Fixed) beat word-count-matched shuffled policy text (Sham) by 7.17 pp in oracle-verified success across 265 matched cells (90% task-clustered interval 1.15–13.36, Sec. 5.1); (2) a read-only terminal verifier rejects 61% of oracle-invalid Retail episodes while withholding 17% of correct ones (Table 5), cutting false passes from 57.21% to 20.96% for under one cent per episode; (3) under a liability-weighted value function (Eq. 2), a standalone verifier captures nearly all the full stack's avoided false-pass benefit (48.4 vs 49.8 pp) at about a twelfth of the incremental cost (Sec. 7.3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do prewritten plans actually help LLM agents, or just add more context?
According to this paper, yes — with a caveat. Across 265 matched test cells on the τ²-bench Retail environment, prewritten task-specific plans improved verified success by 7.17 percentage points over a control of shuffled policy text matched in word count. Because the control received the same amount of text, the gain is attributed to plan content rather than extra context. However, the improvement was concentrated in two of four models and on higher-complexity tasks.
How accurate is an LLM verifier that checks whether an agent really completed a task?
In this paper's Retail experiments, a read-only verifier (the executor's own model reviewing the last eight dialogue messages) rejected 61% of episodes that were truly incomplete while wrongly withholding 17% of episodes that were actually correct — about 84% of its rejections targeted genuinely invalid completions. It cut the false-pass rate from 57.21% to 20.96% for less than one cent of extra cost per episode.
Should I add planning, verification, or both to my LLM agent pipeline?
The paper's answer depends on liability: if a false 'task complete' is cheap, planning guidance adds more value through higher success; if false completions are costly (e.g., refunds or cancellations), the terminal verifier dominates. Notably, a standalone verifier captured nearly all of the full planning-plus-verification stack's false-pass benefit at a fraction of its cost — but that comparison was in high-risk tasks, where the full stack actually showed slightly negative success gains.
What are the main limitations of this agent harness study?
The headline planning effect is statistically fragile: leave-one-model-out intervals include zero after deleting either of the two strongest models, and no model-specific effect survives Holm multiple-comparison correction. The task base is small (23 Retail, 6 Airline tasks), public benchmark tasks may have appeared in model training data, plan-authoring information access was unrecorded, many design cells were missing, and no public code repository was available at publication time.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The embedded text omits several transitions (start of Section 4, parts of Sections 6.3-7.1, and the tail of Appendix L), so a few derivations were reconstructed from surrounding tables.
- Figures 1-5 are available only as text fragments; exact visual patterns (e.g., Figure 3c curve crossings) could not be verified.
- External validity judgments rest on the paper's own accounts of coverage and missingness, which could not be cross-checked against the frozen archive.
- The arXiv ID (2609.20474) and several cited works are dated 2026; claims about the surrounding literature were taken from the paper's related-work section without independent verification.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.