Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models
Utkarsh Soni, Syed Shariyar Murtaza, Yifan Nie, Sachin Chandrasekhar, Eugene Wen
cs.CL · cs.AI
Quick verdict
TAM benchmark shows GPT-5 near-fails long manual procedures (1%–15.5% EM); valuable, though single-model with unquantified label-noise ceiling.
Summary
The paper introduces Tasks over Application Manuals (TAM), a benchmark testing whether LLMs can execute long, rule-based procedures in authoritative manuals rather than short multi-hop QA. Two domains instantiate it: ICD-10-CM clinical coding (predicting the exact diagnosis-code set for 1,000 discharge summaries sampled from 38,332 MIMIC-IV encounters, year-aligned to the 2019 manual) and U.S. federal sentencing (computing the exact total offense level for 200 human-verified CourtListener cases from 2021–2025). The authors evaluate four GPT-5 baselines — single-pass RAG, agentic RAG, ReAct-style tool use, and a LangChain agent harness — all with access to structured, domain-specific manual indexes. Exact match stays at 1% or below for ICD coding (best recall 37%, primary-diagnosis accuracy 50.1%) and reaches at most 15.5% for sentencing (best MAE 2.34). Trajectory analysis of 50 ReAct runs attributes most failures to global inconsistency. Main limitations: only one model is tested; no statistical uncertainty (single runs, no confidence intervals or significance tests); the ceiling imposed by coder label noise on multi-code exact match is acknowledged but not quantified; and 35 of 200 sentencing cases ended early under the agent harness due to provider content filtering. Data and code are public.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing reasoning benchmarks are short-horizon and do not test faithful end-to-end execution of long, rule-based professional manuals.
- TAM instantiates manual-governed procedural execution in two real domains (ICD-10-CM coding; federal sentencing offense levels) with exact outputs and human-validated labels.
- On GPT-5, all four evaluated baseline families reach at most 1% exact match on ICD coding (harness 0.7%) and at most 15.5% on sentencing (ReAct), with best MAE 2.34 (harness).
- The dominant failure mode is global inconsistency (38/50 analyzed ReAct trajectories), i.e., locally plausible but globally invalid decision paths.
- Therefore performance on short-horizon benchmarks overestimates LLM readiness for real-world long, rule-based procedures.
Strengths
Genuine, well-argued gap
§3.5 cleanly separates manual-governed procedural execution from short-hop QA (HotpotQA/MuSiQue), interactive environments (WebShop/ALFWorld), and ICD-as-classification lines (PLM-ICD etc.); the constrained-trajectory formalization in §2.1 motivates why exact match is the right strict criterion (§6.2).
Careful ICD construction
§4.1 pins evaluation to one manual version: 254,377 MIMIC-IV admissions → 65,785 (2017–2019) → 53,751 (codes revised across 2017–2018/2018–2019 addenda excluded) → 38,332 → 1,000 sampled; the slice tracks the cohort (mean 12.85 vs 12.65 codes/case, median 12 vs 11, p90 23), and contamination is discussed with a direction-of-bias argument.
Human-verified legal labels
Honest operational reporting
Reproducibility assets
Weaknesses
Single model undermines generality
No statistical uncertainty
ICD label-noise ceiling unquantified
Content-filter confound in harness results
Thin, non-validated failure analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TAM (Tasks over Application Manuals) is a benchmark for *manual-governed procedural execution*: given a case record and an authoritative manual with tens of thousands of rules, produce one exact output by correctly executing a long, interdependent procedure. §2.1 formalizes this as constrained search over decision trajectories where local plausibility does not imply global validity. Two domains instantiate it: ICD-10-CM diagnosis coding (exact code set from discharge summaries) and U.S. federal sentencing (exact total offense level). The central empirical claim (§7.1, Table 2) is that four GPT-5 baseline families — single-pass RAG, agentic RAG, ReAct-style tool use, and a LangChain Deep Agents harness — all remain far from reliable execution: exact match ≤1% on ICD (0.7% for the harness) and ≤15.5% on sentencing. The interpretive claim (§1, §10) is that short-horizon multi-hop benchmarks overestimate LLM readiness for such tasks. The positioning argument in §3.5 — separating TAM from short-hop QA, interactive agent environments, ICD-as-extreme-multi-label-classification, and constraint benchmarks — is the paper's best argumentation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the TAM benchmark?
TAM (Tasks over Application Manuals) is a benchmark testing whether large language models can execute long, rule-based procedures from authoritative manuals: producing the exact ICD-10-CM diagnosis-code set for hospital discharge summaries, and the exact total offense level under U.S. federal sentencing guidelines. It is scored by strict exact match with human-validated labels.
How well do GPT-5-based systems perform on TAM?
Poorly. All four evaluated baselines — single-pass RAG, agentic RAG, ReAct-style tool use, and a LangChain agent harness — achieve at most 1% exact match on ICD-10-CM coding (0.7% for the harness) and at most 15.5% on sentencing offense-level calculation, despite having structured access to the relevant manuals.
Is the TAM dataset and code publicly available?
Yes. The paper states data are at huggingface.co/datasets/manulife/tam-benchmarks and code at github.com/manulife-ai/tasks-over-application-manual. Note the ICD cases were derived from MIMIC-IV, which itself requires credentialed access if you want to rebuild the pipeline rather than just use the released cases.
Why is ICD exact match so low — could it be label noise rather than model failure?
Possibly partly. Cases average 12.85 diagnosis codes, so matching the entire set exactly is brittle even between human coders. The authors acknowledge this and argue the gap is not noise alone, citing partial metrics (best recall 37%, primary-diagnosis accuracy 50.1%), but they never measure human inter-coder exact-match agreement, so the true ceiling is unquantified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are present only via their captions in the extracted text; the plotted distributions themselves could not be inspected.
- Appendix B prompts are clipped excerpts by the authors' own design (omissions marked '(...)'), so full prompts exist only in the public repository — this is paper design, not extraction loss.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.