How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making
physics.soc-ph · cs.AI
Quick verdict
Claims agent success decays geometrically with step count, not context; rigorous design, open data, but tiny per-cell samples weaken agentic collapse claims.
Summary
This empirical study measures how LLM agent success degrades as task horizon grows. Across four synthetic, oracle-verified task families (three structured-output streaming tasks and one ReAct-style agentic chain-traversal task), nine models (six open, 1.2B-671B parameters; three proprietary), five horizons, and three context regimes, the authors analyze 10,664 trajectories. Success follows a geometric decay law (AIC-selected in 28/36 model-task cells) governed by a single per-step reliability parameter that rises with scale but never reaches 1. On the agentic task, every model collapses between H=2 and H=16; Table 2 shows, e.g., Qwen2.5-72B falling from 1.00 to 0.13. Pooling three context regimes, the authors argue step count, not context length, drives decay: bounding context steepens the logit slope (-0.69 vs -0.44, p=3e-6), contradicting lost-in-the-middle predictions. Within trajectories, per-step accuracy drops from 0.58 to 0.44 between first and last thirds. Projecting a pooled per-step reliability of 0.61 yields 0.42 predicted success at GAIA-like 8-step horizons and 0.24 at 100 steps. Limitations include synthetic tasks, small per-cell samples (Table 2 cells imply n~15), non-monotonic agentic cells, an incomplete longest-horizon agentic condition, and a fixed undisclosed temperature. Code, seeds, and raw trajectories are released.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Task success on multi-step agent workloads follows a geometric decay law governed by a single per-step reliability parameter, AIC-selected over threshold and linear forms in 28 of 36 model-task cells.
- On a genuinely agentic ReAct-style task, all nine models collapse from near-ceiling success at two steps to near-zero within sixteen steps (e.g., Qwen2.5-72B: 1.00 at H=4 to 0.13 at H=16).
- Degradation is driven by step count rather than context length: bounding the context steepens decay (logit slope -0.69 vs -0.44, p=3e-6) and single-turn padded delivery matches the natural regime (-0.40, p=0.51).
- Per-step reliability rises with model scale but never reaches 1 on non-trivial tasks, making long-horizon collapse a mathematical certainty; projecting pooled r=0.61 predicts 0.42 success at GAIA-like 8-step horizons and 0.24 at 100 steps.
- Hazard accelerates within a trajectory: mean per-step accuracy falls from 0.58 in the first third to 0.44 in the last third, with errors behaving as absorbing states.
Strengths
Exact oracle verification removes judge noise
Section 3.1: all four task families are verified by a ground-truth simulator, keeping LLM-judge noise out of the success signal; Section 3.2-3.4 reuses identical task instances across the three context regimes for paired comparisons, increasing power.
Genuinely agentic probe plus deployed systems
The ToolQA family runs an unmodified ReAct loop with hidden-chain traversal (Section 3.1), pre-empting the structured-output-probe objection; the roster mixes six open models (1.2B-671B) with three hosted proprietary systems, so the collapse is not a toy-model artifact.
Contrarian, decision-relevant disentangling result
Unusual openness
Candid limitations section
Weaknesses
Non-monotonic agentic results contradict the claimed monotone geometric decay
Compressed regime conflates context bounding with deleting agent reasoning state
Scale-reliability claim is statistically empty as reported
Benchmark-gap projection pools a task-mix-dependent mean
Regime manipulation tested on streaming families only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: How Fast Do Agents Rot?
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLM agents really degrade on long multi-step tasks?
Yes, and this paper quantifies it: across 10,664 trajectories, nine models, and four oracle-verified task families, success follows a geometric decay law, roughly success = r^H where r is per-step reliability. Because r stays below 1 even for frontier models, success collapses at long horizons; on the agentic task every model fell from near-perfect at two steps to near-zero at sixteen steps.
Is agent failure caused by long context or by the number of steps?
This study's disentangling experiment points to step count. Holding the number of operations fixed, bounding the context window made decay steeper (logit slope -0.69 vs -0.44, p=3e-6), and delivering all operations in a single turn decayed at the same rate as multi-turn interaction. This contradicts the lost-in-the-middle explanation and suggests naive context truncation may backfire.
What per-step reliability do current models achieve and what does it imply?
Per-step reliability rises with model scale but saturates below 1; the study's pooled mean is 0.61. Pushed through the geometric law, that predicts only about 0.42 success at GAIA-like 8-step horizons and 0.24 at 100-step production workflows, which the authors offer as the mechanism behind the benchmark-to-production reliability gap.
Is the paper's code and data available?
Yes. The authors release code, task generators, prompts, random seeds, all 10,664 raw trajectories, and analysis scripts at github.com/shubmittal/agent-horizon-degradation, including an evidence-trail document mapping every reported number to the released data.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 appear only as captions in the extracted text; the plotted curve shapes cannot be verified from text alone.
- One abstract sentence is garbled in the extraction ('...within sixteen steps of (n=10,664 analyzed trajectories'), likely a PDF extraction artifact, so the exact wording there is uncertain.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.