Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course
Evelyn Duesterwald, Benjamin Elder, Lilian Ngweta, Shashanka Ubaru, Malgorzata Zimon
cs.AI
Quick verdict
Proposes consistency-gap metric and episodic-memory guidelines lifting AppWorld GPT-4.1 Pass^5 by +16pp; promising but single-benchmark, no significance tests.
Summary
The paper formalizes the 'consistency gap' of LLM agents: identical tasks yield different outcomes across runs. On AppWorld's 168-task test-normal split, a ReAct agent backed by GPT-4.1 averages 77.4% pass rate (Mean@5) yet passes all five runs on only 53.0% of tasks (Pass^5) — a 24.4-point shortfall. The proposed offline pipeline pairs a black-box Consistency Analyzer, which resamples each inference point N=30 times at temperature 0.5 and scores step stability via SBERT cosine or Jaccard similarity, with a Guideline Generator that converts flip-prone steps into strategy/recovery/optimization guidelines stored in episodic memory and retrieved for later tasks. Pass^5 rises +16.0 points same-task and +13.0 points on similar sibling tasks for GPT-4.1, with aggregate Mean@5 never degrading; GPT-OSS-120B gains +6.0/+8.7 points from a lower baseline. Trajectory consistency predicts pass/fail with AUROC 0.699. Main limitations: a single benchmark and agent architecture, no significance tests or confidence intervals, guidelines admitted without validation, roughly 30× resampling token cost, and an uncontrolled platform non-determinism confound.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On AppWorld test normal, ReAct/GPT-4.1 shows a 24.4 pp consistency gap: Mean@5 of 77.4% versus Pass^5 of 53.0% (GPT-OSS-120B: 23.8 pp).
- A black-box Consistency Analyzer can identify flip-prone inference steps by resampling recorded prompts N=30 times and measuring response variability, without model internals or agent code.
- Injecting consistency-targeted guidelines retrieved from episodic memory raises Pass^5 by +16.0 pp same-task and +13.0 pp on similar sibling tasks for GPT-4.1, without degrading aggregate Mean@5.
- Aggregate trajectory consistency predicts trajectory pass/fail with AUROC 0.699, providing an outcome proxy when no ground-truth rating exists.
Strengths
Clear, reusable metric formalization
Section 2.1 defines Pass@k, Mean@k, Pass^k, Gap@k and normalized Consistency@k=Pass^k/Mean@k with the Bernoulli inequality E[Pass^k]=p^k<=p, cleanly separating capability from reproducibility; this exposes GPT-OSS-120B's aggregate consistency of 0.30 vs GPT-4.1's 0.69.
Dual-model, per-difficulty empirical characterization
Figure 2 quantifies the gap for GPT-4.1 (24.4 pp) and GPT-OSS-120B (23.8 pp) across easy/medium/hard, revealing an informative inversion (GPT-OSS-120B peaks on easy at 38.6 pp) rather than a single-model anecdote.
Black-box, architecture-agnostic analyzer with typed scoring
Generalization tested, not just memorization
Unusually honest limitations section
Weaknesses
No statistical inference anywhere
Single benchmark, single agent scaffold
Missing attribution controls
Off-split, thin validation of the outcome proxy
Advertised study absent from the text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets run-to-run unreliability of LLM agents and makes four claims: (1) a formalization of the 'consistency gap' (Gap@k = Mean@k − Pass^k, §2.1) plus a normalized consistency metric; (2) a black-box Consistency Analyzer estimating step-level flip risk from resampled response variability (§3.1); (3) consistency-targeted guideline generation feeding episodic memory (§3.2); (4) empirical gains of +16.0 pp Pass^5 same-task and +13.0 pp similar-task on AppWorld test normal with ReAct/GPT-4.1 (+6.0/+8.7 pp for GPT-OSS-120B), plus AUROC 0.699 for trajectory-level outcome prediction (§4.4). Code is released in the ALTK-EVOLVE repository.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the consistency gap in LLM agents?
It is the difference between an agent's average pass rate on a task (Mean@k) and the fraction of tasks it passes on all k runs (Pass^k). For a ReAct agent with GPT-4.1 on AppWorld, the average pass rate is 77.4% but only 53.0% of tasks pass all five runs — a 24.4-point gap, meaning apparent accuracy partly evaporates under repeated use.
How much does the proposed framework improve consistency?
On AppWorld with GPT-4.1, the fraction of tasks passing all five runs rises by 16.0 points on the same task (53.0%→69.0%) and 13.0 points on similar sibling tasks, with average pass rate not degrading. GPT-OSS-120B improves by 6.0 and 8.7 points respectively from a lower baseline (10.1% Pass^5).
Does the method require access to model internals like logits?
No. The Consistency Analyzer re-issues the agent's recorded prompts to the same hosted endpoint 30 times at temperature 0.5 and measures response variability (embedding or Jaccard similarity), so it works with black-box commercial APIs.
Is code available for this paper?
Yes. The Detect-and-Generate pipeline is released as a feature of the open-source ALTK-EVOLVE repository (github.com/AgentToolkit/altk-evolve); AppWorld is a public benchmark, though experiments used paid cloud endpoints (Azure GPT-4.1, AWS GPT-OSS-120B).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are extracted as bare axis labels and numeric bar values; visual-only details (e.g., presence of error bars) cannot be verified from the extraction.
- The provided text ends at the alphabetized reference list (Zhou et al. 2025); any appendix material placed after the references would fall outside the extraction — notably, the sample-budget sensitivity study promised in Contribution (4) appears nowhere in Sections 1-7 or Appendices A-B as provided.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.