Making Agents More Consistent: Skills Should Form Habits for Repeat Tasks
cs.AI · cs.CR
Quick verdict
Proposes coverage-gated skill habituation for consistent, conditionally cheaper repeated agent work; candid failure analysis, but self-authored evals, small n.
Summary
The paper argues that LLM agents are inconsistent and wasteful on repeated tasks: across 42 tasks run three times, 38–74% of answers disagreed depending on model, and 95.3–97.2% of generated tokens re-derive plans. It proposes skill habit formation: the agent mines its execution history for deterministic script variants that compete with, rather than replace, the reasoning incumbent. Candidates pass four ascending-cost gates (recoverability, governance, trace conformance against a retained anchor, human-authorized A/B), with the trace gate calibrated from the anchor's own run-to-run variability. On a self-authored text-to-SQL workload, reasoning arms reproduced byte-identical output on 11–26 of 42 questions while a habit-formed variant reproduced on all 456 repeated dispatches, was non-inferior to all four arms (p<0.0001), and cut tokens 14–56% per request (break-even after 7–53 reuses). Failure analysis is candid: the guard mis-admitted 2.6% of natural paraphrases and 26% of boundary inputs; 11 of 13 failures were structurally invisible to the trace gate. A HotpotQA-based routing replication shows mined corrections generalize to unseen intents, but also reveals a severe unsourced-answer error class emerging under habituation and a non-inferiority deadlock for a memory-based guard. Limitations: self-authored benchmarks and labels, small samples behind the key safety rates, and no code, data, or platform release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reasoning arms disagree with themselves on 38–74% of 42 thrice-repeated text-to-SQL tasks and spend 95.3–97.2% of generated tokens re-deriving plans they already hold (§I, §X).
- A coverage-gated deterministic habit reproduced all 456 repeated dispatches exactly, was non-inferior to all four arms (p<0.0001), and cut per-request tokens 14–56%, breaking even after 7–53 reuses (§X).
- Habit formation's failure mode is silent and exactly reproducible: 2.6% false admission on natural paraphrases and 26% near the boundary, with 11 of 13 failures structurally invisible to the trace-conformance gate at any threshold (§X, §XII).
- Separating route selection from parameter extraction raised end-to-end accuracy from 0.888 to 0.952 at 43% of the cost, and admission risk accrues per catalog: a template passing every gate becomes unsafe when an adjacent sibling pushes the bundle past 20 (§X).
Strengths
Unusually candid negative reporting
§X reports the non-inferiority deadlock (embedding-memory guard perfectly flat, memory empty at end), the emergence of a severe unsourced-answer class (0.000→0.071, below the 0.118 floor, direction only), and §XI/XII enumerate unmeasured quantities; the pre-specified t=100→t=200 contrast (−0.003 [−0.028, +0.022]) is reported instead of the biased peak-to-final (−0.015).
Statistical hygiene above venue norm
One-sided permutation tests calibrated from the reference's own Dself with BH FDR control per batch (§VI); non-inferiority with pre-declared 5-point margin (§X); Wilson intervals on small proportions (26.0% [15.9%, 39.6%]); intervals computed over runs, not the 288 re-scored questions; a measured 0.118 same-seed replication floor published and used to veto sub-floor claims.
Rigorous self-indictment of its central gate
Actionable mechanistic decompositions
Weaknesses
Self-authored evaluation workloads throughout
Headline consistency metric is near-definitional in both directions
Small effective samples behind the key safety rates
No code, data, or platform disclosure
Architecture claims exceed what is tested
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets two defects of production LLM agents on repeated work: output inconsistency (38–74% of 42 thrice-run tasks disagree, depending on model) and plan re-derivation cost (95.3–97.2% of generated tokens across the three metering arms; 62.7–75.0% of billed tokens). Its response is an architecture — skill habit formation — in which skills are lineages of variants, deterministic script variants claim declared input regions via coverage predicates, and admission runs through four ascending-cost gates: C0 recoverability probe, C1 static guardrail subset check, C2 trace conformance against a retained anchor, C3 human-authorized A/B. Claims are staged honestly: exact-replay consistency is primary; token savings secondary and explicitly conditional; the accuracy price is measured, not asserted. The authors themselves bound the scope: the full C0–C3 lineage machinery is never exercised, and the experiments cover only coverage-gated dispatch plus the §IX cost inequality.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is skill habit formation for AI agents?
It is this paper's proposal for making LLM agents consistent on repeated tasks: the agent mines its own execution history for deterministic script variants that serve the common, recognizable cases, while a coverage guard sends everything else to the full reasoning model. Candidate scripts must pass four admission gates (recoverability, governance, trace conformance, human-authorized A/B) and never replace the reasoning path, which stays executable so reverting is trivial.
How inconsistent are LLM agents on repeated tasks?
In the paper's text-to-SQL experiment, four frontier reasoning models (Claude Sonnet 5, Kimi K2 Thinking, GLM-4.6, Qwen3-235B-Thinking) produced byte-identical answers across three repeats on only 11–26 of 42 questions, meaning 38–74% of repeated tasks gave non-identical answers. The criterion is exact text equality, so differently worded but functionally equivalent SQL also counts as disagreement — one caveat a careful reader should apply.
What are the main risks of the habit-formation approach?
The coverage guard sometimes claims work it should decline: 2.6% of natural paraphrases and 26% of boundary-region inputs were answered wrongly with no reasoning step running to catch the mistake, and 11 of 13 such failures were invisible to the paper's trace-conformance gate at any threshold because the gate records argument schemas, not argument values. Since the script is deterministic, a wrong answer then repeats exactly every time.
Does habit formation actually save money?
Conditionally. A served request costs 14–56% fewer tokens than the reasoning arm it replaces, but one-time habit formation (7,691 tokens per template) must amortize, so break-even arrives after 7–53 reuses depending on the arm — 53 against the cheapest-planning arm (Sonnet). The guard's own prompt grows linearly with the template catalog (2,360 tokens at 20 templates), and the paper concedes that at design-partner scale auditability, not savings, may be the honest motivation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are available only via their captions; plotted per-checkpoint values cannot be independently verified from the extracted text.
- The abstract in the source extraction is truncated mid-word, but the complete abstract is recoverable from the paper body; no content assumed missing.
- Review is based solely on arXiv v1 (21 Sep 2026); no supplementary material or external code was consulted.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.