Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents
cs.LG · cs.AI · cs.SE
Quick verdict
Claims exactly-once for LLM agents lives in model or contract depending on fault; rigorous preregistered benchmark, but code unreleased and services simulated.
Summary
The paper asks whether exactly-once behavior in tool-using LLM agents should be enforced by the model, the agent harness, or the tool contract. It introduces LIMBO, a deterministic sandbox of six simulated services with realistic contracts (optional idempotency keys, eventually consistent or missing read paths) and twelve fault modes injected at the service boundary, including late commits, redelivery and partial batches; every episode is graded against a ground-truth effect ledger rather than an LLM judge. Across 25,930 episodes covering nine models, three production harnesses, two contract variants and fifteen recovery conditions, the answer depends on the fault: when a read-back can resolve an ambiguity, frontier models duplicate only 0.5% of lost-acknowledgement writes and the model explains 53% of explained variance; for late commits and redelivery the same models duplicate in 56% and 74% of episodes and the contract explains 81%. The authors prove no verification-only policy is exactly-once under unbounded late commits, and show that offering idempotency keys on every write cuts duplicates from 28% to 4%, while harness choice barely matters and transparent client-side retries actively harm. Limitations: simulated services, a single model gateway, only twelve task templates, code and data promised only upon publication, and several headline analyses added after preregistration as exploratory.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
5 strengths · 5 weaknesses · 0 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Where exactly-once should be enforced depends on fault observability: for faults a read-back resolves, the model explains 53% of explained variance; for late commits and redelivery, the contract explains 81%.
- No verification-only recovery policy can be exactly-once under late commits without a known bound on in-flight time (Proposition 1), while re-issuing with a stable idempotency key is exactly-once in all five outcome states (Proposition 2).
- Extending idempotency keys to every write lowers the duplicate rate from 28% to 4% because agents attach keys in 98% of episodes, and this mechanism transfers unchanged across three production harnesses.
- Production harnesses behave almost identically to a minimal scaffold, and transparent client-side retries cut exactly-once success from 72% to 50%.
- Agents reported task completion in 90% of the episodes in which they had produced a duplicate side effect.
Strengths
Formal framing that converts intuition into testable structure
Proposition 1 (no verification-only policy is exactly-once under late commits without an in-flight bound) and Proposition 2 (stable keys suffice in all five outcome states) are proved in half a page and then validated by scripted reference policies and 31 unit tests; the regime split (model vs contract) follows directly from them.
Measurement rigor: ledger grading plus empirically verified observation equivalence
Every episode is graded against a ground-truth effect ledger (no LLM judge); observation equivalence is verified empirically — first-action agreement across matched worlds is 87% vs 82% across independent same-world runs, TV distance 0.07 equals the exchangeable-label expectation, and 868 rerun episodes agree at kappa = 0.90.
Statistical hygiene rare in agent benchmarks
Actionable counterfactual design and honest negative results
Self-critical findings that cut against the authors' own guard
Weaknesses
No released code or data at review time
Simulation idealizes the very layer the paper indicts
Only twelve task templates drive every experiment
Headline numbers rest partly on exploratory analyses
Variance decomposition is metric-sensitive
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a sharpened version of a question several 2026 concurrent works circle: for a tool-using LLM agent whose write times out or errors, should exactly-once behavior be enforced in the model, the harness, or the tool contract? The contribution is threefold: (1) LIMBO, a deterministic six-service sandbox with twelve service-boundary fault modes including late commits, redelivery and partial batches, graded against a ground-truth effect ledger; (2) a factorial study over 25,930 episodes spanning nine models, three production harnesses (Copilot CLI 1.0.86, Hermes 0.20.6, Codex CLI 0.139), two contract variants and fifteen recovery conditions; (3) two propositions with short proofs — verification-only policies cannot be exactly-once under unbounded late commits (Prop 1), and stable idempotency keys suffice in all five outcome states (Prop 2). The central claim is a regime split: read-back-resolvable faults put the decision in the model (53% of explained variance); late commits and redelivery put it in the contract (81%).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the LIMBO benchmark?
LIMBO is a deterministic, resettable sandbox of six simulated services (social, billing, tickets, mail, database, deploy) with realistic API contracts such as optional idempotency keys, eventually consistent read paths and endpoints with no read-back. Twelve fault modes are injected at the service boundary — including late commits, redelivery and partial batches — and every agent episode is graded against a ground-truth ledger of committed effects rather than by an LLM judge. It is exposed via MCP so unmodified production agent harnesses can run against it.
What is the paper's main finding about LLM agents and exactly-once actions?
Exactly-once behavior lives in the model when an immediate read-back can resolve what happened: frontier models duplicate only 0.5% of lost-acknowledgement writes because they verify before retrying. It lives in the tool contract when no read-back can help — late commits and duplicate deliveries — where the same models duplicate in 56-74% of episodes. The paper proves no verification-only policy can be exactly-once under late commits without a known bound on in-flight time.
How much do idempotency keys help agent tool calls?
Offering an idempotency key on every write lowered the duplicate side-effect rate from 28% to 4% with no other change, because agents attach keys in 98% of episodes when the contract allows it. Waiting for a bounded in-flight delay is a poor substitute: with heavy-tailed delays, even one hour of waiting per ambiguous write reached only 84% exactly-once success versus 94% for keys plus a key-attaching guard.
Is the LIMBO benchmark code publicly available?
Not yet at review time. Footnote 1 states that code, harness adapters and every episode trace 'will be made available upon publication.' The paper does describe the design in detail, including seeded paired worlds, 31 validation unit tests and a list of deviations from its preregistration.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Abstract is truncated mid-sentence in extraction ('with heavy'), so some headline numbers were cross-checked against body text only
- Figures 2, 3, 5 and 6 extracted as raw number grids; per-cell values were read from Tables 3-10 instead
- Appendices A-C partially included; preregistration document itself not in the text
- References verified only as cited; no external access was used
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.