Chronicle: Cut-Point Replay for Regression Testing of LLM Agents
cs.CL · cs.AI
Quick verdict
Chronicle turns recorded agent failures into CI regression tests via selective replay; results are mechanism-valid but on 6 self-built simulated incidents.
Summary
The paper addresses the difficulty of reproducing LLM agent failures: non-deterministic inference, stateful tools, and multi-step trajectories make re-runs unreliable. Chronicle records agent runs at non-deterministic boundaries (model calls, tool calls, routing decisions) as immutable envelopes, addressed by boundary name and occurrence. Full replay serves every boundary from the record (zero model calls); the central operation, cut-point replay, serves a chosen subset from the record while executing the complementary subset live with new code, turning a recorded incident into a deterministic CI regression test with per-name call-count checks on stubbed boundaries. On a benchmark of 6 recorded incidents (refund, invoice, trade, email, payout, deletion) with simulated model boundaries, recording adds 23 μs per crossing, full replay is bit-stable across 20 repetitions, cut-point tests fail on unguarded code and pass on guarded fixes and benign edits for all 6, tolerate 30 unrelated rewordings, and kill 51 of 192 tool mutants while a stub-everything baseline kills none. Key limitations, largely acknowledged by the authors: the benchmark is small and self-constructed with guards and assertions co-written, determinism is partly by construction because model boundaries are simulated, and streaming, concurrency, and LLM-judge reliability are unaddressed.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Cut-point replay turns a recorded agent incident into a deterministic CI regression test by serving a chosen subset of boundaries from the record and executing the complementary subset live with new code.
- Recording adds only 23 μs per boundary crossing, and full replay issues zero model calls and is bit-stable across 20 repetitions on the released benchmark.
- Cut-point tests fail on unguarded code and pass on guarded fixes and benign edits for all 6 incidents, tolerating 30 unrelated rewordings.
- Cut-point tests kill 51 of 192 mutants of the guarded tools (62% of the 82 that change behavior on the recorded input), while a stub-every-boundary baseline using the same assertion kills none.
Strengths
Clean negative control against the stub-everything baseline
Section 4's argument and Table 2 show a full-stub baseline using the same assertion fails the unguarded code, the guarded fix, and the benign edit alike (0/6 pass on fix), because its verdict cannot depend on tool code — making Chronicle's 6/6 a soundness demonstration rather than a mere score.
Unusually candid limitations
The authors state determinism is 'partly by construction', call the benchmark 'small and self-constructed... without the loops, retries, or multi-agent routing that motivate this work', flag the count-check's reordering gap (Section 3.2), and leave the LLM-as-judge unevaluated (Section 3.3).
Honest mutation-study decomposition
Overhead measured with real-model statistics
Weaknesses
Circular determinism evidence
Six author-constructed incidents
No empirical positioning against closest prior art
Known soundness gap left unshipped
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Chronicle (arXiv 2609.20625) is a systems/method paper at the intersection of software testing and LLM agents. It claims existing agent tooling records runs only to trace or score them, never to test a code change against a recorded run. Chronicle's abstraction is the boundary (model call, tool call, routing decision); each execution ('crossing') is stored as an immutable envelope with input, output, and drift metadata (model version, sampling parameters), addressed by name and occurrence (Section 3.1, Figure 5). Full replay serves every boundary from the record; cut-point replay (Section 3.2) serves a chosen subset and runs the complementary subset live with new code. Stubbed boundaries carry a per-name call-count check that raises on divergence. Structural assertions test what the agent did; an advisory LLM-as-judge covers non-structural properties but is explicitly not evaluated. Four headline claims: near-zero recording overhead (23 μs/crossing), deterministic zero-model-call replay, correct fail/pass behavior on 6 incidents, and mutation superiority over a stub-everything baseline.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is cut-point replay in Chronicle?
It is a replay mode in which a recorded LLM-agent run is reproduced by serving most non-deterministic boundaries (model calls, tool calls, routing decisions) from an immutable recording, while a chosen subset of boundaries is executed live with new code. This lets a past incident become a deterministic regression test that verifies a code change in continuous integration.
Does Chronicle need LLM API calls when running the tests?
No. Full replay serves every boundary from the recorded envelopes and issues zero model calls. A cut-point test with only tool boundaries live also avoids model calls. The released 6-incident suite completes a pass in about 3.5-3.6 ms at $0 provider cost, so it can run on every commit.
How well does Chronicle detect bugs in LLM agents?
On its own benchmark of 6 recorded incidents with simulated model boundaries, cut-point tests flagged every unguarded tool, passed every guarded fix and benign edit, tolerated 30 unrelated rewordings, and killed 51 of 192 tool mutants; a stub-everything baseline killed none. These are mechanism-validation results on curated incidents, not general fault-detection rates.
Is the Chronicle code and benchmark available?
Yes. The paper states that Chronicle and the 6-incident benchmark are publicly available at github.com/theagentplane/chronicle, including recorded traces, unguarded and guarded tools, and cut-point tests per incident.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are present only as captions and text fragments in the extraction; diagram content could not be independently verified.
- The public repository (github.com/theagentplane/chronicle) was not inspected; code-availability and implementation claims rest on the paper text.
- Author email domains (vitalum.ac.in, bits-pilani alumni) differ from the stated Microsoft affiliation; a metadata inconsistency unresolved by the text.
- Table 2 formatting is partially garbled in extraction; all numbers were cross-checked against the body text of Sections 4-5.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.