arXiv paper2026-09-08· machine learning (LLM agents, uncertainty calibration)· method paper

Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations

Q: What does the paper 'Do Agents Know When They Succeed?' show?

It shows that reading a coding LLM agent's internal activations (residual-stream states) predicts whether its multi-turn task will succeed better than looking at output token probabilities. The proposed ARP probe achieved the best AUROC and Brier scores in all nine tested model–benchmark combinations (InterCode Bash/SQL/Python × three open models), for example 0.842 AUROC for Qwen-14B on SQL.

Q: Is the method free at inference time?

The authors argue yes: the probe applies standardization, a 64-dimension PCA projection, and a logistic function to an activation that already exists in GPU memory during generation, costing under 3.3×10^5 operations (<0.05 ms). However, all reported experiments used offline teacher-forced replay of recorded trajectories, so live-deployment equivalence was not directly measured.

Q: Which models and benchmarks were used?

Qwen2.5-Coder-7B-Instruct, Qwen2.5-Coder-14B-Instruct-AWQ, and DeepSeek-Coder-6.7B-Instruct were evaluated on InterCode's Bash (200 tasks), SQL (1,014 Spider-derived tasks), and Python/MBPP (971 tasks), with greedy decoding, a 10-turn horizon, and 5-fold nested cross-validation.

Q: Is code available for this paper?

The paper text contains no code or data release statement. The protocol (frozen folds, seeds, exclusion indices, hyperparameter grids) is described in enough detail to reimplement, but no repository link is given.

Priyanka Mary Mammen, Emil Joswin, Srujananjali Medicherla

View original paper ↗arXiv:2609.09448

cs.AI

Quick verdict

Claims internal states (LTD/ARP) beat output-based agent confidence baselines across 9 settings; clean protocol but no significance tests, gains may be noise.

Summary

The paper asks whether an LLM agent's internal activations predict multi-turn task success better than output-based confidence signals. It proposes two methods: Latent Trajectory Dynamics (LTD), which summarizes residual-stream motion across trajectory transitions in 28 features, and the Action Representation Probe (ARP), a logistic probe over mean-pooled action-endpoint states with fold-safe standardization, 64-component PCA, and monotone Platt calibration. On InterCode Bash (200 tasks), SQL (1,014), and Python (971) with Qwen2.5-Coder-7B, Qwen2.5-Coder-14B-AWQ, and DeepSeek-Coder-6.7B under greedy decoding, both are compared against calibrated logprob and HTC (48 external trajectory features) using frozen 5-fold nested cross-validation. An internal method attains the best AUROC and Brier score in all nine model–environment settings — ARP reaches 0.842 AUROC on Qwen-14B SQL versus 0.743 for HTC — though margins are sometimes small, LTD trails HTC on AUROC in four of nine cells, and ECE favors the baselines in six of nine cells. Key limitations: no significance tests or confidence intervals; results rely on offline teacher-forced replay while the zero-overhead online claim is argued analytically but untested; evaluation covers only three small code-specialized models on coding benchmarks; and no code release is stated.

Review complete

Assessment at a glance

machine learning (LLM agents, uncertainty calibration)

risk flags identified

● high 1

● medium 3

● low 1

Reproducibility readinessmedium

Code & data availablenot stated

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

5 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

Internal residual-stream representations of a multi-turn coding agent carry success-predictive signal beyond output token probabilities.
LTD and ARP achieve the best AUROC and Brier score in all nine model–environment settings of Table 1, beating calibrated logprob and HTC.
Calibrated aggregate logprob alone is a weak trajectory-success discriminator, with AUROC as low as 0.454 (DeepSeek SQL) and 0.482 (DeepSeek Bash).
The probe adds negligible deployment cost (<3.3×10^5 FLOPs, <0.05 ms per action) and requires no prompt changes or multi-sample rollouts.

Strengths

Leak-free evaluation protocol
Frozen stratified 5-fold outer CV with one out-of-fold prediction per trajectory; inner 4-fold CV selects C from a 9-point grid by Brier; standardization, PCA, and Platt calibration fitted strictly on training folds (§2.2.2, Appendices F, H).
Transparent data hygiene
Appendix D enumerates 19 corrupt InterCode-SQL instances by task index, documents Python harness-exception exclusions, and removes replay-incompatible episodes from the shared population for all methods.
Matched, honestly-implemented baselines
Anti-shortcut design details
Consistent headline result

Weaknesses

No statistical uncertainty anywhere
Superiority claim is metric-selective
LTD's contribution is unstable
Offline/online gap unvalidated
Narrow scope and no code release

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

The paper targets trajectory-level confidence estimation for multi-turn coding agents, asking whether internal residual-stream states predict eventual task success better than output-derived signals. Two methods are introduced: **LTD** (§2.2.1, App E.1), 28 geometric features over trajectory transitions — adjacent-step cosine/relative displacement, inter-layer drift ratio (Eq. 5), cumulative path efficiency (Eq. 6), phase-transition velocities — and **ARP** (§2.2.2, App E.2), which mean-pools final-layer action-endpoint states (submit tokens excluded), standardizes and PCA-projects to 64 dimensions within training folds, then fits an L2 logistic probe with a monotone Platt calibrator. The claims: (1) internal states carry signal beyond token probabilities; (2) the monitor is zero-overhead at deployment.

Methodology Assessment

The setup is unusually careful for this literature. Three InterCode environments (Bash 200 / SQL 1,014 / Python 971 tasks, Table 2) with 19 corrupt SQL instances enumerated and excluded (App D); three open-weight code models (Qwen2.5-Coder-7B, Qwen2.5-Coder-14B-AWQ, DeepSeek-Coder-6.7B) on vLLM with greedy decoding. Frozen stratified 5-fold outer CV yields one out-of-fold prediction per trajectory; an inner 4-fold CV selects C from a 9-point grid by Brier; all preprocessing is training-fold-only (App F, H). Baselines get the matched predictor class: Cal-Logprob (1D l

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 5 strengths & 5 weaknesses, analyzed in detail
5 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What does the paper 'Do Agents Know When They Succeed?' show?

It shows that reading a coding LLM agent's internal activations (residual-stream states) predicts whether its multi-turn task will succeed better than looking at output token probabilities. The proposed ARP probe achieved the best AUROC and Brier scores in all nine tested model–benchmark combinations (InterCode Bash/SQL/Python × three open models), for example 0.842 AUROC for Qwen-14B on SQL.

Is the method free at inference time?

The authors argue yes: the probe applies standardization, a 64-dimension PCA projection, and a logistic function to an activation that already exists in GPU memory during generation, costing under 3.3×10^5 operations (<0.05 ms). However, all reported experiments used offline teacher-forced replay of recorded trajectories, so live-deployment equivalence was not directly measured.

Which models and benchmarks were used?

Qwen2.5-Coder-7B-Instruct, Qwen2.5-Coder-14B-Instruct-AWQ, and DeepSeek-Coder-6.7B-Instruct were evaluated on InterCode's Bash (200 tasks), SQL (1,014 Spider-derived tasks), and Python/MBPP (971 tasks), with greedy decoding, a 10-turn horizon, and 5-fold nested cross-validation.

Is code available for this paper?

The paper text contains no code or data release statement. The protocol (frozen folds, seeds, exclusion indices, hyperparameter grids) is described in enough detail to reimplement, but no repository link is given.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Full text including appendices A–I, Table 1, and Table 2 appears present in the extraction; observed artifacts are PDF line-breaks, not truncation.
Table 1 was extracted as inline text; numeric values were cross-checked against their row/column labels and appear correctly aligned.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper