Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents
Caiqi Zhang, Xiaochen Zhu, Chengzu Li, Yulong Chen, Dharshan Kumaran, Nigel Collier
cs.CL · cs.AI
Quick verdict
Proposes XConf, a training-free confidence estimator that recalibrates LLM outputs against the model's own graded past episodes, and backs the paradigm with unusually careful evaluation across nine benchmarks — though the headline 23-of-24 win count over self-consistency rests heavily on the cost argument, since most nominal wins fall inside overlapping bootstrap CIs.
Summary
The paper argues that confidence estimation should read a model's accumulated graded experience, not just the current inference. XConf stores past episodes (task, self-reflection, stated confidence, verified outcome, post-hoc lesson); Recall retrieves the 50 nearest episodes keyed on task embedding plus stated confidence under a correctness-supervised key and reads off their hit rate; Reflect shows the model this track record, asks it to name a recurring failure pattern, and elicits a revised confidence; the final estimate averages the two readings. It is black-box, training-free, and needs no logit access. Across nine benchmarks (reasoning, coding, multimodal QA, three agent domains) and four models, under five-fold out-of-sample replay with bootstrap CIs, XConf beats or matches ten-sample self-consistency on 23 of 24 comparisons with much lower ECE at roughly one generation's cost; on agent tasks it beats single-rollout baselines and even a trained verifier (e.g., AppWorld .837–.911 AUROC). Abstaining on the 10% least-confident episodes raises delivered success by up to 8.7 points. Limitations: it requires external graded outcomes, assumes in-distribution experience, loses on votable factual recall, and many headline wins are statistically thin.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The current inference process alone is an insufficient basis for confidence, and consulting the model's own graded past episodes corrects habitual miscalibration mechanically (Sections 1, 3).
- XConf matches or beats ten-sample self-consistency on 23 of 24 model-dataset comparisons with much lower ECE, at roughly one-tenth the generation cost, and extends to code and agent trajectories where voting cannot apply (Sections 5.1-5.2).
- Calibration improves monotonically with bank size (Section 5.4), banks transfer across datasets with <=.01 AUROC change when merged, and cross-model transfer incurs a .03-.06 penalty reflecting a self-other asymmetry mirroring human metacognition (Section 6).
- With question difficulty held constant via stratification, XConf still separates correct from incorrect attempts at .79 AUROC, above chance in 35 of 36 cells, so the signal is model-specific difficulty, not generic question difficulty (Section F).
- Using XConf for selective prediction, abstaining on the 10% least-confident episodes raises delivered success rate by up to 8.7 points on agent tasks (Section 5.5).
Strengths
Unusually rigorous anti-leakage and causal protocol
Five-fold OOS replay guarantees the bank is strictly prior (Section 4/K); a near-duplicate audit found 10.8% of MMLU-Pro items have >.95-cosine train-fold neighbours, yet excluding them moves AUROC by <=.001 (Section K); a prequential replay on LiveCodeBench by contest date (Section H) costs only .006–.020 AUROC versus the i.i.d. replay, ruling out leakage as the source of gains. Hyperparameter sweeps (k=5–200, sigma=.02–.32) show a wide plateau (Section I).
Agent-domain results are the real contribution
Where resampling is impractical, XConf reaches .837–.911 AUROC on AppWorld against verbalized .583–.834 and beats the trained HTC verifier in the hardest silent-failure column (Table 4). The Appendix B analysis of why ALFWorld/WebShop are unusable testbeds — WebShop's reward printed verbatim in 30–60% of transcripts — is itself a valuable methodological contribution corroborated by AUQ.
Mechanism is ablated, not just asserted
Honest reporting
Weaknesses
Headline win counts are statistically fragile
SC@10 is anomalously weak on several columns
External grading is a nontrivial operational dependency
Cost claims lack a token/latency table
The 1/2 blend is frozen without sensitivity analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is XConf and how does it estimate confidence?
XConf is a training-free method for estimating how likely a language model's answer is correct. It keeps a bank of the model's own past episodes, each storing the task, the model's self-reflection, the confidence it stated at the time, the verified outcome, and a short lesson. For a new answer, a Recall stage retrieves the 50 most similar past episodes (matched on task meaning and similar stated confidence) and computes how often those actually succeeded; a Reflect stage then shows the model this track record, asks it to name a recurring failure pattern, and elicits a revised confidence. The final estimate averages the statistical hit rate and the model's revised number.
How much better is XConf than self-consistency?
On 23 of 24 model-dataset comparisons across nine benchmarks, XConf matches or beats ten-sample self-consistency on AUROC (discrimination) while reporting substantially lower calibration error (ECE), and it costs about one answer generation versus ten. However, many individual wins fall within overlapping 95% bootstrap confidence intervals, so the robust reading is parity-or-better at roughly a tenth of the cost; the clearest separations are on interactive agent tasks, where resampling is impractical and XConf also beats a trained verifier on the hardest benchmark (AppWorld).
Does XConf require model weights, logits, or fine-tuning?
No. It is black-box and training-free: it works through text prompts and an off-the-shelf text embedder, needs no logit or token-probability access, and never updates the model's weights. It does, however, require externally graded outcomes for past episodes — the paper shows that labels produced by the model's own judgment (e.g., self-consistency majority votes) fail, because they are wrong exactly on the confidently-wrong cases the bank exists to catch.
What does the claimed 8.7-point selective-prediction gain mean?
On AppWorld with Gemini 3.5 Flash, ranking completed episodes by XConf confidence and dropping the least-confident 10% raised the success rate on the remaining kept set from .805 to .892, an 8.7-point gain. Averaged over all 36 evaluation cells, dropping the bottom 10% added 4.8 points. In practice this means shipping the confident outputs automatically and sending the flagged tail to retries, escalation, or human review.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Only main-body text, tables 1-7, and appendix references were extractable; figures (3, 4, 5, 8-14) were judged from captions and surrounding text, not from rendered plots.
- Sections of the appendix referenced heavily (e.g., E, F, K details) are partially summarized in the embedded text; some protocol details may be paraphrased or truncated.
- No explicit code/data release statement appears in the extracted text; a project identifier 'caiqizh/xconf' appears in the margin, so codeAvailable=false reflects the text only.
- Model names and benchmark sizes are 2026-era and were taken at face value from the text without external verification, per the no-network constraint.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.