When Confidence Signals Disagree: Local and Global Confidence in Autoregressive Language Models
cs.LG · cs.AI
Quick verdict
Token-probability vs sampling-frequency confidence disagree in Llama 3.3 70B; careful analysis but tiny scale, one quantized model, partly by construction.
Summary
This empirical study asks whether different quantities that practitioners call 'confidence' in an autoregressive language model actually measure the same thing. Using a locally served, 4-bit-quantized Llama 3.3 70B on 100 MMLU (high_school_us_history) and 99 ARC Challenge questions, the author compares local confidence (the probability of the greedy-selected answer token) with global confidence (the frequency of the modal answer across 60 samples at T=0.7). The two signals correlate weakly (r=0.086 MMLU, r=0.175 ARC) and differ in their association with correctness: global confidence correlates moderately (r=0.494, 0.374) while local confidence shows little association (r=0.102, 0.072). On ARC, larger local–global gaps track sampling instability: higher answer entropy (Spearman ρ=0.418), more distinct answers (ρ=0.41), and lower modal concentration (ρ=−0.42); unstable questions show much larger gaps (rank-biserial 0.76). The gap–entropy association persists across 200 disjoint 15/15 sample partitions (mean ρ=0.397), ruling out shared finite-sample variation. On MMLU, where only 4 of 100 questions are unstable, the relationship is weak (ρ=0.215). Key limitations: a single quantized model, roughly 100 questions from one MMLU subject, a near-saturated local readout, and a partly by-construction coupling between global confidence and the instability measures that the design cannot fully remove. Code is released on GitHub.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Local token confidence and global modal-answer frequency are weakly correlated in Llama 3.3 70B (r=0.086 on MMLU, r=0.175 on ARC) and cannot be treated as interchangeable readouts.
- Global confidence is moderately associated with correctness (r=0.494 MMLU, r=0.374 ARC) whereas local token confidence shows little association (r=0.102, r=0.072).
- On ARC, larger local–global confidence gaps are associated with greater sampling instability (entropy ρ=0.418, distinct answers ρ=0.41, modal concentration ρ=−0.42, stable/unstable rank-biserial 0.76), and the gap–entropy association persists under 200 disjoint-sample partitions (mean ρ=0.397).
- The MMLU relationship is substantially weaker because sampling instability is rare there (4/100 unstable questions), so the diagnostic's observability depends on sufficient instability variation.
- Confidence should therefore be treated as an explicitly defined measurement choice rather than a single intrinsic scalar property of a model.
Strengths
Disjoint-sample robustness analysis directly tests the main confound
Section 5.5 re-estimates the gap–entropy association on 200 random non-overlapping 15/15 partitions; ARC holds at mean ρ=0.397 vs pooled 0.418 (range [0.354, 0.422]), cleanly ruling out shared finite-sample variation as the explanation.
Companion control isolates the gap from mere low local confidence
Section 6.4/Table 3 shows clocal alone vs entropy is ρ=−0.31 on ARC, weaker than the gap's ρ=0.418, and clocal is obtained from a separate greedy call sharing no stochastic samples with the instability estimates.
Statistical reporting is above the bar
Honest scoping of claims versus construction
Constrained-answer design avoids a known confound
Weaknesses
Structural coupling between the gap and instability is under-disentangled
Decoding-regime confound in the readout contrast
Local readout saturates, likely attenuating its correlations
Very small and narrow question sample
Single quantized model with a labeling inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper poses a measurement question: within one autoregressive model (Llama 3.3 70B, 4-bit q4_K_M via Ollama), do two readouts both called 'confidence' — the probability of the greedy-selected answer token (local, Eqs. 2/9) and the modal-answer frequency under 60 samples at T=0.7 (global, Eqs. 3/10) — agree, carry the same information about correctness, and does their absolute gap D(x) (Eq. 4) track sampling instability? The framing is commendably modest: the authors state repeatedly (Sections 3, 4.3) that such disagreement follows from the definitions, and position the contribution as quantifying its magnitude, its differential correctness association, and its link to instability. Headline numbers: r(clocal, cglobal)=0.086 (MMLU, n=100) and 0.175 (ARC, n=99); r(cglobal, correct)=0.494/0.374 versus r(clocal, correct)=0.102/0.072 (Table 2); on ARC, D vs entropy ρ=0.418, vs distinct answers ρ=0.41, vs modal probability ρ=−0.42, and stable-vs-unstable rank-biserial 0.76 (Table 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the difference between local and global confidence in language models?
In this paper, local confidence is the probability the model assigns to the greedy-selected answer token at a single decision point, while global confidence is how often the most common (modal) answer appears across many stochastic samples of the same question (60 samples at temperature 0.7 in the experiments). The two summarize different aspects of the same predictive process, so they need not agree.
Which confidence measure predicts correctness better according to this study?
The sampling-based global confidence. It correlates moderately with answer correctness (Pearson r=0.494 on MMLU, r=0.374 on ARC Challenge), whereas the local token probability shows little association (r=0.102 and r=0.072 respectively) in a 4-bit quantized Llama 3.3 70B.
Why is the result weaker on MMLU than on ARC Challenge?
Because MMLU questions are almost all stable under sampling: only 4 of 100 MMLU questions produced more than one distinct answer at temperature 0.7, versus about 11% on ARC. With almost no instability variation, the association between the confidence gap and sampling instability is weak (Spearman ρ=0.215 on MMLU vs 0.418 on ARC).
Is code or data available for this paper?
Yes. The paper links code at github.com/julioadl/stochastic-parrots. Experiments use public benchmarks (MMLU and ARC Challenge) and a locally served, 4-bit quantized Llama 3.3 70B via Ollama, so the setup is reproducible given roughly 48 GB of RAM/VRAM.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables were extracted as plain text; any figures in the original PDF were not available for analysis, and Figure-independent claims were verified only against Tables 1-3.
- The paper inconsistently labels the model tag in Section 5 (q4_H_K) versus its footnote (q4_K_M); this review assumes the footnote's Q4_K_M is correct.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.