When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models
Gnaneswar Villuri, Hashmath Shaik, Alex Doboli
cs.CL · cs.AI
Quick verdict
Argues small-LLM probe-behavior gaps are mostly a miscalibrated logit threshold, not hidden knowledge; controlled, quantitative, largely convincing.
Summary
The paper asks when linear probes on hidden states genuinely detect knowledge that a language model's answer conceals. Using a controlled verification testbed — 1,200 logical-conclusion trials (half valid, half corrupted by a single semantic edit) built from a natural-deduction corpus — the authors show Qwen3-0.6B answers YES to every trial (50% accuracy) while structure-held-out linear probes reach 0.964 AUC. Tracing the verdict through access levels, they find the YES−NO logit margin already ranks validity at 0.887 AUC; behavior fails because the margin distribution sits +4.6σ above the decision threshold. Across 90 prompt configurations of five models, behavioral accuracy tracks the measured threshold offset as a single function (Spearman −0.93). A one-parameter, cross-validated threshold repair lifts 0.6B accuracy from 50% to 81%, matches few-shot prompting (81.4%), and margin decoding reaches 94.2% at 8B. A maze task with derangement foils validates the diagnostic framework: on long paths it correctly reports absence of a decodable verdict. In the standard generation setting, answer-surface feature baselines match probes, cautioning against probing claims. Limitations: synthetic English tasks, models ≤8B, correlational (non-causal) probes, and a substantial candidate-intrinsic plausibility component in the logic-domain signal.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In a closed-channel verification testbed, the YES−NO logit margin already ranks validity at 0.887–0.980 AUC across scales, so most probe-detectable 'hidden knowledge' is present in the output distribution.
- The behavioral failure is dominated by a single threshold offset (+4.6σ at Qwen3-0.6B), not a readout-direction error; across 90 prompt configurations of five models, behavioral accuracy is a single declining function of offset (Spearman −0.93).
- A one-parameter, cross-validated threshold repair lifts 0.6B behavior from 50.0% to 81.0% and margin-decodes 8B to 94.2%; few-shot prompting acts predominantly by the same recentering (+4.64σ → +0.01σ, rank correlation 0.79).
- Comparing probe to margin separates concealed, miscalibrated, and absent regimes, correctly reporting absence on hard maze paths (margin 0.538, probe 0.581).
- In the standard generation setting, a 10-feature answer-surface baseline matches or beats hidden-state probes and heuristic labels manufacture probe signal, so open-channel probing claims are unestablished.
Strengths
Closed-channel testbed with strong artifact controls
Exact single-token labels, LOSO over 30 deduction structures, two foil generations (503 deterministic rule-generated foils; 73 word-multiset-preserving foils still separated at 0.897 AUC), plus text-only bounds (lexical overlap 0.759, frozen encoder 0.642) rule out the usual surface-cue confounds.
Statistical discipline uncommon in probing work
Nested layer selection inside folds (stable at layer 18 across all 30 folds), cluster bootstraps over structures (2,000 resamples), within-structure permutation tests (p ≤ 10⁻³), and paired CIs on probe−margin gaps (+0.076 [0.057, 0.096] at 0.6B; +0.008 [0.0001, 0.0189] excluding zero at 8B).
The margin baseline is a genuinely useful reframing
Discriminant validity, not just sensitivity
Matched repairs triangulate the mechanism
Weaknesses
Candidate-intrinsic signal inflates headline AUC
Narrow external validity vs sweeping claims
Clustered factorial correlation
Unexplained cross-model probe transfer
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper sits at the intersection of two literatures that rarely cite each other: interpretability claims that models 'know more than they show' (Kadavath et al. 2022; Azaria & Mitchell 2023; Orgad et al. 2025) and the label-prior pathologies of prompted classification (Zhao et al. 2021; Holtzman et al. 2021). Its central claim is that, in a controlled verification setting, most probe-visible 'hidden knowledge' is already present in the model's own YES−NO logit margin, and the behavioral failure is dominated by a single miscalibrated threshold rather than a misaligned readout direction. Subsidiary claims: (i) across 90 prompt configurations of five models, behavioral accuracy collapses onto a single function of threshold offset (Spearman −0.93); (ii) a one-parameter repair recovers most of the gap; (iii) in the standard generation setting, answer-surface baselines reproduce probe results, so open-channel probing claims are unestablished. A three-regime diagnostic (miscalibrated / concealed / absent) is proposed and validated on a maze task.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'behavior-concealed knowledge' mean in this paper?
The paper shows a small language model (Qwen3-0.6B) that answers YES to every logical verification question — appearing to know nothing — while linear probes on its internal states (0.964 AUC) and even its own YES−NO logit margins (0.887 AUC) clearly distinguish valid from invalid conclusions. The knowledge exists but the answers conceal it, mostly because a decision threshold is shifted far to one side, not because the information is buried deep in the network.
Did the authors find that probing is unnecessary for detecting LLM knowledge?
In their controlled setting, mostly: the model's own YES−NO logit margin (readable without any probe training) recovers nearly all of what probes find, and the extra probe advantage shrinks from +0.076 AUC at 0.6B to +0.008 at 8B. In the standard text-generation setting, simple answer-surface features matched probes, so probes added nothing demonstrable there either. But on tasks where the model genuinely fails to compute a verdict (long maze paths), probes correctly find nothing.
How was the 50%-to-81% repair achieved, and was it overfit?
By fitting a single decision threshold on the YES−NO logit margin with leave-one-structure-out cross-validation — the threshold was never fit on evaluated structures. Accuracy rose from 50% to 81% on Qwen3-0.6B, and median-centering (which learns nothing) matched it. Four-shot prompting achieved 81.4% by the same mechanism: it recentered the threshold from +4.64σ to +0.01σ while preserving the margin's ranking (rank correlation 0.79).
What are the paper's main limitations?
Tasks are synthetic and English-only (logic verification and a maze domain); models tested are ≤8B parameters; probes are correlational with no causal activation steering; and part of the logic-domain probe signal reflects candidate-intrinsic plausibility rather than premise-dependent checking — the authors isolate the premise-dependent part in specific foil families and in the maze domain.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The supplement (exact prompts, foil-generation code, folds, scoring rules) is referenced but not part of the extracted text; reproducibility judgments rest on main-text descriptions only.
- The metadata abstract is cut mid-word ('undet'), but the embedded paper text contains the complete abstract and references — no content is missing from the review basis.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.