arXiv paper2026-09-02· machine learning — LLM interpretability and alignment auditing· empirical study

Knowing, and Saying It Only When Asked: LLM Endognostics and the Schizognosis of Minerva-7B

cs.CL · cs.AI · cs.LG

Quick verdict

Proposes 'endognostics' auditing of Minerva-7B: outputs hide internally-present knowledge; ablation evidence is careful but rests on 25 prompts from one model.

Summary

The paper introduces 'LLM endognostics', a white-box auditing framework that reads and causally manipulates latent representations in the residual stream, applied to Minerva-7B-Instruct-v1.0, a natively Italian 7B model. The CEM protocol tests 124 minimal risky/safe prompt pairs: behavior is identical on both members for 79 pairs (63.7%), yet a Jacobian-lens reading shows internal risk differentiation, significant after Holm–Bonferroni correction in 2 of 12 categories (sycophancy, confidentiality). The KR protocol crosses 25 facts with five falsehood framings: presupposed falsehoods cause errors on 18/25 facts while the true token remains readable within the 'workspace band' (layers 19–29). Ablating the false-token direction restores the truth in 11/25 suppressed cases (44%) versus zero for three control directions (exact McNemar p = 0.00098), confirmed by blind human annotation (binary κ = 0.68). An out-of-sample linear probe decodes truth at 77% accuracy (layer 10) but its ablation recovers nothing, supporting the claimed dissociation between linear decodability and causal verbalization. Main limitations, candidly listed in Sections 7/11: a single model and checkpoint; causal claims rest on 25 prompts (18 from one framing); the dead-band threshold was calibrated partly circularly on included pairs; the lens was fitted on only 100 sequences; the five-level rubric is unreliable (κ = 0.54), and the κ values contain an internal inconsistency (0.54 vs 0.45).

Review complete

Assessment at a glance

machine learning — LLM interpretability and alignment auditing

risk flags identified

● high 0

● medium 3

● low 2

Reproducibility readinesshigh

Code & data availableyes

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

5 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

On 63.7% (79/124) of contrastive risk pairs, Minerva-7B's observable behavior is identical for the risky prompt and its safe control, making behavior-only evaluation uninformative (Section 4, Table 1).
A Jacobian-lens internal reading recovers risk differentiation the output discards, significant after Holm–Bonferroni in 2 of 12 categories (sycophancy p = 0.0020; confidentiality p = 0.0029, Table 2).
The model adopts presupposed falsehoods on 18/25 facts while the true token remains readable in the workspace band; ablating the false-token direction restores the truth in 11/25 (44%) with zero corrections across three control directions (exact McNemar p = 0.00098).
An out-of-sample linear probe decodes truth at 77% accuracy (layer 10) but its orthogonal ablation yields 0% recovery, establishing that linear decodability does not imply causal control over verbalization (Section 7).
Four training-origin hypotheses (English-centric SFT/DPO judge, web-noise entity attribution, toxicity-only safety data, economical refusal) are explicitly unverified interpretations of public training documentation (Section 8).

Strengths

Properly controlled causal ablation
Section 6.1 runs three control directions — random equal-norm, unrelated token, true-token — all yielding 0/25 corrections while the target arm changes 22/25 outputs (random arm changes 7, true-token arm 12), ruling out the 'any perturbation works' objection; §6.2 further validates the instrument itself (true-token ablation removes the truth in 99/116 vs 3/116 for random, exact McNemar p ≈ 6e-28).
Rare epistemic honesty as method
Section 7 ('What we could not establish') reports self-weakening measurements: the substring auto-verdict is unreliable, the lens's reading advantage over the logit lens is not significant (4–0 discordance, p = 0.125), and the intervention degrades 3/25 outputs; §11 discloses that the ε = 1.0 dead band was calibrated on 18 of the very pairs it classifies.
Crossed design cleanly isolates framing
The dissociation result is tested from both sides
Annotation integrity process is documented

Weaknesses

Single-model evidence base for a general framework
Thin causal base and underpowered categories
Internal inconsistencies and editing residue
Partly circular threshold calibration
Pipeline-dependent verdicts

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

The paper formalizes 'LLM endognostics', a white-box auditing framework that reads the residual stream through a Jacobian lens (a vocabulary projection of intermediate activations targeting the 'verbalizable subspace', §2.1, following Gurnee et al.) and intervenes on it by ablating token-specific directions. It is applied to a single target, Minerva-7B-Instruct-v1.0, and coins 'schizognosis' for the gap between what the model represents internally and what it emits. Two protocols carry the argument. The Contrastive Endognostic Margin (CEM, §2.2/§5) scores 124 minimal risky/safe prompt pairs across 12 professional-risk categories; the Knowledge Robustness protocol (KR, §2.3/§6) crosses 25 facts with five falsehood framings plus 20 true-premise controls and 15 excluded multi-token items. Central claims: (1) behavior-only evaluation is blind on 63.7% of pairs (79/124, Table 1); (2) internal reading recovers risk differentiation the output discards; (3) presupposed falsehoods are adopted on 18/25 facts while the truth stays readable in the workspace band; (4) targeted ablation restores the truth in 44% of suppressed cases; (5) a linear probe dissociates representation (77% at layer 10) from causal control (0% recovery) — the paper's most interesting theoretical result.

Methodology Assessment

The design is more careful than typical single-model audits. The KR crossed design (fact × framing) with fact-level pairing isolates framing as the manipulated variable, and the framing result (presupposition 18/25 vs 1–4 for assertive framings, Cochran's Q = 42.4, Table 3) depends on no lens, band, or judge — the paper says so correctly. The causal step (§6.1) is properly controlled: three control directions (random equal-norm, unrelated token, true token) all yield 0/25 corrections, the random arm changes 7 outputs and degrades none, and the true-token arm changes

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 5 strengths & 5 weaknesses, analyzed in detail
5 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is 'schizognosis' in this paper?

The authors' coined term for a model internally representing a fact or risk distinction while its generated text suppresses it — for example, answering a question built on a false premise incorrectly even though the true answer is still readable inside its internal activations.

What is LLM endognostics?

A white-box auditing framework that reads a language model's residual stream through a Jacobian lens (a vocabulary projection of intermediate layers targeting the verbalizable subspace) and causally manipulates it by erasing token-specific directions, to compare what the model represents internally with what it actually says.

Which model was studied and what were the main numbers?

Minerva-7B-Instruct-v1.0, an Italian-English 7B model. On 124 risky/safe prompt pairs, 63.7% produced identical behavior on both members; presupposed falsehoods were adopted on 18 of 25 facts; targeted ablation restored the true answer in 11 of 25 suppressed cases (44%) versus zero for three control directions.

Does the paper show this happens in all LLMs?

No. It is a single-model case study of one checkpoint of Minerva-7B. The authors explicitly make no claims about other models and schedule base-checkpoint and English-translation experiments as future work to test generality.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Only one representative contrastive pair per category is reproduced in Appendix B; the full 124-pair set is stated to live in released source files, so pair-level prompt quality cannot be audited from the text alone.
Figures 1–4 appear only as captions in the extraction; every quantitative claim in this review is taken from tables and body text, none from figure pixels.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper