When Cosine Similarity Fails to Reflect Linearly Accessible Structure in Dialogue Models
Yu Sun, Mengyin Lu, Cong Feng, Guangming Lu, Huimin Han
cs.CL
Quick verdict
Argues cosine similarity underestimates linearly decodable persona structure in dialogue LLMs; careful, well-controlled empirical study with modest novelty.
Summary
The paper asks when cosine similarity over transformer hidden states fails to reflect linearly decodable structure. Using three 7–8B chat-tuned models (Llama-3.1-8B-Instruct, Qwen-3-8B, Mistral-7B-Instruct-v0.3) on a 30-persona multi-turn dialogue corpus (600 trajectories per model, activations captured at the last prefill token across six layers), the authors compare linear-probe AUC, ambient cosine kNN, supervised-subspace cosine kNN (top-10 SVD of the probe weight matrix), and a matched-rank PCA control under group-by-trajectory cross-validation. At the headline layers, linear probes reach 0.733–0.972 macro AUC while cosine kNN reaches 0.556–0.771; the 10-dimensional supervised subspace lifts kNN by +0.105 to +0.138, whereas PCA subspaces never help and can hurt (−0.022 to −0.037). Controls include anisotropy corrections, Euclidean kNN, layer and kNN-k sweeps, probe-regularization sensitivity, and a sample-size robustness check. A matched-cardinality contrast (5-class persona-emotion vs SST-5) shows the gap is absent in single-sentence sentiment. Turn-slice and cumulative-pool analyses find no systematic accumulation across turns, though two of three models violate the pre-registered within-subspace AUC-invariance criterion and Llama violates turn-invariance. Limitations: one hand-written persona corpus, a correlational design, last-prefill-token capture only, and code/data only promised for release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On a 30-class persona task across three 7–8B chat models, ambient cosine kNN AUC (0.556–0.771) substantially underestimates the structure linear probes decode from identical hidden states (0.733–0.972 macro AUC).
- A 10-dimensional supervised subspace (top-10 SVD of train-fold probe weights) recovers most of the gap (+0.105 to +0.138 lift), while a matched-rank PCA subspace never helps and can hurt (−0.022 to −0.037).
- The cosine–linear mismatch is regime-dependent: it is absent in single-sentence SST-5 sentiment at matched 5-class cardinality, ruling out attribute cardinality as a sufficient explanation.
- The gap does not systematically accumulate across dialogue turns and subspace projection support is stable, but two of three models violate the pre-registered within-subspace AUC-invariance criterion and Llama violates the turn-invariance criterion.
Strengths
Matched-rank PCA control is the load-bearing design element
Without it, any low-dimensional projection could reduce cosine kNN's curse of dimensionality. PCA lift is ≤0 for every tested ksub ∈ {1,2,5,10,20,50} and all layers (§4.4, Appendix C), which is what licenses the 'task-aligned, not variance-aligned' interpretation of the supervised lift of +0.105 to +0.138 (§4.3).
Pre-registered decision rules reported including their failures
Appendix K specifies thresholds (lift/gap ≥0.05, |ΔAUC|≤0.03, |ΔL|≤0.05) and §4.6–4.7 report violations verbatim: Llama fails turn-invariance (0.053), Llama and Mistral fail within-subspace AUC invariance (−0.072, −0.060). Honest negative reporting is rare and valuable.
Unusually broad robustness battery
Disciplined variance reporting and candid confound acknowledgment
Weaknesses
Pooled-vs-per-turn tension undermines the headline interpretation for two of three models
Internal numeric inconsistencies
Narrow evidential base for the regime-dependence claim
No artifacts available at review time
Statistical reporting gaps in the main text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper documents a measurement phenomenon: on dialogue-conditioned hidden states from three 7–8B chat-tuned LLMs (Llama-3.1-8B-Instruct, Qwen-3-8B, Mistral-7B-Instruct-v0.3), ambient cosine similarity (kNN AUC 0.556–0.771 at k=10) substantially understates the persona structure a linear probe decodes from the same activations (macro one-vs-rest AUC 0.733–0.972; Table 1). The authors position this explicitly as an empirical regularity, not a mechanism: §1 disclaims any theorem or causal claim, and the 'Relation to classical results' paragraph concedes that probes beating distance-based methods is folklore. The genuine contribution claims are (i) regime-dependence — the gap appears in multi-turn dialogue but not SST-5 at matched 5-class cardinality (§4.5) — and (ii) the supervised-vs-PCA subspace contrast: a 10-d probe-derived subspace lifts +0.105 to +0.138 while matched-rank PCA never lifts (−0.022 to −0.037; §4.3–4.4). Scope statements are carefully hedged (§5.4, Appendix D), which is commendable; the main overreach risk is the title's general framing versus one corpus of evidence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
On a 30-persona multi-turn dialogue task across three 7–8B chat LLMs, cosine similarity between hidden states (kNN AUC 0.556–0.771) systematically underestimates what a linear probe can decode from the same activations (AUC 0.733–0.972). Projecting onto a 10-dimensional probe-derived subspace recovers most of the gap, while a matched-rank PCA subspace does not. The effect appears in dialogue-conditioned representations but not in single-sentence SST-5 sentiment classification.
Does this mean cosine similarity is useless for analyzing LLM representations?
No. The claim is narrower: in this dialogue-conditioned, multi-axis persona setting, ambient cosine fails to reflect linearly decodable persona structure. The authors recommend a four-metric diagnostic (linear probe, cosine kNN, supervised-subspace kNN, PCA-subspace kNN) instead of cosine alone. Anisotropy-corrected cosine behaves essentially the same as raw cosine, and single-sentence sentiment shows no such gap.
Is code or data available for this paper?
Not at the time of this review. Appendix L promises release of activations, analysis scripts, and result CSVs 'upon publication'; the persona corpus, prompt templates, and all quantitative artifacts were not yet accessible, so the numbers could not be independently checked.
What are the main caveats to the headline claim?
The evidence comes from one hand-written 30-persona corpus and three 7–8B models with activations captured only at the last prefill token. The design is correlational with no behavioral validation. Two of three models violate the authors' pre-registered within-subspace AUC-invariance criterion and Llama violates turn-invariance, and per-turn slices show near-zero supervised lift for two models at turn 1 — so part of the headline gap may come from pooling across turns.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 appear only as captions in the extracted text; visual claims (monotone trends, error bands, shaded CIs) could not be independently verified.
- Several quantitative artifacts (turn_sweep_summary.csv, C-sweep CSV, per-seed subsample numbers) are referenced but only promised, so some stated values could not be cross-checked.
- The 2026-dated citations (arXiv 26xx) could not be verified against the literature in this offline review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.