Consistency Without Alignment: Item-Sensitive Language Models Indistinguishable From Random
cs.AI · cs.CL
Quick verdict
Argues item-sensitivity ≠ competence via a game-derived task with closed-form references; rigorous and honest, but rests on one tile with no human data yet.
Summary
The paper argues that item-sensitivity statistics — permutation consistency, self-consistency, agreement across resamples — are necessary but not sufficient evidence that a language model is performing a task. The setting is a forced-choice signalling task abstracted from the board game Deception: Murder in Hong Kong, where a fit-maximising strategy, a posterior-maximising (Bayesian) strategy, and uniform random selection are all computable in closed form from human property norms. Seven models (Qwen3 0.6B–8B, two base variants, OLMo-2-7B-Instruct) were scored under three scoring rules: all 21 model-by-rule cells are item-sensitive (excess +0.024 to +0.260), yet 8 are statistically indistinguishable from random selection, 5 describe the target worse than random, and excess correlates with distance from random at only r = 0.30. A literal cosine-similarity baseline outperforms most models; an RSA pragmatic layer moves baselines toward random rather than toward the Bayesian reference; labelled multiple-choice prompts carry no content signal. Main limitations: the primary analysis rests on a single tile (125 size-conflict items), the human condition is designed but not collected, the Bayesian oracle assumes a literal listener, and base models were run under an instruct chat template. Generalisation to all consistency-based evaluations is argued conceptually rather than demonstrated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Item-sensitivity statistics (permutation/self-consistency) are necessary but not sufficient evidence of task competence: all 21 model-by-rule cells are item-sensitive while 8 of 21 are statistically indistinguishable from random selection and the two measures correlate at only r = 0.30.
- A literal cosine-similarity baseline with no pragmatics outperforms most tested LLMs on the fit axis (post_norm 0.640, above 15 of 21 model cells; Sec. IV-F).
- Adding an RSA pragmatic layer over SWOW or ConceptNet baselines moves choosers toward random (0.593→0.510 and 0.651→0.472) rather than toward the Bayesian reference (Sec. IV-H.2).
- A labelled multiple-choice answer format carries no content signal in this setting: permutation consistency is at or below chance in all 32 model-by-tile-by-label cells (Sec. IV-H.4).
- Any evaluation reporting item-sensitivity, permutation consistency, or self-consistency without an independent reference for the measured quantity is exposed to the same confound (argued conceptually, Sec. I and IV-C).
Strengths
Closed-form reference design
The salience (o_fit), Bayesian (o_bayes) and uniform-random references are computed exactly over an enumerable 100-hypothesis space with a parameter-free Shepard-exponential fit on human property norms (Sec. III-D–III-F); the random reference even ships with its own across-item spread (post_norm 0.459, SD 0.062, IV-B), so the dissociation is demonstrated, not assumed.
Pre-registration with falsified predictions reported as results
The instrument (fit function, items, scoring rule, sensitivity battery) was frozen before any model ran (Sec. III); the pre-registered format-L prediction failed and is reported as such (IV-H.4), and the authors correct their own earlier uniform null to a marginal null (IV-A).
Rule-robustness discipline
Infrastructure and honesty hygiene
Weaknesses
Single-tile dependence
Missing human condition
Chat-template confound concentrated in the key cells
Unreported dual decay-family analysis
Overreaching generalisation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes one central methodological claim: item-sensitivity statistics (permutation consistency, self-consistency) are necessary but not sufficient evidence of task competence. It supports this in a purpose-built forced-choice signalling task abstracted from Deception: Murder in Hong Kong (Sec. III-A): a 10 Means × 10 Clues concept grid (100-hypothesis space), one property tile with 3–8 options, where the fit-maximising choice (o_fit), the posterior-maximising choice (o_bayes), and uniform random selection are all computable in closed form from human property norms (THINGSplus) rather than fitted to text or model output. Secondary claims: a literal cosine baseline outperforms most tested LLMs (IV-F); an RSA pragmatic layer moves baselines toward random, not toward the Bayesian reference (IV-H.2); labelled multiple-choice prompts carry no content signal (IV-H.4). The broad generalisation claim ('any evaluation that relies on item-sensitivity...') in the Abstract and Sec. I is argued conceptually, not demonstrated.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'consistency without alignment' mean in this paper?
It means a language model can respond reliably to the specific input it is shown (item-sensitivity, e.g., consistent choices across prompt permutations) while still not tracking the quantity the evaluation intends to measure. The paper shows model cells whose choices are item-sensitive yet statistically indistinguishable from a random chooser on the task's target coordinate, so consistency alone is not evidence of competence without an independent reference.
Did the study include human participants?
No. The human condition (Condition 8) was designed, pre-registered and piloted for instrument testing only, but no human data had been collected when the paper was written; collection is planned via Prolific with informed consent. Every reported result is model-side.
Which language models were tested?
Seven: the Qwen3 instruct ladder at 0.6B, 1.7B, 4B and 8B parameters, the Qwen3-1.7B and Qwen3-8B base models (a post-training ablation), and OLMo-2-7B-Instruct as a cross-family control. Each was scored under three log-probability rules (PMI, unnormalised sum, per-token mean), giving 21 model-by-rule cells.
Why is the random baseline so central to the paper's argument?
Because it is computable exactly in this task, it converts 'how consistent is the model' into 'where does the model sit relative to a chooser that ignores the item'. Eight of the 21 model-by-rule cells sit at coordinates not statistically distinguishable from that random chooser while showing positive item-sensitivity, which is the paper's evidence that sensitivity statistics need an external reference.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 are represented only by captions and extracted axis/label text; the plotted data points in Fig. 2–3 cannot be checked against a visual rendering.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.