Summarization Bias: The Directional Collapse of Objective Projection into Told-Mode Labels in Large Language Models --- A Conceptual Framework and Registered Test Protocol
cs.CL
Quick verdict
Pre-registered, falsifiable construct: LLMs collapse shown-mode prose into told labels. Elegant protocol design, but zero new data and only self-citations.
Summary
This position paper introduces 'summarization bias': the hypothesized systematic tendency of LLMs to convert narrative content requiring reader inference ('shown' mode, per the author's Bulut Doctrine and Objective Projection framework) into the explicit declarative label that names it ('told' mode). The bias is claimed to operate in two regimes — generative (models asked to render emotion through physical detail default to naming it) and evaluative (LLM judges reward told-mode explicitness) — the latter mattering because LLM judges and reward models would impose directional selection pressure on prose. The author explicitly claims no validation: the paper defines the construct, argues its distinctness from verbosity, position, self-preference and sycophancy biases via length-matched stimuli, rereads a completed three-study reliability report as directional evidence — against a human count of 9/100 scenes for 'materialized metaphor', five machine labellers returned 0, 1, 40, 72 and 78, with four of five at chance-level Cohen's kappa — and pre-registers a four-stage test (matched pairs, generative SI comparison, counterbalanced judge test, discriminant checks) with fixed falsifiers including construct withdrawal. Main limitations: no new data; all four references are the author's own Zenodo/arXiv work; core constructs (told/shown coding, Suppressed Information Index) rest on unvalidated rater judgment; and the supporting evidence is, by the author's own admission, equally consistent with ill-defined annotation rules.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM errors on the told-shown narrative axis are directional — predominantly toward explicit declaration — rather than symmetric noise (Section 1.2).
- Summarization bias operates in a generative regime (models default to naming emotions they were asked to render via physical detail) and an evaluative regime (LLM judges reward told-mode explicitness).
- An evaluator with this bias, deployed as a reward model or automated judge, would impose directional selection pressure degrading prose toward flat declaration (Section 1.3).
- The bias is dissociable from verbosity and sycophancy by design — via length-matched stimuli and varied prompt framing — and collapses into those known biases if the preference does not survive (Sections 2 and 5.4).
- A completed three-study reliability study — five machine labellers returning 0, 1, 40, 72 and 78 versus a human count of 9 per 100 scenes on materialized metaphor — is consistent with, but does not confirm, the construct (Section 3).
Strengths
Genuine pre-registration discipline
Section 5 fixes falsifiers, effect thresholds (Cohen's h ≥ 0.3 or OR ≥ 2, 'fixed now') and a κ ≥ 0.60 admission gate before any data exists; §5.5 explicitly bans post-hoc construct-fitting and commits to publishing a withdrawal at the same prominence as a confirmation.
Honest evidentiary hedging
§3 and §6 label the reread evidence 'consistent with, not confirming', and volunteer the disconfirming facts themselves: labellers disagreed 9 vs 82 on a related feature, and the source reliability report finds the spread at least as consistent with loose definitions as with model bias.
Dissociation-by-design from known biases
Directional framing makes a folk claim testable
Weaknesses
No new data whatsoever
Closed citation ecosystem
Motivating evidence is weak as presented
Judgment-dependent measurement chain
Underpowered stimulus plan
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a pre-validation position paper: it introduces the construct of *summarization bias* — the hypothesized tendency of LLMs to convert narrative content that a reader must reconstruct from physical cues (shown mode, per the author's Bulut Doctrine and Objective Projection framework) into the explicit label that names it (told mode) — and pre-registers a test before any data collection. The author is explicit (Abstract, §6) that nothing is validated. The claim structure is carefully scoped: (i) errors on the told–shown axis are *directional*, not symmetric noise (§1.2); (ii) the bias operates in a generative regime (H1g) and an evaluative regime (H1e), the latter consequential because LLM judges and reward models would impose selection pressure toward flat declaration (§1.3); (iii) the construct is dissociable from verbosity, position, self-preference and sycophancy biases (§2). Section 3 rereads a completed three-study reliability report [3] as directional motivation; Sections 4–5 fix operational definitions and a four-stage protocol with pre-committed falsifiers, including construct withdrawal (§5.5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'summarization bias' in large language models?
A hypothesized — not yet proven — tendency of LLMs to replace narrative content that readers must infer from physical detail ('shown' writing) with the explicit label naming it ('told' writing), both when generating prose and when judging its quality. The term and its test protocol are defined in this paper.
Has summarization bias been proven or measured?
No. The paper explicitly collects no data. It defines the construct and pre-registers a four-stage experiment whose outcome would confirm, bound, or formally withdraw the construct. The only supporting evidence is a reinterpretation of the author's earlier annotation-reliability study.
What evidence does the paper offer for the bias?
In the author's reliability study, human raters found 'materialized metaphor' in 9 of 100 scenes while five machine labellers returned counts of 0, 1, 40, 72 and 78, with four of five at chance-level Cohen's kappa. The author himself notes this is equally consistent with poorly defined annotation rules, so it motivates but does not confirm the bias.
What result would falsify summarization bias?
Per the registered protocol: if LLM judges do not prefer told-mode texts over matched shown-mode texts significantly more often than human raters do (Stage 3), or if that preference disappears under length-matching or neutral prompt framing (Stage 4), the construct is withdrawn and the withdrawal published.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The companion reliability report [3] and the Sn pilot [4] were assessed only as summarized within this paper; their full protocols and tables were not directly examined.
- The report is short, self-contained prose with no figures or tables in the extracted text, so nothing appears lost to extraction.
- All quality judgments apply to the protocol design rather than results, since the paper is pre-data by explicit design.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.