Clarification Is Not Correction: LLMs Fail to Let Go
Jianzhe Lin, Xiaolin Li, Fei Wang, Robert Douglas, Rajeshkumar Golani, Jubin Chheda
cs.AI
Quick verdict
Claims LLMs commit early and treat later clarification as context, not correction; compelling diagnosis, but judge circularity and no stats tests weaken it.
Summary
The paper argues that a distinct class of multi-turn LLM failures is not memory loss but premature commitment: an ambiguous early turn collapses into one task interpretation, and later clarification refines rather than invalidates that state — a failure mode the authors call 'early posterior collapse', diagnosed through 'old-hypothesis contamination' in final outputs. Using controlled writing, planning, and coding dialogues (290 generated tasks plus 50 hand-crafted ones; ~13,900 API calls and 7,160 trials on Gemini-2.5-Pro and Flash, judged by Gemini-2.5-Pro at temperature 0), they find ambiguous-first ordering lowers success relative to clarified-first (42.8% vs 50.0% for Pro; 45.7% vs 55.4% for Flash), coding is most vulnerable (24.4% success, 0.256 contamination), and verbal acknowledgement outpaces behavioral revision in coding (+7.1 points). Standard remedies fail or backfire: summary memory raises contamination (0.228 vs 0.177 for raw history), single-state ledgers are worse (0.295), and naive multi-hypothesis prompting is worst (1.029). Small rollback and two-phase interventions eliminate measured contamination, motivating uncertainty-preserving state management. Main limitations: no significance tests or confidence intervals, a judge model shared with an evaluated model, auto-generated tasks with no released prompts or data, and only two models from a single family.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLMs exhibit 'early posterior collapse': they commit to one interpretation of an ambiguous early turn, and later clarification refines rather than invalidates that committed state, leaving old-hypothesis contamination in final outputs.
- Information order changes outcomes despite equivalent final information: ambiguous-first ordering lowers success versus clarified-first (42.8% vs 50.0% for Gemini-2.5-Pro; 45.7% vs 55.4% for Flash, Table 1).
- Coding tasks are most vulnerable to contamination from invalidated hypotheses (24.4% success, 0.256 contamination in ambiguous-first, Table 2) because early assumptions become embedded in interfaces and control flow.
- Standard memory and reasoning strategies do not remove contamination and often worsen it: summary memory 0.228 vs raw 0.177, single-state ledger 0.295, naive multi-hypothesis 1.029 (Tables 4, 14).
- Explicit state rollback and a two-phase ask-then-execute policy eliminate measured contamination in small controlled settings (Tables 6-7), motivating uncertainty-preserving state management as a research agenda.
Strengths
Crisp conceptual contribution
Section 3 cleanly separates forgetting (evidence absent) from failed state invalidation (evidence present but not acted on), and defines old-hypothesis contamination (Eq. 7) as a behavioral, output-level diagnostic rather than a claim about internals.
Predictive task-structure gradient
The structural-embedding hypothesis is borne out by Table 2 (coding: 24.4% success, 0.256 contamination vs writing 69.3%/0.125) and Table 3's acknowledgement-revision decomposition (coding gap +7.1 points), giving the account falsifiable content beyond anecdotes.
Honest negative results
Calibrated claim scoping
Substantial evaluation scale
Weaknesses
Zero inferential statistics
Judge circularity
Unbounded, loosely defined contamination metric
Opaque task provenance
Single-family generalization and an unexplained anomaly
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a diagnostic/position paper arguing that a recognized class of multi-turn LLM failures is misattributed to memory. The authors propose 'early posterior collapse' — an ambiguous early turn collapsing into one committed task state — and operationalize it via 'old-hypothesis contamination': residual traces of a corrected interpretation in final outputs (Section 3, Eqs. 2-7, through a behaviorally implied posterior p_eff). Five claims are tested on controlled writing/planning/coding dialogues with Gemini-2.5-Pro and Flash (~13,900 API calls, 7,160 trials; 290 generated plus 50 hand-crafted tasks): (1) order effects exist with equivalent final information (Table 1); (2) coding is most vulnerable (Table 2: 24.4% success, 0.256 contamination vs writing 69.3%/0.125); (3) verbal acknowledgement outpaces behavioral revision in coding (+7.1 points, Table 3); (4) standard memory/reasoning strategies fail to remove contamination (Tables 4-5, 14); (5) explicit rollback and two-phase policies eliminate measured contamination in small settings (Tables 6-7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'early posterior collapse' in LLM dialogue?
A failure mode proposed by this paper: when a user's first request is ambiguous, the model collapses it into one fixed interpretation before the ambiguity is resolved. Later clarifications are then treated as extra context that patches the stale interpretation instead of replacing it, so the final answer still carries traces of the wrong interpretation — what the authors call old-hypothesis contamination.
Do summaries, chain-of-thought, or more hypotheses fix the problem?
Not in this study. Summary memory raised contamination (0.228 vs 0.177 for raw history), a single-state ledger scored worse (0.295), naive multi-hypothesis prompting was worst (1.029), and chain-of-thought did not improve success (44.3%). Only explicit state-rollback instructions and a two-phase ask-first-then-execute policy eliminated measured contamination, and those tests were small.
Which models were tested and how were answers judged?
Gemini-2.5-Pro and Gemini-2.5-Flash on about 7,160 controlled writing, planning, and coding dialogues (~13,900 API calls). Gemini-2.5-Pro at temperature 0 served as the automatic judge — including for Gemini-2.5-Pro's own outputs — which the review flags as a circularity risk the paper does not quantify.
Is the code or dataset available for this paper?
No release is stated in the paper text. The 290 main tasks were auto-generated, and no prompts, judge rubrics, or task files are included, so independent reproduction currently requires rebuilding the entire task distribution and metric from the paper's prose description.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is referenced but not analyzable from the extracted text (caption only).
- Tables were extracted as flattened text and per-cell sample sizes are never reported anywhere in the paper, so statistical judgments here are inferential.
- Main text plus Appendices A-C appear complete, but any supplementary files, prompts, or judge rubrics beyond the extracted text could not be checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.