What if LLMs Ate Their Words: Causal History Effects in Multi-Turn Interaction
Jinnan Li, Zheren Fu, Yue Wang, Jinzhe Li, Yuan Wu, Yi Chang
cs.CL · cs.AI
Quick verdict
Claims assistant history causally affects multi-turn LLM outcomes; rigorous replay design with length control, but effects are conditional and heterogeneous.
Summary
The paper asks whether an LLM's own prior responses actively shape later performance in multi-turn interaction. Building on the sharded simulation framework (627 instructions, six task families from BFCL-V3, HumanEval, ToTTo, Spider, GSM8K, and Haystack; five models including GPT-5.6 Luna, Grok 4.5, GPT-4o, DeepSeek V3.2, Qwen3-14B), it first maps FULL-to-SHARDED degradation (gaps .137-.224) and shows one-shot rankings do not predict robustness. It then retrospectively replays recorded user turns while replacing assistant history with a neutral placeholder: neutralization yields +.027 aggregate effect across 2,973 trajectories, and on a prespecified length-controlled subset (n=264) short and length-matched neutralization are nearly identical (+.069 vs +.068), ruling out simple context shortening. Turn Surgery neutralizes one intermediate assistant turn at a time; on 237 selected degraded trajectories, 63.7% contain at least one beneficial intervention, 57.6% of tested positions leave outcomes unchanged, and 48.4% of binary trajectories admit a fail-to-success reversal. A Qwen3-14B case study links behaviorally consequential surgeries to larger downstream residual-state divergence, but restoration and other probes are task-dependent with some confidence intervals crossing zero. Main limitations: effects are conditional on replayed user messages (no user adaptation), the neutral filler is artificial, surgery covers a selected degraded subset, and internal analysis covers one model and few tasks.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- FULL-to-SHARDED degradation (.137-.224 across five models) is structured by task and model, and stronger one-shot performance does not predict interaction robustness.
- Neutralizing assistant history changes downstream outcomes (+.027, 95% CI [.013,.041], n=2,973), and the effect persists when history length is matched (+.068 vs +.069 on the prespecified n=264 subset), ruling out simple context shortening.
- History sensitivity is turn-selective: 63.7% of 237 selected degraded trajectories contain at least one beneficial single-turn intervention while 57.6% of tested positions leave outcomes unchanged.
- In Qwen3-14B, behaviorally consequential interventions show larger downstream residual-state divergence (task-macro AUC contrast +.00547 [.00069,.01390]), but restoration and other internal probes are task-dependent rather than universal.
Strengths
Clean prespecified length-matched control
Section 3.4-4.2.2: 300 candidates selected before outcomes observed; all 1,250 positive-length replacements matched exactly under a shared whitespace-span estimator; short vs matched effects nearly identical (+.069 [.024,.115] vs +.068 [.027,.110]; difference -.001 [-.039,.035] on n=264), genuinely excluding context-shortening as the explanation.
Turn-level counterfactual design with dual aggregation
Section 3.5: Turn Surgery reports both trajectory-equal (+.148 [.116,.180]) and branch-weighted (+.110 [.077,.151]) estimands, quantifies selectivity (31.5% positive / 57.6% no-change / 10.8% negative), and shows positional structure is not recency (Spearman rho=-.139 relative position; -.005 distance-to-final).
Unusually honest reporting of null and mixed results
Statistical and procedural care
Held-out internal analysis with replication cohort
Weaknesses
Effects are replay-conditional, not interactive
Single artificial intervention type
Selected-set prevalence statistics
Temperature-0 instability implies noisy single-run branch estimates
Internal analysis is narrow and partly null
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: What if LLMs Ate Their Words
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'sharded' multi-turn evaluation?
A framework from Laban et al. that compares FULL - the complete task instruction given at once - with SHARDED, where the same information is revealed progressively across user-assistant turns. This paper applies it to 627 instructions drawn from six benchmarks (BFCL-V3, HumanEval, ToTTo, Spider, GSM8K, Summary of a Haystack) and five LLMs, measuring how much performance drops in the multi-turn setting.
What does 'neutralization' mean in this paper?
Retrospectively replacing every non-final assistant response in a completed multi-turn conversation with a fixed neutral placeholder ('Acknowledged'), then regenerating and re-scoring the final answer. The score change estimates how much the model's own earlier responses affect its later performance. A length-matched variant repeats the filler to match original history length, showing the effect is not just context shortening.
Do the results mean shorter context is better for LLMs?
No. On the prespecified length-controlled subset (n=264), length-matched neutralization (+.068) matched short neutralization (+.069), so the benefit is not explained by shorter context. Effects are also bidirectional: neutralization rescued 50.9% of baseline binary failures but harmed 14.7% of baseline successes, and one model (DeepSeek) showed a negative overall effect (-.149).
Is code and data available for this paper?
Yes. The paper states that code and data are available at github.com/jinnanli/llms-ate-their-words, and all six underlying benchmarks (BFCL-V3, HumanEval, ToTTo, Spider, GSM8K, Summary of a Haystack) are public, making the pipeline re-runnable, though proprietary models cannot be re-run exactly.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal detail is recoverable only from captions and partially extracted axis labels (e.g., Figure 5C positional histograms and Figure 6 panels); per-panel values beyond those quoted in body text or captions could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.