MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors
Wei He, Aline Villavicencio, Rodrigo Wilkens, Zhenyun Deng
cs.CL
Quick verdict
Proposes MWE-ECL, a bilingual access-vs-use long-context diagnostic; matched controls are rigorous, but only DeepSeek V4 Pro shows a Holm-significant gap.
Summary
The paper introduces MWE-ECL, a bilingual long-context diagnostic that asks whether distant context a model can explicitly retrieve actually changes a competing local decision. Each multiword-expression family pairs a literal and a figurative case sharing one trigger sentence and option set; a distant anchor licenses one reading while neutral filler scales distance from 0 to 128K tokens. Three matched heads separate anchor-conditioned interpretation, explicit anchor retrieval, and a no-anchor prior control, yielding override (OR), retrieval-conditioned override (OR|Ret), and preservation metrics on model-specific prior-conflict sets. Across eight English API panels (DeepSeek V4 Flash/Pro, MiniMax-M3, Qwen3.6-Plus/35B, GLM-4.7/5.1, Gemma-4-26B), retrieval on prior-conflict rows reaches 0.989–1.000 while OR spans 0.806–1.000 and preservation stays at 0.977–1.000. A same-call control reproduces the gap for DeepSeek V4 Pro (1.000 retrieval versus 0.900–0.920 interpretation) but not for GLM-5.1, bounding generality; only DeepSeek V4 Pro survives Holm correction (p = 0.002). Foil-consistent cues shift the no-anchor prior (Δ −0.378 for DeepSeek V4 Flash at 16 cues) far more than retrieval, indicating semantic interference rather than anchor loss. Limitations include 25 English and 10 Chinese families, an idiom-heavy selection, single-pass snapshots for six panels, mismatched reasoning configurations, and a code-and-data archive not included in the arXiv submission.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across eight English API panels, anchor retrieval on prior-conflict rows is near ceiling (0.989–1.000) while prior-conflict override spans 0.806–1.000, with preservation of prior-correct decisions at 0.977–1.000.
- A same-call dual-head control reproduces the retrieval–interpretation gap for DeepSeek V4 Pro (1.000 retrieval vs. 0.900–0.920 interpretation), showing separate invocations are not its sole explanation; GLM-5.1 shows no gap, bounding generality.
- Failures concentrate where local defaults conflict: correction failures (11–34 per non-ceiling panel) dwarf anchor-induced corruption (0–4), all errors in the five highest-miss families occur on literal variants, and the cold feet family retrieves the literal anchor yet answers figuratively in 19/19 events.
- Foil-consistent cues shift the no-anchor prior far more than retrieval (e.g., Δ −0.378 vs. −0.007 for DeepSeek V4 Flash at 16 cues), supporting semantic interference rather than simple anchor loss, and joint ECL is right-censored coverage (all panels pass τ = 0.8 at 128K).
Strengths
Clean separation of access from use with matched heads
The three-head design (interpretation, anchor retrieval, no-anchor prior control; §3.1, Figure 1) conditions override on each model's own prior errors (Eqs. 2–3), avoiding the trap of penalizing models for having different defaults; preservation (Eq. 6) separately captures anchor-induced corruption, and the paper shows corruption is rare (0–4 events per panel, Table S5).
Unusually rigorous statistical accounting
10,000-resample whole-family bootstrap CIs, exact family-clustered sign-flip tests with Holm correction over eight prespecified within-model tests, conflict-set overlap, and a fixed 182-item consensus-hard subset (Tables S3–S4). ECL is reported as right-censored coverage rather than a spurious failure boundary (all panels pass τ = 0.8 at 128K; DeepSeek V4 Pro drops to 512 tokens at τ = 0.9).
Artifact controls directly target the obvious objections
Credible human validation and leakage audits
Behaviorally concrete failure evidence
Weaknesses
Corrected evidence supports the dissociation strongly in only one model
Reproducibility is asserted but not demonstrated
Small, idiom-heavy catalog limits external validity
Configuration heterogeneity confounds deployment comparisons
Model-specific OR denominators impede cross-model reading
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MWE-ECL (arXiv:2609.27590) is a bilingual long-context diagnostic built around one question: does explicitly recoverable distant context actually change a decision that conflicts with a strong local lexical prior? The authors construct matched task families — e.g., *spill the beans* preceded at document start by either a literal bean-warehouse anchor or a figurative confidential-plans anchor — with neutral filler scaling nominal distance over 0/512/2K/8K/32K/64K/128K tokens. Three matched heads separate (1) anchor-conditioned interpretation, (2) explicit anchor retrieval, and (3) a no-anchor prior control (§3.1, Figure 1). The headline claims: across eight English API panels, retrieval on prior-conflict rows is near ceiling (0.989–1.000) while override spans 0.806–1.000; the gap survives a same-call control for DeepSeek V4 Pro; and failures concentrate on prior-conflict, literal-variant items. Claims are appropriately hedged as behavioral and deployment-specific.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MWE-ECL?
MWE-ECL (Multiword Expression Effective Context Length) is a bilingual English/Chinese long-context benchmark that tests whether language models actually use distant context they can retrieve. Each test case puts a distant sentence (the 'anchor') at the start of a long document; the anchor should change the correct meaning of a familiar multiword expression like 'spill the beans' appearing at the end. Three matched questions measure interpretation with the anchor, direct retrieval of the anchor, and the model's default answer with no anchor.
What is the paper's main finding?
Models often retrieve distant anchors almost perfectly (0.989–1.000 accuracy on conflict items across eight models) yet still keep their locally preferred reading; override of wrong defaults ranges from 0.806 to 1.000. The access–use gap is largest and statistically robust for DeepSeek V4 Pro (Holm-corrected p = 0.002), smaller and correction-fragile for three other models, and absent for Qwen3.6-Plus, GLM-4.7, and GLM-5.1.
Is the gap just because retrieval and interpretation are asked in separate API calls?
No. In a control where both questions appear in a single prompt, DeepSeek V4 Pro still retrieves all 50 anchors (1.000) while interpreting correctly only 0.900–0.920 of the time at 64K/128K. However, GLM-5.1 shows no gap even in the separate-call setting, so the effect is model-dependent rather than universal.
Does this mean long-context models cannot use far-away information?
No. The paper identifies a specific failure mode: distant information that is retrievable but behaviorally inert when it conflicts with a strong local habit (here, a conventional idiomatic reading). Several 2026-era models show no gap up to 128K tokens, and the benchmark reports right-censored coverage at tested lengths rather than a true context-window limit.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are available only as embedded caption and axis text in the extraction; visual elements could not be independently analyzed.
- The code-and-data archive referenced in Appendix A is stated to be excluded from the arXiv submission, so row-level claims could not be verified against raw outputs.
- Review is based on the provided text extraction, which appears complete through Appendices A–J; any truncated figures or supplementary tables were not assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.