Record Grouping Controls Evidence Weight in Language Models
Zhongxuan Liu, Sicheng Zhou, Hongzhi Wang
cs.CL
Quick verdict
Rigorous 104k-trial study shows record partitioning causally shifts LM evidence weighting; strong controls, but checkpoint sign reversals limit guidance.
Summary
The paper argues that how retrieved records are partitioned into groups before generation is a controllable lever on evidence weight in language models. Formally, it defines claim-relative evidence roots and a canonical group–content quotient that removes within-group copies while preserving complementary content, proves equal group counts can encode different evidence states (Proposition 2), and derives a worst-case-sharp content-aware partition-error bound (Proposition 12). Empirically, across 104,402 candidate-choice likelihood trials on six public checkpoints (Qwen3-8B/4B/14B-AWQ, Phi-4-mini, Phi-4, Mistral-7B), using PERSPECTRUM and ConflictingQA plus a 48-item synthetic GEO panel, content-fixed false splits raise attack-side probability by 10.27–32.66 points and false merges remove 9.13–31.79 points; a token-matched six-slot control stays positive in 16/16 cells. A controlled campaign panel shows checkpoint-dependent sign reversals (e.g., Mistral-7B at −4.44 for same-root splits) and large order interactions (+41.43 vs −49.92 for Qwen3-8B). Limitations: the formal guarantees bind the external representation, not the model; all models are ≤14B; balanced effects average two record orders differing by over 90 points; semantic grouping is validated only on template-like synthetic data.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The supplied partition of retrieved records is a controllable pre-generation representation variable that causally reallocates model-facing evidence weight.
- Content-fixed false splits add 10.27–32.66 percentage points and false merges remove 9.13–31.79 percentage points of attack-side influence across four checkpoints and two natural-text domains (Table 1).
- Equal group counts can encode different evidence states, and the content-aware discrepancy ∆q yields a worst-case-sharp bound on partition-error distortion (Propositions 2 and 12).
- Behavioral responses to record boundaries are checkpoint-dependent, with sign reversals (Mistral-7B, Qwen3-8B) and order interactions exceeding 90 points in the controlled GEO panel.
Strengths
Content-fixed causal contrasts with serialization confound eliminated
Table 1's split/merge contrasts hold the ordered 160 attack words fixed and vary only record placement; the Phase 11 six-slot control (Table 3) additionally equalizes object count, headers, R1–R6 identifiers, separators, and final character/byte/tokenizer length via parse-inert whitespace, isolating content placement rather than token counts.
Theory executed with worst-case sharpness proofs
Propositions 11–12 come with explicit witnesses (e.g., the 3B = B·H two-record example), and the ∆q discrepancy is shown to strictly refine the count-based bound H via a concrete example where H = 4 but ∆q = 0; Proposition 2 shows equal-count partitions {1,2},{3,4} vs {1,3},{2,4} are separable by bounded aggregators.
Honest reporting of reversals and interactions
Multiple-comparison and preregistration discipline
Verifiable stored-output artifact and internal consistency
Weaknesses
Theory–behavior gap: the guarantees do not bind the model
Prescriptive conclusion outruns checkpoint-dependent evidence
Balanced mirror estimand averages opposite-signed, order-dominated effects
Limited external validity in scale and grouping feasibility
Underpowered cells and selection in the natural-text panels
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets evidence weighting in retrieval-augmented language models: when retrieved records are partitioned into groups before generation, the partition itself — beyond record count or content — determines how much influence that evidence carries. Three claims: (1) a formal representation (the canonical evidence quotient, Eq. 2) that removes within-group copies while preserving complementary content, with equal group counts able to encode distinct evidence states (Proposition 2); (2) a content-aware partition-error bound that is worst-case sharp (∆q ≤ H, Propositions 11–12, each with explicit witnesses); (3) causal evidence from 104,402 candidate-choice trials over six public checkpoints that content-fixed false splits add 10.27–32.66 points and false merges remove 9.13–31.79 points of attack-side influence, with checkpoint-dependent sign reversals in a controlled GEO panel.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'record grouping controls evidence weight' mean in practice?
The paper shows that how retrieved text records are divided into groups (and serialized) before being fed to a language model changes how much the model relies on that evidence. Splitting one source into multiple records inflates its influence (false splits add 10.27–32.66 percentage points), while merging distinct sources into one record suppresses it (9.13–31.79 points), even when the words shown are identical.
Which language models were tested?
Six public open-weight checkpoints: Qwen3-8B, Qwen3-4B, Qwen3-14B-AWQ, Phi-4-mini, Phi-4 (14B, NF4 quantized), and Mistral-7B. All are at most 14B parameters; no proprietary frontier models were evaluated.
Does grouping records always improve answers?
No. Effects reverse for some checkpoints: in the controlled 48-item campaign panel, Mistral-7B showed −4.44 points for a same-root split and Qwen3-8B showed −4.25 points for an independent-root merge loss. Record order also caused swings exceeding 90 points for Qwen3-8B, so the practical benefit is checkpoint- and presentation-dependent.
Is the code or data available?
The paper describes a verification artifact (~65 MB, 285 files) that recomputes all stored experimental outputs with a maximum numerical difference of 5.33×10⁻¹⁵, and states that fresh execution uses public model checkpoints plus the public PERSPECTRUM and ConflictingQA datasets. A public artifact URL is not stated in the text.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures appear only as captions in the extraction (e.g., Figure 5's dose-response curves); plotted values inside figures could not be independently checked, though every number cited in this review comes from tables or body text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.