Count Evidence, Not Sentences: Tempered Evidence Fusion of LLM Judgments for Long-Text Value Measurement
Yuhe Wu, Rui Qian, Guangyu Wang, Yuran Chen, Yuanchao Zhu, Junjie Yang, Zhengheng Li, Jiulin Cai, et al.
cs.CL · cs.AI
Quick verdict
TEF entropy-weights sentence log-odds, beating naive voting by ~4.5 acc points; effective and simple, but weak baselines and synthetic data temper confidence.
Summary
The paper recasts long-text value-orientation measurement with LLMs as a decision-fusion problem. Instead of asking for a document-level label or aggregating sentence votes equally, Tempered Evidence Fusion (TEF) converts each sentence's answer-token posterior into one-vs-rest log-odds and multiplies it by the normalized information gain (1 − entropy/log K), so uncertain sentences contribute almost nothing while confident ones keep near-Bayesian weight. The authors introduce MIND, a benchmark of 8,358 Chinese and English posts (5,474/2,884) spanning January 2020–January 2025 over six value dimensions, annotated in three rounds with LLM-agent balancing and semantic editing. Across five LLMs (Qwen2.5-7B, LLaMA3-8B, Qwen3-14B, DeepSeek-V3.2, GPT-4o-mini) and two languages, TEF beats the strongest of Direct, Majority Vote, and Soft Vote by an average of 4.5 accuracy and 4.6 macro-F1 points, is best or tied in 116 of 120 comparisons, shows lower expected calibration error on Qwen2.5-7B, and survives three prompt variants. Main limitations: the benchmark's undisclosed proportion of synthetic/agent-edited samples (labels fixed at construction, possible circularity with evaluation LLMs); baselines limited to naive voting despite citing stronger fusion rules (Chair–Varshney, log-linear pools); an unaddressed conditional-independence assumption for correlated sentences; and no significance tests or confidence intervals. Ablations show both components are needed, but on DeepSeek-V3.2 each ablated variant falls below Soft Vote.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- TEF, a training-free rule that weights each sentence's one-vs-rest log-odds by normalized information gain (1 − H/log K), suppresses uncertain sentences while preserving near-Bayes-optimal weight for decisive ones.
- MIND is a new five-year, cross-lingual benchmark of 8,358 Chinese and English posts (5,474/2,884) over six value dimensions, built with three-round human annotation plus agent-driven balancing and editing.
- TEF outperforms the strongest of Direct, Majority Vote, and Soft Vote by an average of 4.5 accuracy and 4.6 macro-F1 points across five LLMs and two languages, winning or tying 116 of 120 comparisons.
- TEF yields better document-level calibration (lowest expected calibration error on Qwen2.5-7B), remains best under three prompt variants, and costs the same tokens as voting.
Strengths
Simple, training-free, principled fusion rule
TEF (§3, Algorithm 1) is ~15 lines: entropy-weighted one-vs-rest log-odds with only two clip constants (ε=1e-6, M=10); grounded in the additive Bayes log-odds identity (Prop. 2) with honest acknowledgment that LLM posteriors are only approximately calibrated.
Broad, internally consistent empirical grid
Table 3 covers 5 models × 2 languages × 6 dimensions × 2 metrics; TEF is best-or-tied in 116/120 comparisons and never trails the strongest baseline by more than 2.6 points, with gains largest where baselines are weakest (Qwen2.5-7B Chinese culture 44.6→72.1; LLaMA3-8B English politics 49.9→67.5).
Robustness and cost checks included
New longitudinal cross-lingual benchmark
Honest ablation reporting
Weaknesses
Benchmark validity: undisclosed synthetic content
Baselines omit the paper's own cited competitors
Theory-practice gap on dependence and tempering
No statistical uncertainty anywhere
Missing experimental details
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This ICASSP-style paper (arXiv:2609.27165) addresses long-text value-orientation measurement: inferring a binary orientation across six value dimensions (culture, economy, society, politics, technology, environment) from long social media posts in which only a minority of sentences carry stance. It claims three contributions: (1) TEF, a training-free fusion rule that multiplies each sentence's one-vs-rest log-odds by the normalized information gain w_i = 1 − H(p_i)/log K, softly censoring uncertain sentences while decisive ones keep near-Bayes-optimal weight; (2) MIND, a new cross-lingual benchmark of 8,358 posts (5,474 Chinese, 2,884 English; January 2020 – January 2025); (3) TEF beats the strongest of Direct/Majority Vote/Soft Vote by +4.5 accuracy and +4.6 macro-F1 points on average across five LLMs and two languages.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Tempered Evidence Fusion (TEF)?
TEF is a training-free rule for combining sentence-level LLM judgments over a long document. Each sentence's class probabilities are converted to one-vs-rest log-odds and multiplied by a weight equal to the normalized information gain, 1 − H(p)/log K. Uncertain sentences therefore contribute almost nothing while confident ones contribute strongly, and the class with the highest summed score becomes the document-level label. It requires no training and only two clipping constants (ε=1e-6, M=10).
What is the MIND benchmark and what does it contain?
MIND (Multi-event Insight Network Dimensions) is a benchmark of 8,358 social media posts — 5,474 Chinese and 2,884 English — collected from January 2020 to January 2025 and labeled along six value dimensions (culture, economy, society, politics, technology, environment), each with paired opposing orientations. Labels come from three-round annotation with 2/3 agreement plus arbitration, followed by an LLM-agent stage that generates synthetic samples for underrepresented classes and revises ambiguous ones; the paper does not disclose what fraction of the final set is synthetic or agent-edited.
How much better is TEF than majority or soft voting, and what does it cost?
Across five LLMs (Qwen2.5-7B, LLaMA3-8B, Qwen3-14B, DeepSeek-V3.2, GPT-4o-mini) and two languages, TEF beats the strongest of direct prediction, majority vote, and soft vote by an average of 4.5 accuracy points and 4.6 macro-F1 points, and is best or tied in 116 of 120 comparisons. It reads the same single-token log-probabilities as voting, so token cost is identical (2.7M tokens vs 0.5M for direct prediction on Qwen2.5-7B) with about 2 ms extra time to first token.
Is the MIND dataset and TEF code publicly available?
The paper states that both the MIND dataset and code are available via the authors' GitHub repository (github.com/Kzczc/ICASSP2027-TEF). Caveat: a portion of MIND was generated or edited by LLM agents during dataset construction, and the undisclosed proportion means benchmark users should treat labels on such samples with care.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 (pipeline, dataset distributions, calibration) appear only as caption-level text; ECE values and per-dimension distribution numbers cannot be verified from the extraction.
- Metadata lists 8 authors while the full text lists 14 — likely a metadata truncation artifact; the review uses the full-text list.
- The GitHub repository's actual contents (data completeness, code) were not verified; availability is as stated in the paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.