LOOMSUM:Weaving Quantitative and Narrative Evidence for Faithful Long Text-Table Summarization
Meng Zhou, Wenhao You, Wei Yuan
cs.CL
Quick verdict
LOOMSUM links table facts to narratives, training-free; TGF gains are plausible but modest, possibly metric self-favoring, and human validation is thin.
Summary
LOOMSUM is a training-free pipeline for summarizing long documents that mix narrative text and tables, targeting financial reports. It decomposes the source into atomic text and table claims, explicitly aligns table-derived quantities with narrative statements that explain or qualify them, consolidates claims, plans a discourse structure, and generates the summary with GPT-5-nano guided by retrieved in-context demonstrations. It also proposes Table-Grounded Faithfulness (TGF), a reference-free metric scoring Numeric Grounding, Analysis Support, and Relation Consistency per claim, aggregated non-compensatorily via min(), plus a G-Cov coverage companion. On FINDSum-ROO/Liquidity, LOOMSUM achieves the best TGF (71.38/79.43 vs 69.47/76.34 for RAG-Sum) and top ROUGE-1/R-2/R-L; on USTT it leads R-1, R-L, and NS but TextRank beats its TGF (66.41 vs 63.71). Human validation (100 FINDSum documents, two annotators) yields moderate correlations for RC (ρ=0.598) and NG (ρ=0.482) but a non-significant one for AS (ρ=0.326); TGF outperforms AlignScore/SummaC on relation judgments. Main weaknesses: a small two-annotator study without inter-annotator agreement, no significance tests for benchmark gains, an unnamed judge model and unspecified K/L hyperparameters, missing pipeline prompts, no code release, and possible circularity since method and metric share the same claim-linking inductive bias.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing long text-table summarizers frequently pair individually supported quantitative facts with incorrect narrative interpretations ('quantitative-narrative misalignment'), a failure generic factuality metrics do not detect.
- LOOMSUM, a training-free claim-to-discourse pipeline, improves analytical faithfulness, achieving the best TGF on both FINDSum subsets (71.38 ROO, 79.43 Liquidity) with competitive ROUGE and numerical-selection scores.
- TGF's Relation Consistency dimension agrees with human relation judgments (ρ=0.598) far better than AlignScore (ρ=0.138) or SummaC (ρ=0.194).
- Cross-modal alignment, discourse planning, and ICL each causally contribute to the gains, per the cumulative ablation (ROO TGF 55.68 to 71.38).
- Faithfulness and coverage must be read jointly: TextRank's high TGF on USTT reflects conservative extraction that G-Cov exposes as low grounded coverage.
Strengths
Well-motivated failure formalization
Section 1 and Figure 1 define quantitative-narrative misalignment with a concrete FINDSum example (a supported +$14.1M net-interest-income fact paired with the wrong explanation) and quantify it: omitted links reach 32.3% for Direct-LC and 34.4% for GCG on a FINDSum-ROO subset vs 24.5% for LOOMSUM (Figure 1b).
Non-compensatory metric design
TGF (Section 3) aggregates cross-modal claims by min(NG, AS, RC), so correct facts cannot mask a broken fact-interpretation link; the companion G-Cov (Appendix C) anticipates the conservatism objection, and jointly they explain TextRank's high-TGF/low-coverage USTT result.
Large-scale, component-isolated evaluation
Honest reporting of unfavorable results
Relation Consistency validated against humans
Weaknesses
Thin, single-benchmark human validation
Metric-method circularity risk
Missing statistical hygiene and internal inconsistencies
Reproducibility gaps
Single base model, unexamined judge bias, unreported cost
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets long text-table summarization, where quantitative results live in tables while their causes and caveats sit in distant narrative passages (FINDSum sources average 34,630-36,447 tokens; USTT averages 1,680). It formalizes *quantitative-narrative misalignment*: a generated claim whose numeric and narrative components are each source-supported but whose asserted link is not — Figure 1(a) pairs a correct +$14.1M net-interest-income fact with an unrelated explanation. Three central claims: (1) existing summarizers omit or mismatch such relations at scale (Figure 1(b): omitted links 32.3% for Direct-LC and 34.4% for GCG vs 24.5% for LOOMSUM on a FINDSum-ROO subset); (2) LOOMSUM, a training-free claim-to-discourse pipeline, reduces these failures; (3) the proposed TGF metric detects relation errors that AlignScore and SummaC miss, with human validation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does LOOMSUM solve?
When LLMs summarize documents containing both tables and narrative text, they can state a correct number and a correct explanation but wrongly link them — for example, attributing a revenue increase to the wrong cause. This is called quantitative-narrative misalignment. LOOMSUM is a training-free pipeline that extracts atomic claims, explicitly links each table fact to its correct narrative explanation, plans the summary structure, and only then generates the text.
What is the TGF metric?
Table-Grounded Faithfulness decomposes a generated summary into atomic claims and scores each on three dimensions: Numeric Grounding (is the number right?), Analysis Support (is the interpretation supported by the source?), and Relation Consistency (is the asserted link between number and interpretation supported?). A claim's score is the minimum of its required dimensions, so a broken link cannot be compensated by correct facts. It requires no reference summary.
Did LOOMSUM outperform all baselines on every benchmark?
No. On FINDSum (English financial reports) it achieved the best TGF scores (71.38 on ROO, 79.43 on Liquidity). On USTT (Chinese) it led ROUGE-1, ROUGE-L and numerical-selection metrics, but the unsupervised extractive TextRank baseline scored higher TGF (66.41 vs 63.71) because it emits few, well-supported claims while covering less reference content.
Is code or data available for LOOMSUM?
The paper does not mention a code or data release. The TGF evaluation prompts are included in Appendix E, but the prompts for the summarization pipeline stages, the identity of the judging model, and hyperparameters such as the number of demonstrations and repeated judgments are not specified in the text.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 3/4 numeric values were recovered from embedded plot-data streams; method-to-value assignment was cross-verified against text-stated scores (e.g., LOOMSUM 74/72/69 on ROO; TextRank NG 69 on USTT), but cells not stated in prose carry minor decoding uncertainty.
- Appendix B (base-model sensitivity) contains only its descriptive text; no results table appears anywhere in the provided text, and the document continues into Appendices C-E, so this is a gap in the paper itself, not end-of-text truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.