Controlled Attribute-Specific Summarization of Interrogative Dialogues
A Aditya Bhardwaj, Arjit Singh Arora, Md Shad Akhtar
cs.CL · cs.AI
Quick verdict
CASPER prompting + RoleEval hierarchy improves forensic dialogue summarization on ED/FD, but CD lags SumCoT and the labels ablation undercuts the core design.
Summary
The paper introduces attribute-specific summarization of interrogator-witness dialogues, decomposing testimony into event details, factual details, and character descriptions. The authors build MINDSum from the MIND corpus: 376 testimonies, 5,785 utterance pairs annotated at utterance level (Cohen's κ = 0.83) with gold attribute summaries. CASPER chains a 5W guiding prompt, entity extraction, and attribute-specific prompting with Llama 3.1 8B, then refines summaries through RoleEval, a hierarchical LLM-as-judge loop (officer → inspector → senior inspector, GPT-4o mini). On the test split (76 testimonies), CASPER (k=1) reaches an event-details RoleEval score of 86.06 and factual-details 81.62, edging ToT and GPT-4o, but drops to 66.68 on character descriptions, well behind SumCoT (80.12); Reflexion also tops factual ROUGE-1 (44.43 vs 42.87). A paired t-test (n=120) supports gains vs ToT/GPT-4o/Reflexion (p ≤ 0.022). Human evaluation (30 raters, α = 0.91) favors CASPER, and RoleEval correlates with human judgment (r = 0.87). Limitations: the w/o-labels ablation slightly beats the full method on event/factual RoleEval, weakening the core design claim; refinement and metric share an LLM (only partially mitigated in §5.8); the test set is small; the full dataset release is pending acceptance.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Decomposing interrogation summaries into event, factual, and character attributes with structured prompting yields more forensically aligned summaries than existing controlled-summarization and reflection baselines.
- MINDSum is a new expert-annotated dataset for attribute-specific forensic dialogue summarization with strong annotation agreement (Cohen's κ = 0.83).
- RoleEval, a hierarchical role-based LLM-as-judge, correlates with human judgment (Pearson r = 0.87) far better than ROUGE-L (0.42) or BERTScore (0.58).
- Single-pass refinement (k=1) is optimal and further iterations degrade summaries, while CASPER attains the lowest summary error density (Pe = 0.18 vs 0.34–0.62 for baselines).
Strengths
New task formulation and a carefully annotated resource
Attribute-specific forensic summarization (ED/FD/CD) is operationalized as MINDSum: 376 testimonies over 149 crime events, 5,785 utterance pairs with two-level annotation (utterance multilabels + gold attribute summaries), three pilot rounds, linguist review, and Cohen's κ = 0.83 (§3.1, Table 1).
Multi-layer evaluation design
ROUGE/BERTScore plus RoleEval plus a double-blind human study with 30 raters (Cronbach's α = 0.91, Table 13); metric-human correlation is reported (RoleEval Pearson r = 0.87 vs ROUGE-L 0.42, Table 15), plus severity-stratified analysis with practical role-skipping guidance (§5.4.2, Tables 6–7).
Candor about failure modes
Quantified error taxonomy
Weaknesses
Core ablation contradicts the central design
Metric circularity only partially mitigated
Abstract overclaims relative to results
Thin statistics and a small test set
Reporting inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper formalizes **attribute-specific summarization of interrogative dialogues**: each interrogator–witness testimony must yield three disjoint summaries — event details (ED), factual details (FD), and character descriptions (CD) — motivated by deposition-summary practice in legal settings (§1, Fig. 1). Three contributions are claimed: (1) **MINDSum**, an extension of the MIND corpus with 376 testimonies over 149 crime events and 5,785 utterance pairs, annotated at two levels (utterance-level ED/FD/CD/filler multilabels, Cohen's κ = 0.83, plus gold attribute summaries, §3); (2) **CASPER**, a prompting pipeline over Llama 3.1 8B (5W guiding prompt → entity extraction → attribute-specific Q&A → ≤150-word summary, §4); and (3) **RoleEval**, a hierarchical LLM-as-judge in which GPT-4o-mini plays officer → inspector → senior inspector, scoring five criteria (coverage, consistency, clarity, attribute relevance, factual completeness) on 1–5 Likert scales. The abstract claims significant gains over existing baselines on ROUGE and BERTScore plus human-confirmed alignment with expert reasoning.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CASPER?
CASPER (Chain-of-Thought Attribute-Specific Prompting for Evaluative Summarization) is an LLM pipeline that summarizes witness interrogations into three separate summaries — event details, factual details, and character descriptions. It uses Llama 3.1 8B for generation and a three-role police-hierarchy review (officer, inspector, senior inspector, played by GPT-4o mini) to critique and refine each summary.
What is the MINDSum dataset?
MINDSum extends the MIND eyewitness-deception corpus with 376 interrogator-witness testimonies (5,785 utterance pairs) across 149 crime events. Every utterance pair is tagged as event detail, factual detail, character description, or filler (inter-annotator agreement κ = 0.83), and each testimony gets three gold summaries. Code and a subset are on GitHub (flamenlp/CASPER); the full dataset is promised on acceptance.
How well does CASPER perform compared to baselines?
It scores best on event details (RoleEval 86.06) and factual details (81.62), narrowly beating ToT and GPT-4o with paired t-test significance (n=120, p ≤ 0.022). However, on character descriptions it scores 66.68, well behind SumCoT (80.12), and Reflexion edges it on factual ROUGE-1 (44.43 vs 42.87). Human raters (30 people, α = 0.91) prefer it most on coverage and attribute relevance.
Is RoleEval a reliable automatic evaluation metric?
RoleEval correlates with human scores far better than standard metrics (Pearson r = 0.87 vs 0.42 for ROUGE-L and 0.58 for BERTScore, per the paper's Table 15). Caveat: it is also the mechanism used to refine CASPER's summaries, and the authors show LLMs favor their own outputs, so same-model refinement and scoring inflates results; using different models narrows the gap.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 (overview, annotation schema, architecture) are image-only; their content was inferred from captions and body text.
- Tables were extracted linearly from the PDF; row/column alignment was re-verified manually, so minor misreads (e.g., of dagger marks denoting best scores) are possible.
- The 'Cambridge Default Journal (2026)' header is a template placeholder; the actual peer-review venue and status of this arXiv preprint (2609.28004) are unknown.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.