Can Classical Semantic-Extractive Summarization Be Evaluated in Hindi? A Replication Study
Showket Ahmad Khan, Mudasir Mohd, Nasrullah Sheikh, Mohsin Altaf Wani, Abid Hussain Wani, Hilal Ahmad Khanday, Niyaz Ahmad Wani
cs.CL · cs.AI
Quick verdict
A rigorous replication showing Hindi benchmarks reward lead copying; well-supported, though the class-level claim rests on just two unsupervised systems.
Summary
This replication study ports the distributional-semantics extractive summarizer of Mohd, Jan and Shah (2020) to Hindi, substituting Devanagari-appropriate components (danda-aware splitting, Hindi stopwords and stemmer, Stanza POS, fastText cc.hi.300 vectors), and evaluates it on XL-Sum Hindi and FIRE ILSUM 2.0 Hindi under a Devanagari-aware ROUGE implementation cross-validated against the XL-Sum authors' own scorer (0.0000 mean absolute difference on ROUGE-1/2/L; SU4 not externally validated), with all comparisons as 1000-resample paired bootstraps. At a three-sentence budget, the equal-weight seven-feature system trails Lead-3 significantly on both corpora (−0.042 ROUGE-1 F on XL-Sum, −0.265 on ILSUM). Ablations show position is the only contributing feature: position-only exactly reproduces Lead-3, minus-position is the weakest configuration, and a validation-tuned position+TF-IDF weighting at best ties Lead-3 (XL-Sum: statistical tie; ILSUM: −0.005, detectable but negligible). TextRank fails identically, supporting a class-level reading. Position-decile analysis and a case study show non-position features reward long, proper-noun-dense sentences while references reuse the article lead; ILSUM Lead-3 reaching 0.451 ROUGE-2 F implies near-verbatim lead references. The authors conclude current Hindi benchmarks cannot reward non-lead content selection and motivate purpose-built resources. Limitations: 200-document test slices, a single three-sentence budget, ROUGE-only scoring, and only two unsupervised methods supporting the class-level claim.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The replicated equal-weight distributional-semantics extractive system is significantly worse than Lead-3 on both XL-Sum Hindi (−0.042 ROUGE-1 F) and ILSUM 2.0 Hindi (−0.265) under instrument-validated ROUGE.
- Sentence position is the only contributing feature: position-only exactly reproduces Lead-3, minus-position is the weakest configuration, and validation-tuned weighting can at best equal — never exceed — Lead-3.
- TextRank fails identically to the replicated system, so the failure is class-level for salience-driven unsupervised extraction rather than an implementation defect.
- Current Hindi summarization benchmarks (XL-Sum Hindi, ILSUM 2.0 Hindi) have reference styles that reuse the article lead, so they cannot reward correct non-lead content selection and cannot discriminate a good content selector from a positional heuristic.
Strengths
Metric instrument validated before use
Section 1: hinexsum/rouge.py was cross-checked against the XL-Sum authors' csebuetnlp multilingual scorer on 20 documents × 3 systems, mean per-document absolute difference 0.0000 for ROUGE-1/2/L; a constructed danda-adjacent case (Δ = 0.167) proves the external scorer path is genuinely exercised rather than aliased, and the SU4-only gap is disclosed rather than hidden.
Disciplined evaluation protocol
Section 1: validation/test separation on both corpora (200-doc validation slices; XL-Sum test docs 201–400; ILSUM 200 test-split docs), identical danda-aware segmentation for all systems, and 1000-resample paired bootstraps drawn from a single shared resample matrix (Tables 2, 4), making gap intervals internally valid.
Class-level claim earned via an independent second method
Mechanistic evidence beyond scores
Calibrated tuned-ceiling analysis
Weaknesses
Small evaluation slices with borderline inference
Replication fidelity unverifiable across languages
Class-level claim from only two methods
ROUGE-only automatic evaluation
Single operating point
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper replicates the distributional-semantics extractive summarizer of Mohd, Jan and Shah (2020), ports it to Hindi by substituting a Devanagari-appropriate component at each language-specific step (NFC normalization with danda-aware sentence splitting, a Hindi stop-word list and light suffix stemmer, Stanza POS tagging, 300-d fastText Common Crawl vectors), and asks whether the result can be meaningfully evaluated on existing Hindi resources. Two empirical claims carry the paper. First, in its published equal-weight seven-feature configuration the replicated system is significantly worse than a three-sentence lead baseline on both corpora: −0.042 ROUGE-1 F on XL-Sum Hindi and −0.265 on FIRE ILSUM 2.0 Hindi (Tables 2 and 4). Second, the failure is class-level rather than an implementation artifact, because TextRank — an independently implemented method — fails identically, and because benchmark references reuse the article lead, so non-lead selection cannot be rewarded. The title question is answered negatively for current benchmarks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can classical extractive summarization be evaluated in Hindi?
This paper argues current Hindi benchmarks (XL-Sum Hindi, ILSUM 2.0 Hindi) cannot meaningfully evaluate non-lead content selection: their reference summaries reuse the article's opening, so ROUGE rewards lead copying and any salience-driven extractive method is penalized regardless of whether its selection is actually informative.
What did the replication find about the Mohd, Jan and Shah (2020) method in Hindi?
In its published equal-weight configuration the method is significantly worse than simply taking the first three sentences (Lead-3): −0.042 ROUGE-1 F on XL-Sum Hindi and −0.265 on ILSUM 2.0 Hindi. Scoring sentences by position alone exactly matches Lead-3, and validation-tuned feature weights at best tie the lead baseline.
Why does a lead baseline beat sophisticated summarization systems on Hindi news corpora?
Because the reference summaries are derived from the article opening — ILSUM Lead-3 reaches 0.451 ROUGE-2 F, implying near-verbatim reference reuse of the lead — so maximal ROUGE overlap is achieved by reproducing the opening sentences rather than by selecting informative body content, which the benchmark cannot credit.
Is the code for this Hindi summarization study available?
Yes. The HinExSum package (MIT licence, github.com/mudasirmohd/HinExSum) contains the Hindi summarizer, the Devanagari-aware ROUGE module and the full evaluation harness, plus a Code Ocean capsule (doi:10.24433/CO.6462433.v1); the corpora come from public Hugging Face distributions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review based solely on the extracted paper text; the released HinExSum repository and Code Ocean capsule were not inspected, so code-availability claims are taken as stated.
- No figures exist in the extracted text; all quantitative claims were verified against inline Tables 1–7.
- Text appears complete (abstract through references, with page-1 footnotes), but layout artifacts mean affiliation/ORCID details may be imperfectly extracted; core results are unaffected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.