What Limits Us? Analyzing Self-Reported Limitations in NLP Research
Tawan Thaepprasit, Peeranuth Kehasukcharoen, Ding Wang, Remi Denton, Peerapon Vateekul, Piyawat Lertvittayakumjorn
cs.CL
Quick verdict
Human-AI coding of 16,067 ACL/EMNLP Limitations sections (2020–25) finds scope-limitations surging; rigorous and honest, but validation rests on 150 papers.
Summary
The paper presents a large-scale analysis of self-reported Limitations sections in NLP, covering 16,067 papers from ACL and EMNLP (2020–2025) across Long, Short, and Findings tracks, with the annotated dataset released on GitHub. The authors built a human-AI framework for iterative hybrid qualitative coding: Gemini 2.5 Flash segments each section into semantic units, Gemini 3.1 Pro assigns codes from an evolving codebook and proposes new codes, and LLM-based consolidation and recommendation modules triage proposals before humans confirm codebook updates (ADD/MERGE/EXPAND/REJECT). Starting from 25 codes, the codebook grew to 40 limitation and 10 non-limitation codes. On a 150-paper human-annotated evaluation set, the best prompt configuration achieved F1 0.801 and Krippendorff's α 0.647 versus a human-only baseline of 0.644. Key findings: Scope Limitation rose from about 30% of papers in 2020 to over 65% in 2025 while Methodological Constraints declined; median section length stabilized at 138–150 words while unique codes per paper grew from 1.63 to 3.02; research-area effects are sparse (32 of 1,160 cells significant after Bonferroni; Cramér's V = 0.071); papers without large-company affiliations report Scope Limitation more often (63.6% vs 56.6%, adjusted p = 0.019); and 1.4% of 2025 papers contain only non-limitation content in their Limitations sections. Main limitations: annotation quality was validated only against the initial codebook on 150 papers, codebook drift across iterations may confound temporal comparisons, and the respective contributions of the mandatory policy and the LLM paradigm to the trends remain unresolved.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Self-reported Scope Limitations rose from about 30% of ACL/EMNLP papers in 2020 to over 65% in 2025, overtaking Methodological Constraints around the EMNLP 2022 mandate.
- An iterative human-AI hybrid coding framework can annotate 16,067 Limitations sections with LLM agreement (F1 0.801, Krippendorff's α 0.647) comparable to the human inter-annotator baseline (0.644).
- Limitation reporting is largely homogeneous across research areas: only 32 of 1,160 (area, code) cells deviate significantly after Bonferroni correction, with Cramér's V = 0.071.
- A Soft Landing discursive pattern is statistically real: Future Work (40.2%) and Contextual Justification (19.7%) follow limitation codes significantly above the permutation null of 32.8% and 17.0%.
- A small fraction of papers (90 papers, 1.4% in 2025) contain only non-limitation content in their Limitations sections, implying limitations are reported solely implicitly via future work.
Strengths
Unmatched corpus scale with public release
16,067 Limitations sections from ACL/EMNLP 2020–2025 released with LLM annotations and a human-annotated evaluation subset (§1 contribution 2; footnote 3, github.com/Sundione/nlp-self-reported-limitations) — an order of magnitude beyond prior manual analyses such as Zhou et al. 2025 (57 papers).
Measured, ablated LLM validity
Table 2 ablates prompt components (definitions only F1 0.738; few-shot only 0.776; both 0.801) against a reported human inter-annotator baseline of Krippendorff's α 0.644, and Appendix A (Table 7) gives per-code α with explicit low-support warnings.
Statistical discipline throughout
Consistent honesty about confounds and weak results
Full prompt transparency
Weaknesses
Validation gap: final codebook never human-audited
Codebook drift confounds the temporal analysis (RQ1)
Over-granular codebook tail
Single-checkpoint extraction validation
RQ2 coverage is partial
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper analyzes self-reported Limitations sections from 16,067 ACL and EMNLP papers (Long, Short, Findings; 2020–2025), asking how disclosure content evolved (RQ1, §4.2), how it correlates with research area, format, and affiliation (RQ2, §4.3), and which discursive patterns recur (RQ3, §4.4). The methodological claim is that an iterative human-AI hybrid coding framework scales rigorous qualitative analysis beyond manual coding (Zhou et al. 2025, 57 papers) and topic modeling (Al Azher et al. 2025b), which under-detects minor topics. Claims are generally well scoped: analysis is restricted to dedicated Limitations sections (§3.1), and the authors decline to attribute the 2022 crossover to the mandatory policy without a controlled analysis (§4.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did this study find about how NLP researchers report limitations?
Across 16,067 ACL and EMNLP papers from 2020–2025, deliberately scoped disclosures (Scope Limitation) grew from about 30% of papers in 2020 to over 65% in 2025, while Methodological Constraints declined. Limitations sections became shorter (median stabilizing at 138–150 words) but more diverse, averaging 3.02 unique limitation types per paper by EMNLP 2025 versus 1.63 at ACL 2020.
Is the Limitations section mandatory at ACL and EMNLP?
It became mandatory at EMNLP 2022 and ACL 2023, and the ACL Rolling Review adopted the requirement in December 2023. This paper measures disclosure compliance reaching 100% by EMNLP 2025.
Did the study use AI (LLMs) to analyze the Limitations sections?
Yes. Gemini 2.5 Flash split each section into semantic units and Gemini 3.1 Pro assigned codes under a codebook that humans iteratively updated. On 150 human-annotated papers the LLM reached F1 0.801 and Krippendorff's α 0.647, close to the human-only inter-annotator baseline of 0.644.
Where can I get the dataset of annotated Limitations sections?
The authors released the 16,067 annotated Limitations sections plus a human-annotated evaluation subset on GitHub (github.com/Sundione/nlp-self-reported-limitations); all pipeline prompts are published in Appendix K of the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Section 4.1, the latter half of Section 4.4, and the Conclusion/Discussion were omitted from the extracted text, so closing author claims were not directly reviewed.
- Figures 1, 3, 4, 6, and 7 appear only as axis labels and captions; trend values were taken from in-text numbers.
- Portions of Appendices D–J (full format/affiliation tables, theme organization) are truncated in the extraction.
- The arXiv metadata spelling (Thaepprasit) differs from the byline in the PDF text (Thaeprasit); author identity assumed identical.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.