Extracting Arguments, Not Just Classifying Them: Instruction-Tuned LLMs for Generative Component Detection
Sofiane Elguendouze, Erwan Hain, Elena Cabrio, Serena Villata
cs.AI · cs.CL
Quick verdict
将 ACD 重构为指令微调文本生成；PE 增益强劲（0.8931 Macro-F1），但 ED/WD 缺乏外部基线，且严格评分可能导致结果虚高。
Summary
本文将论证组件检测（ACD）——即联合划定和标注声明（claims）与前提（premises）——重新框定为一项文本生成任务。研究没有采用序列标注或对预分割单元进行分类的方法，而是对仅解码器（decoder-only）的指令微调大语言模型（GPT-2-XL 1.5B, OPT 6.7B, Llama-3-8B）进行微调，使其在复现纯文本的同时插入 XML 样式的声明/前提标签，并在三个公开基准数据集（Persuasive Essays, USElecDeb60To16, Web Discourse）上以 1024-token 分块进行训练。在 Persuasive Essays 上，Llama-3-8B 在五个随机种子下达到 0.8931±0.0059 的 Macro-F1，高于 CRF 基线（0.8670），名义上超过了人类一致性上限（0.8860），但作者本人提醒这并不代表超越了人类。相同模型的零样本/单样本提示（zero/one-shot prompting）的严格 Macro-F1 均低于 0.23，并且在辩论记录、嘈杂的网络话语和合并的跨领域设置中，生成模型的性能下降幅度小于 RoBERTa/DeBERTa-v3 编码器。一项三协议评估（严格、序列比对、保真度感知）显示，文本改动导致 Llama-3 损失了约 0.05 F1（0.8451 和 0.8471）。主要局限性：ED、WD 和 Merged 数据集缺乏外部基线，因此最先进（state-of-the-art）的声明仅依赖于 PE；严格协议在评分时排除了未对齐的 token；没有报告显著性检验；并且非 PE 的结果似乎是单次运行得出的。
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reframing ACD as instruction-tuned generation over raw text (ITFACD) jointly performs boundary delimitation and claim/premise classification and outperforms existing approaches, reaching 0.8931±0.0059 strict Macro-F1 on Persuasive Essays with Llama-3-8B.
- Zero- and one-shot prompting of the same LLMs is insufficient for full ACD, with strict Macro-F1 generally below 0.23, demonstrating that instruction tuning — not generative capability alone — drives the gains.
- The generative formulation transfers better across heterogeneous domains (ED, WD, Merge) than encoder-based sequence-labeling baselines, which collapse from ~0.69–0.71 on PE to 0.32–0.49 elsewhere.
- This work is among the first to model ACD fully as a generative task, distinct from prior text-to-text AM that jointly optimized relations (Kawarada et al. 2024).
Strengths
Genuine task reframing with a clean formulation
§3.1–3.3 recast ACD as text-to-structure generation: BIO corpora become XML-tagged text, 1024-token chunks never split components, and one causal-LM objective learns boundaries and labels jointly, removing the pre-segmentation assumption that §2.2 shows most prior LLM work (Chen et al. 2024; Cabessa et al. 2025) silently relies on.
Honest, multi-protocol fidelity evaluation
§3.5 plus Tables 4–5 quantify text alteration per model (OPT 99.11%, GPT-2 99.77%, Llama-3 92.10% exact preservation) and report three scoring protocols; the strict→fidelity-aware drop for Llama-3 (0.8931→0.8471) is disclosed rather than hidden, with alteration taxonomies built from fuzzy matching, WordNet and spaCy.
Reproducibility-oriented experimental design
Careful defusing of the human-upper-bound comparison
Qualitative analysis with concrete failure/success taxonomies
Weaknesses
No external baselines beyond PE
Headline metric excludes hard tokens from scoring
No significance testing and unstable model ranking
Encoder baselines are structurally handicapped
Flagship model has unresolved generation fragility
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets Argumentative Component Detection (ACD): jointly delimiting argumentative spans and labeling them as claims or premises. The core claim is methodological — ACD is better modeled as a single text-to-structure generation problem than as sequence labeling or classification over pre-segmented units. ITFACD instruction-tunes decoder-only LLMs (GPT-2-XL 1.5B, OPT 6.7B, Llama-3-8B-Instruct) to reproduce plain text verbatim while inserting `<claim>`/`<premise>` tags. Secondary claims: (i) results surpass state-of-the-art (§1, §5.1); (ii) prompting alone is insufficient (§5.2); (iii) this is among the first fully generative ACD formulations (§2.2, §6). Scope is deliberately narrow — relation prediction, stance, and graph construction are excluded (§3.1), a defensible choice the authors justify via negative-transfer concerns.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ITFACD?
ITFACD (Instruction-tuning for Full ACD) is a method that fine-tunes instruction-tuned LLMs (GPT-2-XL 1.5B, OPT 6.7B, Llama-3-8B) to detect argumentative components by regenerating the input text while inserting XML-style <claim> and <premise> tags — so span boundary detection and label classification happen jointly in a single generation step, without needing pre-segmented argument units.
Does the model beat human annotators on argument component detection?
No. On the Persuasive Essays benchmark the best model (Llama-3-8B) scores 0.8931 Macro-F1 under the paper's strict protocol, nominally above the 0.8860 inter-annotator agreement upper bound from Stab & Gurevych (2017). The authors explicitly warn this is not human superiority — it reflects agreement with one fixed gold standard. Under their stricter fidelity-aware protocol the score drops to 0.8471, below the bound.
Which model performs best in the paper?
It depends on the dataset: Llama-3-8B is best on Persuasive Essays (0.8931) and Web Discourse (0.5615), while OPT-6.7B is best on the election-debate corpus ED (0.7600) and on the merged cross-domain setting (0.7822). GPT-2-XL 1.5B is competitive on PE (0.8419) despite being five times smaller than Llama-3.
Is the code and data for this paper available?
The paper states that the code and the datasets used are openly available in a GitHub repository. All three benchmark corpora (Persuasive Essays, USElecDeb60To16, Web Discourse) are public datasets from prior work. Note: the repository URL is not visible in the extracted text, so availability could not be independently verified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- XML tag names inside Figure 1's prompt were stripped by text extraction (e.g., the instruction reads 'Use ... for central assertions'), so exact prompt wording is partially unverifiable.
- The GitHub repository URL mentioned in the abstract is absent from the extracted text; code availability is taken from the paper's statement only.
- The paper is a COLM 2026 camera-ready per its header (arXiv 2609.24855); final venue version might differ from this extraction.
- ED/WD/Merge seed counts are inferred to be single runs from the absence of ±std in Table 2; the text does not state this explicitly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.