Does Linguistic Structure Enrichment Enhance Coherence Assessment? Not With Current Architectures
Victor Mazzotti, Luiz Pereira, Marina Bitencourt dos Santos, Helena Maia, Carlos Caetano, Nádia Felix, Sandra Avila
cs.CL · cs.AI
Quick verdict
Useful negative result: RST/POS input enrichment loses to plain text in coherence classification, but the incompatibility explanation exceeds the evidence.
Summary
This paper asks whether enriching input text with linguistic annotations improves coherence classification. The authors fine-tune XLM-RoBERTa Longformer on a binarized GCDC corpus under three pipelines: plain text, text enriched with 35 RST-derived special tokens (nuclearity:relation markers around EDUs from the DMRST parser), and text enriched with POS tags from spaCy's 17 UPOS classes. Across five runs, plain text wins on balanced accuracy (72.4±0.1) over POS (71.7±0.2) and RST (71.5±0.3); per-subset results show POS beating RST on Clinton and Yahoo, with Yelp weakest throughout. Brier scores invert the ranking: POS is best calibrated (0.243) and Plain worst (0.331). To explain the failure, the authors argue Transformers are non-isomorphic (the final LayerNorm's image lies on a lower-dimensional manifold), so hierarchical RST structure is 'flattened' into noise. Zero-shot transfer to Brazilian Portuguese disinformation (FakeTrueBR) yields 73.1±0.8 for Plain, offered as evidence that coherence can proxy for misinformation. Strengths include the honest negative result, calibration analysis, per-subset breakdowns, and released code/models. Main limitations: the architectural-incompatibility explanation is loosely argued — non-surjectivity applies equally to plain text and the cited injectivity result contradicts information loss — while confounds (longer inputs, untrained new-token embeddings, asymmetric tuning) are uncontrolled, no significance tests are reported, and the handling of medium-coherence GCDC texts is unspecified.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Input-side enrichment with RST (35 special tokens) or POS (17 UPOS tags) lowers balanced accuracy versus plain text on binarized GCDC (Plain 72.4±0.1 vs POS 71.7±0.2, RST 71.5±0.3).
- The underperformance is attributed to structural/syntactic incompatibility between hierarchical discourse structure and the Transformer, argued via a non-surjectivity (non-isomorphism) result for the architecture.
- Enrichment improves probabilistic calibration despite hurting accuracy: POS attains the best Brier score (0.243 overall) and Plain the worst (0.331).
- Coherence classifiers transfer zero-shot across language and task: the Plain model reaches 73.1±0.8 balanced accuracy on FakeTrueBR disinformation, supporting coherence as an indirect disinformation signal.
Strengths
Honest, well-scoped negative result
Plain text beats both enrichment pipelines on binarized GCDC (Table 5: 72.4±0.1 vs POS 71.7±0.2, RST 71.5±0.3 balanced accuracy) and on FakeTrueBR zero-shot (Table 8: 73.1±0.8 vs 61.5/59.5); the claim is explicitly limited to input-token enrichment with current architectures.
Accuracy-calibration dissociation
Table 7 shows POS achieves the best Brier score (0.243 overall) while Plain is the worst (0.331), a non-obvious finding that enrichment improves confidence calibration even as it hurts discrimination; Fig. 6 visualizes this by text length.
Per-subset heterogeneity analysis
Reproducibility and transparency posture
Zero-shot cross-lingual demonstration
Weaknesses
Causal explanation outruns the evidence
Uncontrolled confounds in the enrichment comparison
Thin statistical reporting with an internal inconsistency
Dataset binarization underspecified
Disinformation proxy claim weakly supported
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tackles binary coherence classification (coherent vs. incoherent) and asks a narrowly framed question: does injecting linguistic annotations into the input text — Rhetorical Structure Theory (RST) trees from the DMRST parser, or Part-of-Speech (POS) tags from spaCy — improve a Transformer classifier? The title answers negatively. Two satellite claims follow: (i) the failure is due to architectural incompatibility of hierarchical RST with Transformers; (ii) coherence classification transfers zero-shot to Brazilian Portuguese disinformation (FakeTrueBR, ~3,500 texts), reaching 73.1±0.8 balanced accuracy with the plain-text pipeline. Scope is limited to input-side enrichment; architectural modification (the paper's category i) is explicitly future work, so 'Not With Current Architectures' really means 'not via input tokens.'
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
Adding linguistic annotations directly into the input text does not improve coherence classification with current Transformers. On the GCDC corpus, a plain-text XLM-RoBERTa Longformer reached 72.4% balanced accuracy, beating POS-tag enrichment (71.7%) and RST-discourse enrichment (71.5%). The authors argue that hierarchical discourse structure inserted as special tokens is effectively ignored or treated as noise by the architecture.
Can a coherence detector help identify fake news?
Possibly, as an indirect signal. A model trained only for English coherence classification distinguished real vs. fake texts in the Brazilian Portuguese FakeTrueBR corpus with 73.1% balanced accuracy without any in-language training — roughly 20 points below supervised systems (about 94.5% reported for the corpus) but well above chance, suggesting incoherent writing correlates with misinformation. The authors caution that coherent texts can still be false, and poorly structured texts can be true.
Why do the authors claim linguistic enrichment fails?
They offer a mathematical argument that Transformers are not isomorphic maps: the final LayerNorm projects outputs onto a lower-dimensional manifold, so the model cannot preserve a tree-like rhetorical structure embedded in the token sequence, which then acts as noise. This argument is theoretical; critics note it applies equally to plain text and that simpler explanations (longer inputs, newly initialized token embeddings) were not experimentally ruled out.
Is code and data available for this paper?
The paper states that code and trained models are available at github.com/ittozzamV/cohereclassifier. Both datasets (the English GCDC coherence corpus and the Brazilian FakeTrueBR disinformation corpus) are public, as are the tools used (DMRST RST parser and spaCy for POS tagging).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 are present in the extraction only as captions and axis labels; visual content (e.g., the length-probability distributions in Fig. 6) could not be assessed.
- Table 5 combines three pipelines per row; per-pipeline configuration-to-result pairing was read from column order, which is unambiguous, but the Plain column contains five configurations versus three each for RST and POS, making the search-space comparison partly inferential.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.