The Impact of Synthetic Data Augmentation on Discourse-Pragmatic Function Classification
Sara Sorahi, Kevin Tang, Reza Kazemian
cs.CL
Quick verdict
Augmentation lifts 'look' classification (macro-F 0.386→0.499), but the placement ranking is statistically untested and Table 3's AS F1 looks inconsistent.
Summary
This empirical study asks whether the geometric placement of synthetic training examples matters for discourse-pragmatic function classification. The authors manually annotate 410 British National Corpus instances of English 'look' across four functions (Attention Signal 287, Directive 71, Discourse Marker 34, Interjection 18), generate function-specific synthetic examples with Llama 3.1 8B (via Ollama) after back-translation and cross-translation failed, and partition synthetic instances into NEAR/MIDDLE/FAR tiers by mean cosine distance to same-class real training data in frozen RoBERTa-base embedding space. Six logistic-regression conditions with constant augmentation quantity (287 per minority class) are compared over five stratified 80/20 splits; test sets contain only real BNC data. All augmented conditions beat the real-only baseline on macro-F and accuracy in 5/5 splits (macro-F 0.386→0.499 under NEAR; accuracy up to 0.748 under BALANCED), while AUC never improves — interpreted as a decision-boundary shift rather than better probability estimates. Gains concentrate in scarce classes (INTJ F1 0.14→0.36 under NEAR). Key limitations: the central between-condition ranking (NEAR vs RANDOM vs BALANCED) is never tested statistically and differences fall within one standard deviation; n=5 splits without confidence intervals or multiple-comparison correction; per-class test sets hold only ~4–7 instances; the reported AS F1 (0.14–0.20) is arithmetically incompatible with the reported accuracy; no real-data oversampling baseline; no code or data release.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- All five augmentation conditions improve macro-F and accuracy over the real-only baseline in all five stratified splits (macro-F 0.386→0.499 under NEAR; accuracy up to 0.748 under BALANCED).
- No condition improves AUC — augmentation shifts the decision boundary rather than improving the model's underlying probability estimates.
- Core-proximal (NEAR) synthetic examples yield the largest macro-F gain (+0.113), and a distance-balanced mix achieves the highest accuracy (0.748), implying placement in embedding space matters as much as quantity.
- Gains scale with class scarcity: INTJ (18 real instances) benefits most, with F1 rising from 0.14 to 0.36 under NEAR.
Strengths
Controlled, leakage-free evaluation design
Augmentation quantity is held constant across all six conditions (287 per minority class, Table 1), the RoBERTa encoder stays frozen so effects attribute to training-data composition (Sec 4.1), and test sets contain only real BNC examples in all five stratified 80/20 splits (Sec 4.3).
Genuine metric-level dissociation finding
The flat AUC (baseline 0.726±0.053 is the highest; all augmented means negative with 0/5–3/5 wins, Table 11) against rising macro-F/accuracy is interpreted as a decision-boundary shift rather than better probability estimates (Sec 5.1) — a more honest and interesting conclusion than a uniform-improvement story.
Transparent reporting of failed methods and generation pipeline
Unusually candid limitations section
Consistent replication across splits
Weaknesses
Internally inconsistent results table
Central claim rests on untested between-condition differences
Fragile statistics: n=5, no CIs, no correction
Tiny per-class test sets make per-class claims brittle
Missing key baseline: oversampling of real minority data
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether *where* LLM-generated training examples land in embedding space matters for discourse-pragmatic function classification, using English **look** (Attention Signal / Directive / Discourse Marker / Interjection) as the testbed. It claims: (1) every augmentation condition beats a real-only baseline on macro-F and accuracy in 5/5 stratified splits; (2) core-proximal (NEAR) examples give the largest macro-F gain (+0.113, 0.386→0.499) while a distance-balanced mix gives the highest accuracy (0.748); (3) AUC never improves, so augmentation shifts the decision boundary rather than improving probability estimates; (4) gains scale with class scarcity (INTJ F1 0.14→0.36). Data: 410 BNC instances (287/71/34/18), synthetic examples from Llama 3.1 8B via Ollama, frozen RoBERTa-base embeddings, multinomial logistic regression.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper find about synthetic data augmentation for discourse-pragmatic classification?
Using 410 manually annotated British National Corpus instances of the word 'look' in four functions, the authors show that every LLM-generated augmentation condition (Llama 3.1 8B) improves macro-F and accuracy over a real-only logistic-regression baseline in 5/5 stratified splits (macro-F 0.386→0.499 with core-proximal examples), but AUC never improves — suggesting augmentation shifts the decision boundary rather than improving probability estimates.
Does it matter where synthetic examples land in embedding space?
The paper claims yes: examples nearest the real class core (NEAR) gave the largest macro-F gain (+0.113) and a distance-balanced mix the highest accuracy (0.748). However, the differences between placement strategies (e.g., NEAR 0.499 vs RANDOM 0.486 macro-F) are smaller than the split-to-split standard deviation (±0.065) and were never tested statistically, so the ranking is not established.
What data, models and classifier were used?
410 BNC instances of 'look' annotated as Attention Signal (287), Directive (71), Discourse Marker (34) or Interjection (18); synthetic minority-class examples generated with Llama 3.1 8B Instruct via Ollama (Q4_K_M, temperature 1.4) and human-filtered; frozen RoBERTa-base mean-pooled embeddings; multinomial logistic regression; five stratified 80/20 splits tested only on real BNC data.
What are the main limitations of the study?
A tiny corpus (410 instances; only 18 INTJ), frozen possibly anisotropic RoBERTa embeddings with cosine-based distance tiering, only five splits without confidence intervals or multiple-comparison correction, per-class test sets of ~4–7 instances, a single English word, an unverified input-length confound between synthetic INTJ mini-scenes and real BNC contexts, and no public code or data release.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction interleaves figure axis/legend text with body prose (Figures 1–3); all numeric statements in this review are taken from Tables 1–12, which extracted cleanly.
- Appendix C appears in the extraction as tables only (Tables 11–12) with no narrative prose; the name of the statistical test may be specified in unextracted accompanying text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.