Error-Supervised Synthetic Learner Writing for Automated Essay Scoring
cs.CL
Quick verdict
Error-supervised synthetic essays beat conventional baselines in 11/12 large-data AES comparisons; single-run evidence and no significance tests temper it.
Summary
The paper proposes a simple modification to synthetic essay generation for Automated Essay Scoring (AES): the generator (Qwen2-7B with QLoRA) is fine-tuned on error-annotated learner texts of the kind used in GED/GEC research, so generated essays carry realistic learner errors rather than being overly well-formed; error tags are then stripped deterministically. Three English learner corpora are studied — CLC FCE (manual annotations), Write & Improve (ERRANT M2), and ELLIPSE (annotations derived automatically via GECToR-2024 plus ERRANT) — with BERT as the sole AES scorer in a prompt-unaware setting, at 50/100/200 essays (five seeds) and full/larger-data settings (1,725 and 3,480; ELLIPSE capped at 2,000 for compute). In larger-data settings the proposed synthetic data beat conventional synthetic data in 11 of 12 dataset-metric comparisons, approaching authentic-data performance on W&I (QWK 0.8209 vs 0.8325) and even exceeding it on CLC FCE MAE (1.8062 vs 1.8417). Low-resource results (50-100 essays) remain mixed, with no consistent advantage. Error analyses (JSD mostly below 0.25, lower error density than authentic text) support learner-likeness but reveal over-representation of some error types (ELLIPSE missing punctuation +16.89 pp). Main limitations: a single AES architecture, no statistical significance testing, single-run full-data comparisons, license-restricted data, and no released code.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Fine-tuning an essay generator on error-annotated learner texts produces synthetic essays whose error-type distributions broadly resemble authentic learner writing at matched proficiency levels (JSD mostly below 0.25).
- In larger-data settings, AES scorers trained on error-supervised synthetic essays beat conventionally generated synthetic essays in 11 of 12 dataset-metric comparisons.
- Error-supervised synthetic data can approach authentic-data performance (W&I full-data QWK 0.8209 vs 0.8325) and even exceed it on some metrics (CLC FCE full-data MAE 1.8062 vs 1.8417).
- Error supervision does not require manual annotations: automatically derived tags (GECToR-2024 plus ERRANT on ELLIPSE) still yield modest downstream gains.
- Advantages are inconsistent in extremely low-resource settings of 50-100 training essays.
Strengths
Honest reporting of negative and mixed results
Abstract, §6.2 and §8 explicitly state that low-resource (50/100) advantages are inconsistent and that W&I at n=200 favors the conventional baseline on 3 of 4 metrics; every table includes the authentic-data reference.
Annotation-provenance span strengthens external validity
Two manually annotated corpora (CLC FCE Cambridge taxonomy; W&I ERRANT M2) plus one automatically annotated corpus (ELLIPSE via GECToR-2024+ERRANT, §6.2.3) show the method works beyond hand-annotated GED resources.
Unusually complete experimental disclosure
Dual error-profile evaluation
Clear positioning against prior synthetic-AES work
Weaknesses
No statistical inference for the headline claims
Single, small scorer architecture
Residual distribution mismatch in generated errors
Reproduction gated on restricted data and no code
Low-resource benefit — the paper's motivating scenario — is not demonstrated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether supervising a synthetic-essay generator with learner-error annotations — the kind used in Grammatical Error Detection/Correction (GED/GEC) — yields synthetic training data more useful for Automated Essay Scoring (AES) than conventionally generated synthetic essays. Two research questions structure the work (§1): downstream AES utility (RQ1) and resemblance of error patterns (RQ2). The central claims are: (1) in larger-data settings the proposed synthetic data beat the conventional synthetic baseline in 11 of 12 dataset-metric comparisons; (2) some results approach or slightly exceed authentic-data training (CLC FCE full-data MAE 1.8062 vs 1.8417; W&I QWK 0.8209 vs 0.8325); (3) automatically derived annotations (ELLIPSE) suffice as supervision; (4) advantages in extremely low-resource settings (50-100 essays) are inconsistent — a restriction the abstract states honestly. Positioning in §2 is precise: unlike rule-based corruption (DREsS, Yoo et al. 2025) or unannotated proficiency-conditioned generation (Chen et al. 2026; Do et al. 2026), error creation happens inside the generator, and unlike the Arabic error-injection framework of Qwaider et al. (2025), no post-hoc injection pipeline is needed.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is error-supervised synthetic essay generation?
Instead of generating polished essays, the authors fine-tune a language model (Qwen2-7B) on learner essays carrying inline grammatical-error annotations of the kind used in grammatical error detection/correction (GED/GEC) research. The model learns to write learner-like essays containing realistic errors, which are then stripped of tags and used as training data for automated essay scoring systems.
Can synthetic essays replace real human-scored essays for training essay scorers?
Not fully. Error-supervised synthetic essays approach real-data performance and even beat it on some metrics (CLC FCE full-data MAE 1.8062 vs 1.8417), but on ELLIPSE a clear gap remains (QWK 0.6116 vs 0.7010 at 2,000 samples), and at 50-100 training essays synthetic data shows no consistent advantage over conventional generation.
Which datasets and models were used in this study?
Three English learner corpora: CLC FCE (manually error-annotated exam scripts, scores 0-20), Write & Improve (ERRANT-annotated, CEFR-labelled), and ELLIPSE (holistic scores 1.0-5.0, errors derived automatically with GECToR-2024 and ERRANT). The generator is Qwen2-7B-Instruct fine-tuned with QLoRA; the essay scorer is bert-base-uncased in a prompt-unaware setting.
Is code or data from this paper publicly available?
The paper does not mention a released code repository. CLC FCE and Write & Improve are license-restricted (Cambridge University Press & Assessment granted permission for appendix examples), while ELLIPSE is publicly available under CC BY-NC-SA 4.0.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (error density and JSD plots) could not be analyzed from extracted text; per-level values rest on the prose in §5 and Appendix B.2
- Appendix Table 7 example essays contain minor extraction artifacts (spacing, tag rendering); qualitative judgments partly based on garbled text
- Extracted text appears complete through Appendix C (Table 9), so table-level conclusions rest on solid extraction
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.