A Human-in-the-Loop Framework for AI-Assisted Scoring in Large-Scale Writing Assessment
María Eugenia Curi, Germán Capdehourat, Isabel Amigo, Magdalena Romano, Rosana Serra, Adrián Silveira, Andrés Peri
cs.CL · cs.AI
Quick verdict
Claims GPT-5 prompt scoring with human-review routing can safely cut grading workload ≥50%; credible real-data evidence, yet validation is simulation-only.
Summary
This study evaluates whether GPT-5 can support scoring of Spanish argumentative writing in Uruguay's Acredita EB, a high-stakes national test certifying lower secondary education for adults over 21. Using two editions (2024, 6,204 participants; 2025), the authors operationalized a 15-item analytic rubric as per-item Spanish prompts with structured outputs; the spelling item was replaced by deterministic LanguageTool after unstable LLM counts. On random 1,000-text samples per edition, item-level accuracy ranged 60-80%, about 5-15 percentage points below human inter-rater agreement measured on a 50-text, 10-rater calibration set; consistency across 10 runs was ~90%. Aggregating items through replicated IRT and bookmark-style cut scores, the model shows a systematic conservative bias: 15.3% (2024) and 16.5% (2025) of human pass decisions would become AI fails, versus only 0.2-0.6% in the opposite direction. The proposed human-in-the-loop workflow accepts AI-passed cases, routes AI-failed cases to human raters only when the overall outcome depends on Writing, and is simulated to cut full human scoring by at least half. Main limitations: single-rater reference labels, purely offline simulation without an operational pilot, unquantified agreement between automated cut scores and official levels, no baseline model comparisons, and confidential data and code.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GPT-5 with per-item rubric prompts reaches 60-80% item-level accuracy against operational human scores, within roughly 5-15 percentage points of human inter-rater agreement on most rubric items.
- The model's residual errors are systematically conservative, flipping 15.3% (2024) and 16.5% (2025) of human pass decisions to fail while producing reverse errors in only 0.2-0.6% of cases.
- A HITL workflow that accepts AI-passed cases and routes AI-failed cases to human raters — only when the overall outcome depends on Writing — preserves pass/fail quality while cutting full human scoring workload by at least 50%.
- Prompts developed on the 2024 edition generalize to 2025 with only minor topic-related adjustments.
Strengths
Real high-stakes operational context with longitudinal data
Two full editions of Uruguay's national Acredita EB exam (6,204 participants in 2024; ~5,000-6,000 in 2025), not a benchmark dataset — rare deployment-relevant evidence for AES in Spanish (Sections 1, 4).
Human inter-rater ceiling analysis
Ten raters independently scored the same 50 calibration texts (Fig 2), letting the authors benchmark AI accuracy against actual human agreement (gap ~5 points for over half the items, never above 15) instead of an arbitrary target (Sections 5.1, 6.1).
Decision-oriented evaluation rather than item-accuracy chasing
Cross-year generalization demonstrated
Honest handling of model weakness
Weaknesses
Modest item-level agreement on several dimensions
Single-rater operational ground truth
Headline workload claim is simulation only
Automated cut-score replication unvalidated quantitatively
No baselines and low reproducibility
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper evaluates GPT-5 as an assistant — not a replacement — for scoring the Writing section of Acredita EB, Uruguay's national accreditation exam granting lower-secondary certification to adults over 21 (ANEP; results currently take about three months because essays are graded manually). The task: 150-200-word Spanish argumentative texts scored on a 15-item analytic rubric spanning discursive, textual and orthographic domains (Section 4.3). The central claims: (1) prompt-based per-item scoring attains usable agreement with human raters (60-80% item accuracy, 5-15 points under human inter-rater agreement); (2) residual errors are systematically conservative (under-grading), exploitable in workflow design; (3) a HITL flow that accepts AI passes and routes AI fails to human review — only when the overall outcome depends on Writing — preserves pass/fail quality while halving human workload.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which AI model did the study use to grade the essays?
OpenAI GPT-5, accessed through an enterprise API (which does not retain data for training), run with medium reasoning effort and structured JSON outputs. The authors wrote one Spanish prompt per rubric item (15 items total). The spelling item was scored by the deterministic rule-based tool LanguageTool instead of the LLM, because the model counted spelling errors inconsistently across repeated runs.
How much teacher grading workload could this approach save?
In simulation over the 2024-2025 exam data, at least 50% of essays would not need full human scoring. The saving comes from the pass rule: candidates who cannot pass regardless of their Writing result, or who pass under any plausible Writing error, are not reviewed at all; only essays the AI marks as failing — and whose outcome depends on Writing — go to human raters. This is a retrospective estimate, not yet an operational deployment.
How accurate was the AI compared to human graders?
On random 1,000-essay samples per edition, per-rubric-item accuracy was 60-80%, roughly 5-15 percentage points below the agreement measured between ten trained human raters on a 50-text calibration set. In pass/fail terms, the AI would fail 15.3% (2024) and 16.5% (2025) of essays that humans passed, while only 0.2-0.6% of essays went the other way — a systematic 'stricter than humans' bias that the proposed workflow turns into a review trigger.
Are the data or code from this study available?
No. The essays and scores come from a confidential national assessment administered by Uruguay's ANEP and cannot be shared; no code repository is released, and the paper only sketches the prompt structure rather than publishing full prompts. Reproducing the results would require access to comparable operational exam data and the same commercial model.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-7 are embedded images; only in-text statements, ranges and axis labels were extracted, so exact per-item accuracy values beyond the stated 60-80% band and 5-15 point gap could not be verified from plotted values.
- Cell orientation within the Fig 6 proficiency confusion matrix (rows vs columns) is inferable only from the caption context, so cell-level claims were avoided; the pass/fail matrix (Fig 7) labels are unambiguous and were used instead.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.