How Far Can Sub-3B Open Language Models Go in Zero-Shot Essay Scoring on an 8 GB Consumer GPU?
Nguyen Dung Son, Dang Quang Minh, Nguyen Huu Loi, Truong Viet Vu, Nguyen Thai Anh
cs.LG
Quick verdict
Claims rubric-decomposed prompting best rescues sub-3B local zero-shot essay scoring; a candid, statistically careful benchmark whose best QWK 0.388 still loses to a length-only baseline.
Summary
This benchmark-style study asks how much zero-shot automated essay scoring (AES) capability survives in sub-3B open-weights models run entirely locally in FP16 on a single 8 GB consumer GPU. Four models (Qwen2.5 0.5B/1.5B/3B, SmolLM2-1.7B) are evaluated on all eight ASAP-AES prompts over a score-stratified sample of 1,199 essays, comparing holistic prompting against rubric/trait decomposition with three aggregations, plus bootstrap confidence intervals, Holm-corrected paired tests, and cost accounting across 21,584 cached model calls. Rubric decomposition beats holistic for every model under batch min-max aggregation (+0.103 to +0.192 macro QWK), though both families plateau at the 1.5-1.7B scale. Aggregation interacts sharply with grader calibration: Qwen2.5-3B compresses trait scores into the 2-4 band, collapsing naive mean aggregation (0.204); min-max normalization repairs it to 0.388, and freezing statistics on 30 essays retains 0.361. Signed error falls with essay length in eleven of twelve configurations, opposite to larger LLM judges' verbosity bias. Crucially, the best configuration (0.388) remains below both the human inter-rater ceiling (0.769) and a length-only baseline (0.523), so the authors recommend only human-supervised, formative use. Limitations include 150 essays per prompt, transductive batch normalization, and no code-release statement.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Rubric-decomposed prompting beats holistic prompting for every tested model under batch min-max aggregation, with Holm-corrected significant gains on up to six of eight prompts (+0.103 to +0.192 macro QWK).
- Trait-score aggregation is calibration-dependent: Qwen2.5-3B compresses traits into the 2-4 band so naive mean aggregation collapses to 0.204, while min-max normalization restores 0.388 and a frozen 30-essay calibration retains 0.361.
- Signed error falls with essay length in eleven of twelve configurations, opposite to the verbosity bias of large LLM judges, so the longest essays would be penalized most in deployment.
- A length-only baseline (0.523) outperforms every LLM configuration, and the best result (0.388) stays far below the human inter-rater ceiling (0.769), restricting sub-3B local scoring to formative, human-supervised use.
Strengths
Unusually honest anchoring of results
The paper volunteers a length-only baseline (macro QWK 0.523, Sec. 5 'Deployment robustness') that beats every LLM configuration (best: 0.388), reports the 0.769 human inter-rater ceiling, and gives mean absolute errors in raw score points (Sec. 5 'Cost and error magnitude': 5-13 points on prompts 7-8), instead of burying weak absolutes behind relative gains.
Clean cross-family characterization of the calibration effect
Sec. 5 (Fig. 3, Tables 4-5) documents that Qwen2.5-3B compresses trait scores into 2-4 (mean 3.4, s.d. 1.5), collapsing naive mean aggregation to 0.204, that batch min-max repairs it to 0.388 but is not universally better (SmolLM2 mean 0.345 vs min-max 0.331), and that a frozen 30-essay calibration retains 0.361 - an actionable, deployment-safe finding.
Above-average statistical hygiene for a prompting study
Broad, fully local coverage
Actionable practitioner guidance
Weaknesses
No configuration beats a trivial length baseline
Headline numbers depend on transductive normalization
Small evaluation sample caps inferential strength
No code, data, or cache release statement
Questionable scope choice for the deployment story
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a benchmark-style empirical study of zero-shot automated essay scoring (AES) under a deliberately strict deployment constraint: sub-3B open-weights models, FP16, greedy decoding, one 8 GB consumer GPU (RTX 2070 Max-Q), no API calls. Four models (Qwen2.5-Instruct 0.5B/1.5B/3B, SmolLM2-1.7B) are compared on all eight ASAP-AES prompts over a score-stratified sample of 1,199 essays, contrasting holistic prompting with rubric/trait decomposition (MTS-style, 0-10 per trait) under three aggregations (mean, batch min-max, frozen 30-essay calibration). Three headline claims: (i) decomposition beats holistic for every model under batch min-max (+0.103 to +0.192 macro QWK, Table 2); (ii) aggregation is calibration-dependent - Qwen2.5-3B compresses traits into 2-4 (mean 3.4, s.d. 1.5), collapsing mean aggregation to 0.204, repaired to 0.388 by min-max, with frozen statistics retaining 0.361; (iii) signed error falls with essay length in 11/12 configurations, opposite to LLM-judge verbosity bias. The paper is explicit that capability is 'real, cheap, private, and strictly formative' (Sec. 7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main result of this sub-3B essay scoring paper?
Rubric-decomposed prompting (scoring each rubric trait in a separate call) outperforms holistic single-score prompting for all four tested models (Qwen2.5 0.5B/1.5B/3B, SmolLM2-1.7B) under batch min-max aggregation, improving macro quadratic weighted kappa by 0.103 to 0.192 on ASAP-AES, with Holm-corrected significant gains on 1-6 of the 8 prompts.
How accurate are sub-3B local models at essay scoring?
The best configuration reaches only 0.388 macro QWK, far below the 0.769 human inter-rater agreement and even below a length-only baseline of 0.523. The authors therefore recommend these models strictly for formative feedback with a human in the loop, not for autonomous grading.
Why does aggregation choice matter for trait-based LLM essay scoring?
Because small models calibrate the shared 0-10 trait scale differently. Qwen2.5-3B compresses its trait scores into the 2-4 band, so mapping the trait mean into the prompt score range collapses performance (0.204); min-max normalization repairs it (0.388), and freezing the normalization statistics on 30 essays keeps 0.361 in deployment.
Do small LLMs show verbosity bias when grading essays?
The opposite: in eleven of twelve model-method configurations, signed error falls as essays get longer - the models shift from overrating short essays to underrating long ones. This is opposite to the verbosity bias documented for large LLM judges, and means the longest student essays would be penalized most.
Can I run this essay-scoring pipeline on an 8 GB GPU?
Yes in principle - all models run in FP16 on a single 8 GB consumer GPU (RTX 2070 Max-Q), with the most accurate configurations under four seconds per essay and the full 21,584-call study taking about 4.6 GPU-hours, using publicly downloadable weights and the public ASAP-AES dataset; the paper itself does not release code.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-4 are available only as garbled axis labels in the extracted text; figure-level conclusions rely on the tables and prose.
- Appendix A's full trait descriptions and some appendix content are not fully present in the extraction, so prompt-template fidelity could not be audited.
- No code/data availability statement appears in the text; if one exists outside the extracted pages, reproducibility readiness would rise.
- Abstract is truncated mid-sentence in the extraction; no claims were inferred from the missing tail.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.