Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Prompting
Raphaël Thieffry, Matej Martinc
cs.CL
Quick verdict
Careful benchmark of 5 open LLMs on readability levels: native reasoning and 1-shot prompting help most; findings credible but lacks significance testing.
Summary
The paper benchmarks five open-weight LLMs (gpt-oss-20b, phi-4, gemma-3-27b-it, GLM-4.7-Flash, Qwen3.5-9B) as zero-training readability assessors that predict discrete educational levels, across four corpora (Newsela, OneStopEnglish, WeeBit in English; Učbeniki Slovene schoolbooks), each stratified-sampled to at most 567 documents. Crossing four output regimes (direct, elicited CoT, native reasoning, two-pass) with prompt format and k∈{0–3} labelled examples per class, it evaluates 160 configurations against five classical formulas and supervised encoders from prior work. Key findings: native reasoning channels yield the only reliable gains (+0.136 Spearman ρ, +0.186 quadratic weighted κ on average over direct answering); elicited or two-pass rationales do not help. The first labelled example per class carries most of the few-shot benefit (mean ρ 0.603→0.646, then flat), with further examples refining exact-level accuracy but not ranking. The best configuration (Qwen3.5-9B, native reasoning, 1-shot) reaches ρ 0.753 on Slovene, beating all formulas there (+0.311 on Učbeniki), but trails sentence length on the two parallel rewritten corpora (Newsela 0.788 vs 0.948) and remains below supervised ceilings. Limitations acknowledged by the authors: no significance testing, serving-config instability flipping up to 23.1% of predictions, context truncation affecting ~25% of few-shot prompts on two corpora, and sample-based ranking unverified at scale.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
6 strengths · 6 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reasoning improves LLM readability classification only when the model produces it natively: native reasoning gains +0.136 ρ and +0.186 quadratic weighted κ on average over direct answering, while elicited CoT and two-pass rationales do not reliably help (Table 4).
- One labelled example per class captures most of the few-shot benefit (mean ρ 0.603→0.646 from k=0→1, then flat), and it substitutes for reasoning mainly in ranking: +0.080 ρ from k=0→1 for direct scoring vs +0.016 for native reasoning (§5.2).
- The best training-free configuration (Qwen3.5-9B, native reasoning, 1-shot) beats every classical formula on the Slovene corpus (ρ 0.753 vs 0.437–0.442 best) and on WeeBit, but trails average sentence length on the two parallel rewritten corpora and remains below supervised encoders.
- Declared multilingual coverage predicts Slovene performance: the three multilingual models lead both English-centric ones, and gpt-oss (ρ 0.696) stays below the direct condition of gemma and qwen (§5.3).
Strengths
Clean separation of native vs elicited reasoning
Table 4 contrasts each regime against the same model's direct condition within-model: native reasoning gains +0.136 ρ / +0.186 QWK / +0.086 F1 on average, while elicited (+0.006 ρ / −0.027 κ) and two-pass regimes fail — a genuinely novel and practically useful distinction for LLM-as-judge work.
Discrete-level framing with ordinal metrics
Unlike Grossman & Chen's continuous expected-value scores, the paper treats readability as absolute classification over each corpus's integer levels, reporting QWK and weighted F1 alongside Spearman (§3.4), which matches what educational frameworks actually need; the vanilla-score choice is explicitly justified to keep regimes comparable.
Exemplary honesty about confounds
Strong reproducibility package
Extends to an under-served language with a coverage finding
Informative prompt ablation
Weaknesses
No inferential statistics across 160 configurations
Sample-based ranking never verified at scale
Few-shot findings confounded by context truncation
Regime conclusions rest on few model-corpus pairings
Supervised baselines imported, not retrained
Minor numeric inconsistency between text and table
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper benchmarks five open-weight instruction-tuned LLMs (gpt-oss-20b, phi-4, gemma-3-27b-it, GLM-4.7-Flash, Qwen3.5-9B; Table 1) as training-free readability assessors that must emit a discrete level from each corpus's own scale, across four datasets (Table 2): Newsela (11 US grades), OneStopEnglish (3 ESL bands), WeeBit (5 age bands) and the Slovene schoolbook corpus Učbeniki (13 grades), each stratified-sampled to at most 567 documents. Four output regimes (direct, elicited JSON rationale, native reasoning channel, two-pass) are crossed with k ∈ {0,1,2,3} labelled examples per class and two prompt formats, yielding 160 configurations compared against five classical formulas and supervised encoders from Martinc et al. [19]. Central claims: (1) reasoning helps only when a model produces it natively; (2) one example per class captures most of the few-shot gain; (3) the best configuration beats all formulas on Učbeniki and WeeBit but not on the two parallel rewritten corpora; (4) supervised ceilings remain higher. The framing as absolute classification with QWK/F1/ρ — rather than Grossman & Chen's continuous expected-value scores — is a real advance aligned with how educational frameworks consume readability levels.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 6 strengths & 6 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLMs beat traditional readability formulas like Flesch or Dale-Chall?
It depends on the corpus. In this benchmark of five open-weight LLMs, the best training-free configuration (a 9B reasoning model with one labelled example per class) clearly beat all five formulas on a Slovene schoolbook corpus (Spearman ρ 0.753 vs 0.437–0.442) and on the WeeBit children's texts (0.719 vs 0.649). But on two English corpora built by rewriting single articles across grade levels (Newsela, OneStopEnglish), the trivial average-sentence-length formula won (0.951 vs 0.788 on Newsela), because sentence shortening was exactly what the corpus annotators manipulated.
Does chain-of-thought prompting improve LLM readability assessment?
Only when the model produces reasoning through its own native reasoning channel. Across four corpora, models with native reasoning gained on average +0.136 Spearman ρ and +0.186 quadratic weighted κ over direct answering. Asking non-reasoning models to produce a rationale (as JSON, or in a separate analysis pass) yielded no reliable improvement — in some configurations it hurt. The practical lesson: pick a model trained to reason; don't try to prompt a rationale out of one that isn't.
How many few-shot examples should I give an LLM for readability classification?
One labelled example per output class appears sufficient. In this study, moving from zero-shot to one example per class raised mean Spearman ρ from 0.603 to 0.646, but adding a second and third example per class brought no further gains in ranking quality (0.640, 0.634), though exact-level accuracy kept improving slightly. The first example helped direct-scoring models about five times more (in ρ) than models already using native reasoning — examples partly substitute for reasoning.
Can LLMs assess text readability in low-resource languages like Slovenian?
Yes, and they are the strongest training-free option there. On a Slovene schoolbook corpus with 13 grade levels, zero-shot native reasoning reached Spearman ρ 0.876 (Qwen3.5-9B) and 0.811 (GLM-4.7-Flash), far above the best classical formula (0.437–0.442). Models documented as multilingual beat the English-centric models, and results were still below supervised models trained on annotated Slovene data (κ 0.885) — so LLMs win precisely where no annotated training corpus exists.
Is the code and data for this readability benchmark available?
Yes. The authors state that all code, prompts and evaluation scripts are publicly available on GitHub (raphael-thieffry/LLM-Readability-Assessment). The four evaluation corpora (Newsela, OneStopEnglish, WeeBit, Učbeniki) are existing public datasets, and all five models are open-weight, so the experiments are reproducible given GPU resources.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review based on the extracted full text; no figures were present in the extraction, but all tables (1–4) appear intact.
- The paper is arXiv v1 (Sep 2026); several citations are dated 2026 (e.g., ACL 2026), so some cited work may still be in press and unverifiable from this text alone.
- Minor numeric discrepancies between §5.4 text and Table 3 formula baselines (0.442 vs 0.437; 0.948 vs 0.951) could not be resolved from the text.
- Prompts themselves are only described, not shown in the text; assessment of prompt wording relies on the authors' descriptions and the public repository.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.