StudentBench: AI and human tutoring yield equivalent GRE learning gains
Curtis Northcutt, Inaara Hasmani, Kevin Feng, Trevor Khangi, Andreas Plesner, Jonas Mueller
cs.AI · cs.CY
Quick verdict
Claims pooled AI tutoring matches human GRE gains (TOST p=.015); large RCT but Verbal equivalence fails and the 140-student clustered human arm is fragile.
Summary
The paper introduces StudentBench, a platform that evaluates LLM tutoring by real student learning gains. 2,383 adults were randomized to one hour of AI tutoring (13 LLM tutors), live human tutoring by video, or a no-tutoring control, with GRE-style pre/post tests newly written by former ETS and Kaplan exam creators and counterbalanced (P/Q order swapped) to prevent training-data contamination. TOST equivalence testing at ±0.25-SD bounds found pooled AI gains equivalent to human tutoring (p=.015; combined difference −0.58 pp, 90% CI [−2.18, 1.03]); six of twelve shared AI tutors passed individual tests, and open-weight Gemma 4 31B was equivalent at 918× lower cost per percentage point ($0.0052 vs $4.81). In a second study, 51 expert tutors made 2,028 pairwise rubric evaluations of AI lesson plans and practice problems, with Bradley–Terry rankings led by Anthropic models. Quantitative-only analyses link faster AI replies to more engagement, more correct practice, and larger gains. Key limitations: Verbal equivalence was not established (p=.085); the human arm is small (n=140, seven clustered tutors, 1.59 effective df); a covariate-free sensitivity check failed to establish equivalence; equivalence bounds are wide; costs omit platform overhead; and gains are immediate, without retention follow-up or a self-practice control.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Pooled AI tutoring across 13 LLM tutors produced GRE learning gains statistically equivalent to expert human tutoring (TOST p=.015 at ±0.25-SD bounds).
- Six of twelve shared AI tutors passed individual equivalence tests; open-weight Gemma 4 31B matched humans at 918× lower cost per percentage point gained ($0.0052 vs $4.81).
- Expert human tutors (n=51) preferred Anthropic models' lesson plans and practice problems across 2,028 pairwise rubric evaluations.
- In Quantitative sessions, faster AI replies were associated with more engagement, more correct practice, and larger gains (all p<.002, Holm-corrected across 276 tests).
- The authors argue these results establish the first condition of recursive human self-improvement (RHSI): the technology improves human capabilities.
Strengths
Real learning outcomes at scale, not LLM-judged proxies
2,383 participants and 175k+ messages with gains measured on unaided 27-question post-tests; items newly written by former ETS/Kaplan creators with P/Q form counterbalancing explicitly to avoid training-data contamination (Section 2, Appendix A.1).
Methodologically rigorous equivalence framing
TOST at ±0.25-SD bounds with 90% CIs, ANCOVA with quadratic pretest, CR2 clustering with Satterthwaite df (B.2), Holm correction over enumerated families including a 276-test exploratory family (Table B.1); data-quality filters never use learning gain or post-test outcomes (A.2).
Unusually honest negative reporting
Cross-validated multi-dimensional evaluation
Cost-effectiveness unit is novel but asymmetric
Weaknesses
Headline equivalence is specification-dependent and underpowered
Title/abstract overclaim relative to section-level results
Per-tutor equivalence tests uncorrected
No self-practice control condition
Modality confound unaddressed for learning outcomes
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
StudentBench (arXiv:2609.28470) pairs a randomized comparison of AI vs. human tutoring on GRE-style learning gains with a public benchmark platform spanning five leaderboards: learning gain, lesson planning, practice-problem design, conversational pedagogy, and cost. The central empirical claim is that pooled AI tutoring across 13 LLM tutors (2,139 sessions) produced learning gains statistically equivalent to expert human tutoring (140 sessions) under TOST at ±0.25-SD bounds (p=.015; combined difference −0.58 percentage points, 90% CI [−2.18, 1.03]). Supporting claims: six of twelve shared AI tutors pass individual equivalence tests, with open-weight Gemma 4 31B equivalent (p=.044) at 918× lower cost per percentage point gained; 51 expert tutors prefer Anthropic-generated lesson plans and practice problems across 2,028 pairwise rubric evaluations; and in Quantitative sessions faster AI replies chain through engagement and correct practice to larger gains (all p<.002, Holm-corrected). The paper wraps these results in a recursive human self-improvement (RHSI) framing that claims to establish the first of three conditions; this framing is rhetorically ambitious and adds little evidentiary weight.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did AI tutors match human tutors in the StudentBench study?
In the pooled analysis across 13 LLM tutors (2,139 sessions), learning gains were statistically equivalent to the 140 human-tutored sessions under TOST equivalence testing (p=.015, bounds ±0.25 SD; difference −0.58 percentage points). However, equivalence held only marginally for Quantitative (p=.028) and failed for Verbal (p=.085), where human tutors retained higher mean gains, and the pooled test had only 1.59 effective degrees of freedom.
What does the 918× cost claim mean?
The open-weight model Gemma 4 31B cost about $0.0052 per percentage point of learning gain, versus $4.81 for a human tutor priced at $75/hour — a 918× ratio. It counts model inference costs only; platform development, orchestration, and recruitment overheads are excluded, and one compared Gemini price was a promotional rate.
Is the StudentBench data and code available?
Yes. The session data is released on HuggingFace (handshake-ai-research/studentbench) and the analysis code on GitHub, under IRB protocol 26-206-2643 (BRANY SBER) with anonymized session identifiers and no identity mappings or recordings.
Do the learning gains persist over time?
Unknown. The study measured immediate post-test gains only and does not report retention; the authors explicitly call for longitudinal studies of learning retention after AI tutoring as a limitation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Provided text truncates the abstract mid-sentence and omits parts of Section 5, Appendix G, and scattered passages; some cost/engagement definitions were reconstructed from appendix cross-references.
- Figures appear only as embedded caption/axis text; visual CI overlap in Figures 1, 3-6 could not be independently verified.
- Pre-registration status and full expert-review rubric wording (Appendix E) could not be verified from the extraction.
- Model names reflect the paper's 2026 setting (GPT-5.5, Opus 4.8, Gemini 3.6, etc.) and were not independently verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.