I code or AI code: A comparative evaluation of AI-rated scores in classroom observations
Y. Fong, J. Xiang, T. Y. D. Chan, K. Lee, E. Y. H. Lau
cs.CL · cs.AI
Quick verdict
GPT-5 CLASS scoring agrees with human raters only modestly and by domain (max r=.41); a promising screening aid, but paired data were analyzed as independent.
Summary
This preprint evaluates whether a large language model (Perplexity-hosted GPT-5.0) can score early-childhood classroom observations on the CLASS Pre-K framework comparably to trained human raters. From 87 videotaped observations (38 K1 classrooms, 30 Hong Kong kindergartens, ~26.5 hours), bilingual transcripts were prepared; the AI received the CLASS manual and four prompt-based calibration cycles on 16 transcripts with human consolidated scores, then independently scored the remaining 71. Human scores came from eight raters (two per video, kappa .672-.861). Agreement was modest and domain-dependent: Emotional Support correlated .406 with AI ratings, while Classroom Organization (.123) and Instructional Support (.195) were non-significant; among dimensions only Positive Climate (.438), Teacher Sensitivity (.313), Regard for Student Perspectives (.296), and Quality of Feedback (.269) reached significance. Pooled weighted kappa was .681. Independent-samples t-tests indicated raters scored Emotional Support substantially higher than AI (d = 1.34) and Instructional Support lower. Key limitations: one proprietary model scored once, transcript-only input, no test of the AI's own scoring stability, equivalence inferred from non-significant tests, paired data analyzed as independent, and a heavily quality-screened subsample (83% of videos excluded).
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A transcript-based Perplexity GPT-5.0, calibrated through four prompt cycles on 16 human-annotated transcripts, can apply the full CLASS Pre-K framework and reaches pooled weighted kappa of .681 with consolidated human raters.
- AI-human convergence is construct-dependent: significant only for the Emotional Support domain (r = .406) and for four of ten dimensions (Positive Climate .438, Teacher Sensitivity .313, Regard for Student Perspectives .296, Quality of Feedback .269).
- Raters systematically score Emotional Support higher than the AI (e.g., +1.15 on Positive Climate, d = 1.52) while the AI scores Instructional Support higher than raters, with Classroom Organization showing essentially no alignment (r = .123).
- Transcript-based AI cannot yet reproduce calibrated human judgment across the full CLASS framework and should serve as a screening or formative complement rather than a replacement for trained observers.
Strengths
Held-out evaluation against a calibrated human benchmark
The AI was calibrated on 16 transcripts with human consolidated scores and evaluated on 71 unseen ones, avoiding train-test contamination; the eight human raters completed two five-video calibration rounds (weighted kappa .672-.861) and each evaluation video was double-coded and averaged (Sections 2.3.2-2.3.3).
Multi-level, quantified comparison
Domain and dimension descriptives (Table 2), correlations (Table 3), pooled and per-rater kappas (.681; .515-.712), and effect-sized mean comparisons (Table 4, Cohen's d up to 2.73 for Negative Climate) go well beyond accuracy-style reporting.
Score-distribution diagnostics reveal the mechanism of divergence
Candid treatment of its own hypothesis
Ecologically valid deployment setting
Weaknesses
Paired data analyzed as independent
Equivalence claimed from non-significance
No reliability of the AI itself; single run of one proprietary model
Under-specified headline kappa
Selection and missing-data burden
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can GPT-5 score classroom observations as reliably as trained human raters?
In this Hong Kong study of 87 kindergarten videos, a GPT-5-based model reached a pooled weighted kappa of .681 with human raters, which falls within the range CLASS treats as acceptable (about .60-.75). However, correlations with human scores were significant only for the Emotional Support domain (r = .406) and four of ten dimensions. The authors conclude AI is usable as a screening and formative aid, not as a stand-alone or high-stakes evaluator.
What is the CLASS framework and how was it used with AI?
CLASS Pre-K (Pianta et al., 2008) rates teacher-child interactions on ten dimensions (e.g., Positive Climate, Behavior Management, Quality of Feedback) on 1-7 scales, grouped into Emotional Support, Classroom Organization, and Instructional Support domains. In this study, transcripts of classroom videos were given to GPT-5 along with the manual and 16 annotated example transcripts for calibration, after which the model scored 71 unseen transcripts; these were compared with ratings from two trained human raters per video.
Where did AI and human raters disagree most in scoring classrooms?
Raters scored Emotional Support substantially higher than the AI (e.g., Positive Climate +1.15 points, d about 1.5), while the AI scored Instructional Support higher than raters (Concept Development +0.41). The weakest alignment was Classroom Organization (r = .123), a domain that depends on nonverbal, procedural, and spatial cues invisible in transcripts. Human raters also clustered at the ceiling on reversed Negative Climate (63 of 71 videos at score 7), which the AI did not.
Can AI classroom observation replace human observers?
The authors argue no, not yet. AI-human alignment was partial and construct-dependent, and the study used only one LLM, transcript-only input (no video or audio cues), and a quality-screened subsample (over 83% of recordings excluded). They recommend AI as a first-pass screening and reflection tool with humans retaining judgment, and call for multimodal, multi-model follow-up research.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 (CLASS overview, training procedure, Negative Climate distribution) appear only as captions in the extracted text; their content was inferred from the surrounding prose and numeric reporting.
- Appendices A and B (the actual prompts and calibration-cycle details) are referenced but not included in the extracted text, limiting assessment of the exact AI configuration.
- The extracted text contains embedded line numbers and minor OCR artifacts; numeric values in Tables 2-4 were cross-checked for internal consistency and appear intact.
- It is not specified whether the 16 AI-training transcripts overlap with the 10 human-calibration videos (16 + 71 = 87 is consistent, but the split's provenance is unstated).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.