Edustories: A Collection of Real-world Case Studies from Classroom Practices
Michal \v{S}tef\'anik, Jan Nehyba, Jirina Karasova, Martin Fico, Lucie \v{S}karkov\'a, Mark\'eta Ko\v{s}atkov\'a, David Kosatka
cs.CL · cs.AI
Quick verdict
Introduces Edustories, a novel 1,492-case teacher-written dataset; solid resource, but LLM-vs-expert comparison lacks statistical rigor and construct clarity.
Summary
The paper introduces Edustories, a public dataset of 1,492 case studies written by 241 Czech trainee teachers during classroom practice between 2023 and 2026. Each case narrates a classroom situation, student anamnesis, problem, teacher intervention, and perceived outcome, enriched with expert-annotated categorical labels (14 problem, 15 intervention, 3 outcome categories) plus student age, hobbies, and diagnoses. Texts were anonymized in two stages (UFAL NER plus manual review), covered by informed consent, and translated to English with GPT 5.2, chosen over DeepL, Google Translate, and OPUS in a 50-sample human evaluation. As a first application, the authors benchmark six open-weight LLMs (Llama 3, Qwen 3, Mistral v0.3, Olmo 3) on three-way outcome prediction from description, anamnesis, and intervention, comparing against five education experts who rated 310 cases via behavior-management rubrics. The best model (Qwen3-30B, 0.580) trails the best expert (0.642) and only overlaps weaker experts; the majority-label baseline is 0.446. Experts excel at identifying long-term success (84.4%) while models confuse short- and long-term success. Reported limitations include novice-only authors, 5-10% translationese, no models above 100B parameters, and no proprietary APIs. The resource is valuable and the evaluation transparent, but statistical testing is absent, the prompt was selected on the evaluation set, and outcome labels ultimately reflect trainees' self-reported perceptions.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Edustories is the first public dataset of holistic teacher-written case studies documenting real collective-classroom situations, interventions and outcomes (1,492 cases; §1, §4).
- Current open LLMs fall short of human experts at predicting intervention outcomes: best model 58.0% vs best expert 64.2% accuracy on 310 cases, with a 44.6% majority-class baseline (§3.2, Figure 2).
- Model scale is not the primary performance factor, as Llama-3.3-70B underperforms Llama-3.2-3B (§3.2).
- Humans and models show complementary error patterns — experts rarely miss long-term success (84.4%) while models confuse short- and long-term success — supporting assistive rather than replacement use of AI (§3.2, Tables 2-5).
Strengths
First-of-kind public resource
Per §1.1/§2, related classroom datasets are video-only (MM-TBA, TBU, FSCB) or classification-oriented (CEReD); Edustories' 1,492 holistic consented narratives with expert categorical annotations (14 problem, 15 intervention, 3 outcome classes; Table 1) fill a genuine gap.
Careful data governance
§2: two-stage anonymization (UFAL Czech NER randomization plus manual review), written informed consent, ethics-committee approval (footnote 1), and Czech originals retained alongside translations (A.4).
Transparent LLM evaluation setup
Informative error analysis
Strong expert agreement for a subjective task
Weaknesses
No statistical testing
Prompt selected on the evaluation set
Construct validity of outcome labels
Internal inconsistencies and sloppiness
Narrow model scope vs abstract framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Edustories (arXiv 2609.20484) is a dataset paper with an embedded benchmark study. The dataset comprises 1,492 case studies written by 241 trainee teachers at Czech elementary and high schools (2023-2026), each with five narrative fields (situation description, student anamnesis, problem, intervention, perceived outcome) plus expert categorical annotations — 14 problem, 15 intervention, and 3 outcome categories — and structured student metadata (age, hobbies, 72 diagnosis and 103 disorder categories; Table 1). The headline claims: (1) this is the first public dataset documenting collective-classroom situations at narrative depth (§1, §4); (2) state-of-the-art LLMs still trail human experts at predicting intervention outcomes, 58% vs 64% (Abstract, §3.2); (3) model scale is not the deciding factor (§3.2); (4) complementary human-model error profiles argue for assistive rather than replacement roles (§3.2). Claims are mostly supported by the presented data, with caveats below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Edustories dataset?
A public collection of 1,492 real classroom case studies written by 241 Czech trainee teachers between 2023 and 2026. Each case describes a challenging situation, student background, the teacher's intervention, and the outcome, with expert-annotated category labels and English translations (HuggingFace: MU-NLPC/Edustories-en; edustories.cz).
How well do LLMs predict whether a teacher's intervention will succeed?
On a 310-case subset, the best model (Qwen3-30B) reached 58.0% three-way accuracy versus 64.2% for the best of five human education experts; the majority-class baseline was 44.6%. Models mainly confuse short-term with long-term success, while experts rarely miss long-term success.
Which AI models were evaluated in the paper?
Six open-weight models prompted identically: Qwen3-30B-A3B-Instruct-2507, Qwen3-4B-Instruct-2507, Llama-3.3-70B-Instruct, Llama-3.2-3B-Instruct, Mistral-7B-Instruct-v0.3, and Olmo-3-7B-Instruct. No proprietary APIs (GPT, Claude, Gemini) were tested, citing classroom-data privacy.
Is the Edustories data anonymized and ethically cleared?
Yes: personal names, dates and locations were randomized with a Czech named-entity-recognition service and then manually reviewed; all case authors gave written informed consent; and the collection protocol received a positive review from the coordinating institution's ethics committee.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2 bar-to-model assignment for the two lowest LLM values (0.493/0.482) is inferred from label order and §3.2 text (Llama-3.3-70B below Llama-3.2-3B); the pair could in principle swap.
- No explicit dataset license is quoted in the extracted text; availability is stated only via HuggingFace and edustories.cz links.
- The exact selection procedure for the 310-case human-evaluation subset (stratification details) is not documented beyond 'representatively covering the range of students' age'.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.