A Dataset for Modeling Iterative Problem-Solving
Fagun Patel, Sang T. Truong, Duc Q. Nguyen, Kazunori Fukuhara, Benjamin W. Domingue, Sanmi Koyejo, Nick Haber
cs.CL
Quick verdict
CodeInsight dataset (3M+ submissions) + shared benchmark; RSSM best on 3/4 courses; careful LLM analysis, but no significance tests and gated data.
Summary
The paper introduces CodeInsight, a dataset of over 3 million code submissions from 3,286 undergraduates across two introductory C++ courses (Programming Fundamentals and Data Structures & Algorithms) taught in two academic years at a Vietnamese university, with per-test-case pass/fail outcomes, timestamps, source code, and admission-track labels. On it, the authors build a benchmark for next-attempt prediction that, for the first time, evaluates parametric (IRT, CIRT), sequential (BKT, DKT, Code-DKT, TIKTOC), and generative models (an RSSM with discrete latents adapted from DreamerV2, plus a frozen LLM predictor) under one calibration-and-scoring protocol with a shared student split. Their adapted RSSM attains the highest AUC on three of four courses (e.g., 0.831 vs 0.813 for CIRT on DSA HK231 in the filter-sensitivity baseline), with gains concentrated at later attempts; the frozen LLM predictor (Qwen3-14B) trails nearly everywhere. Analyses show the LLM's signal is driven by problem difficulty rather than solver-specific behavior: per-student Kendall tau collapses toward zero after removing problem means, and direct-solve pass rates correlate r=0.80 with context-conditioned rates versus r=0.40 with real students. Stronger coders (Opus 4.6) predict worse. Limitations: single institution, language, and platform; PF courses dominated by floor effects (8.4%/17.5% pass rates); no significance testing around the single-seed ranking; and the dataset is released only on request under a departmental license.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CodeInsight is the first large-scale dataset combining per-test-case pass/fail labels, full source code, and timestamps across over 3 million submissions from iterative C++ coursework.
- Under a shared calibration-and-scoring protocol, an RSSM adapted with discrete latent variables achieves the highest predictive accuracy (AUC) on three of the four courses.
- A frozen, non-fine-tuned LLM predictor generates full code submissions but is less accurate than all trained baselines at nearly every attempt.
- The LLM's predictive signal is driven by problem-level difficulty rather than solver-specific behavior, and stronger coders (e.g., Opus 4.6) predict worse than weaker ones (Qwen3-14B).
- LLMs in this setting are better understood as generative solvers conditioned on context than as faithful simulators of individual students.
Strengths
Unmatched data granularity
3,074,795 submissions with per-test-case pass/fail labels across 3,888 test cases, full source code, and timestamps (Table 1); unlike CodeWorkout (aggregate pass %), ACcoding (single judge verdict), and Blackbox (no correctness labels) as the paper itself contrasts in Section 2.
First cross-tradition comparison under one protocol
Section 3 and Figure 2 put parametric (IRT/CIRT), sequential (BKT/DKT/Code-DKT/TIKTOC), and generative models (RSSM, frozen LLM) on a shared 55/15/30 student split with frozen item parameters transferred from train students — a genuinely missing comparison in the literature.
Filter sensitivity analysis
Careful LLM failure diagnosis
Exemplary documentation
Weaknesses
No uncertainty quantification for the headline ranking
Skewed PF courses inflate and distort comparisons
Proficiency-inversion claim rests on three frozen models
Dataset is access-gated
Sensitive content in the release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes two coupled contributions. First, **CodeInsight**: 3,074,795 submissions from 3,286 undergraduates across 394 problems and 3,888 test cases, collected from two introductory C++ courses (PF and DSA) each offered twice across 2022-2024 at VNU-HCM University of Technology, with per-test-case pass/fail labels, timestamps, source code, and admission-track labels (Section 4, Table 1). Second, a **next-attempt prediction benchmark** presented as the first to compare parametric (IRT, CIRT), sequential (BKT, DKT, Code-DKT, TIKTOC), and generative models (a DreamerV2-style RSSM; a frozen LLM predictor) under one calibration-and-scoring protocol (Section 3, Figure 2). Central claims: the RSSM wins AUC on 3 of 4 courses; the frozen LLM (Qwen3-14B) trails trained baselines at nearly every attempt; and the LLM's signal is difficulty-driven rather than solver-specific (Section 5.2, Figures 7 and 20).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the CodeInsight dataset?
CodeInsight is a dataset of over 3 million code submissions from 3,286 undergraduates in two introductory C++ courses (Programming Fundamentals and Data Structures & Algorithms) across the 2022-2024 academic years at a Vietnamese university. Each submission carries per-test-case pass/fail outcomes (3,888 test cases in total), timestamps, source code, and admission-track labels. It is released upon request under a departmental license restricted to educational research purposes.
Which model best predicts a student's next coding attempt?
In the paper's benchmark, an adapted Recurrent State Space Model (RSSM) with discrete latent variables achieves the highest AUC on three of the four course offerings — for example, 0.831 versus 0.813 for the best parametric baseline (CIRT) on DSA HK231 — with its advantage concentrated at later attempts. On the remaining course (PF HK232), the TIKTOC model performs best.
Can LLMs simulate individual students' coding mistakes?
Not in this frozen, non-fine-tuned setting. The LLM predictor generates plausible code, but its accuracy signal comes mainly from problem difficulty: after removing per-problem averages, per-student Kendall tau falls toward zero, and pass rates when solving without student context correlate r=0.80 with those obtained with context but only r=0.40 with real students. Stronger coders, such as Claude Opus 4.6, were worse predictors than weaker ones like Qwen3-14B.
Is the CodeInsight dataset publicly downloadable?
No. The accompanying code is public on GitHub (aims-foundations/dynamic-irt), but the dataset itself requires an access request and is governed by a departmental license restricted to educational purposes. Approved requesters receive the source code of submissions, all test cases with per-test-case labels, grading scripts, problem statements, and the authors' train/test split indices.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-20 are image content; only their captions and in-text numeric references were available for review, so visual claims (e.g., heatmap patterns in Figure 4) rest on captions rather than inspection of the figures themselves.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.