Stable and Faithful Explanations for Knowledge Tracing
Praveena Padi, Arun Morampudi, Ujval Sai Gopal Irrinki, Pradeep Kumar Dolabehera Kakitapelli
cs.LG
Quick verdict
Validates TreeSHAP explanation stability and faithfulness for knowledge tracing with unusual rigor; limited by one platform and untuned deep baselines.
Summary
This study asks whether a gradient-boosted model with post-hoc TreeSHAP explanations can match deep knowledge-tracing architectures while producing feature attributions whose stability and faithfulness are measured rather than assumed. The authors engineer thirteen behavioral features from ASSISTments 2009 and 2012, rebuild the 2009 corpus to remove duplicate multi-skill rows whose correctness labels leaked into history features, and compare XGBoost against DKT, SAKT, AKT, and SimpleKT under an information-matched protocol. XGBoost reaches AUC 0.786 (rebuilt 2009) and 0.777 (2012), 0.775/0.771 without current-response latency; restricted to identifier-and-correctness information it performs like DKT (0.717 vs 0.720; 0.697 vs 0.700), locating the accuracy gap in supplied information rather than model family. Feature rankings are stable across folds, seeds, and both TreeSHAP conditioning schemes (Spearman 0.989-1.000), and removing top-ranked features hurts retrained AUC more than random removal, though split-gain and permutation rankings perform comparably. Notably, the manuscript transparently withdraws earlier claims and documents two leakage defects caught by the faithfulness checks. Main limitations: both datasets come from one platform, deep baselines are untuned and self-implemented, folds overlap so stability is measured within one population, local per-student explanation validity is untested, and no paired significance testing supports the equivalence claims.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under an information-matched protocol, XGBoost restricted to identifier-and-correctness-derived features performs the same as DKT and SAKT (0.717 vs 0.720 on 2009; 0.697 vs 0.700 on 2012), so the accuracy gap between model families is due to the information supplied, not the family.
- The global TreeSHAP feature-importance ranking is highly consistent across cross-validation folds, random seeds, and both TreeSHAP conditioning schemes (Spearman 0.989-1.000) on both datasets.
- Removing the features TreeSHAP ranks highest and retraining degrades held-out AUC more than random removal at every k tested, but split-gain and permutation rankings behave comparably, so TreeSHAP is competitive rather than uniquely faithful.
- The uncorrected ASSISTments 2009 skill-builder release leaks current correctness labels into history features through duplicated multi-skill rows, and rebuilding the corpus changed both AUC and explanation rankings.
- These results establish model reliance under the stated ablation procedure, not causation, and are scoped to unseen students from the same platform and cohort.
Strengths
Information-matched reciprocal control cleanly attributes the accuracy gap
Section 3.1 and Table 5: restricting XGBoost to features derivable from the identifier-and-correctness stream (0.717 on 2009, 0.697 on 2012) matches DKT (0.720, 0.700), while enriching DKT with all behavioral features moves it only to 0.708/0.723 - isolating information supplied from model family and justifying withdrawal of the earlier 7.7-9.7-point advantage claim.
Quantified data-integrity contribution on ASSISTments 2009
Section 2.2 documents that duplicate multi-skill rows leak the current label into history features, with a clean diagnostic (univariate AUC 0.965 on duplicate copies vs 0.674 on first copies), and the rebuilt corpus (241,212 interactions, 2,062 students) aligns with pyKT and Xiong et al. conventions; all downstream results were recomputed.
Faithfulness testing against non-random competing rankings
Exemplary statistical self-correction
High reproducibility
Weaknesses
No paired or equivalence testing behind the central parity claim
Untuned, self-implemented deep baselines
Stability measured on overlapping folds within one population
External validity confined to one platform
Local explanations illustrated but unvalidated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a validation study, not a new model: it asks whether an XGBoost model with post-hoc TreeSHAP attributions can (RQ1) match deep knowledge-tracing architectures, (RQ2) yield feature-importance rankings stable across folds, seeds and TreeSHAP conditioning schemes, and (RQ3) pass a retraining-based faithfulness test. The central empirical claims are narrow and mostly earned: (1) with matched information, XGBoost performs like DKT (0.717 vs 0.720 on 2009; 0.697 vs 0.700 on 2012, Table 5); (2) rankings are near-identical across resampling (Spearman 0.989-1.000, Table 10); (3) top-ranked-feature removal degrades AUC more than random removal at every k (Table 11), but split-gain and permutation rankings behave comparably (Table 14). The authors explicitly withdraw earlier claims (a 7.7-9.7-point accuracy advantage; cross-dataset ranking divergence; monotonicity as redundancy evidence) and document two leakage defects - duplicated multi-skill rows on 2009 and leave-one-out target encoding - that the faithfulness checks themselves exposed.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
That an XGBoost model explained with TreeSHAP matches deep knowledge-tracing models when both receive the same information, and its global feature-importance ranking is stable across resampling and faithful under remove-and-retrain testing - though alternative importance measures (split gain, permutation importance) perform just as well under the same test.
Why did the authors rebuild the ASSISTments 2009 dataset?
The uncorrected skill-builder release writes each multi-skill interaction as one row per skill, and those duplicate rows share a single correctness label, which leaks the current answer into the history features. The paper shows duplicates separate correctness at AUC 0.965 versus 0.674 for genuine rows, rebuilds the corpus to 241,212 clean interactions, and recomputes all results, in line with prior work by Xiong et al. and the pyKT benchmark.
Is XGBoost better than deep learning for knowledge tracing?
Not necessarily. When restricted to the same information deep models receive (student/item identifiers and correctness only), XGBoost performs about the same as DKT (0.717 vs 0.720; 0.697 vs 0.700 AUC). Its higher overall AUC (0.786/0.777) comes from extra behavioral features such as item difficulty and timing, not from the model family itself.
Are the SHAP explanations in this paper reliable?
The global feature rankings were highly stable across folds, random seeds, and both TreeSHAP conditioning schemes (Spearman 0.989-1.000), and removing top-ranked features hurt retrained performance more than random removal. However, split-gain and permutation rankings performed comparably, individual students' local explanations were not validated, and importance reflects model reliance, not causation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-8 (signed SHAP values, dependence plots, learning curves, local case-study plots) are referenced but not analyzable from the extracted text, so directional and visual claims could not be independently checked.
- The abstract is truncated in the extraction; the full body through the references section is present and internally consistent.
- The review is based on the manuscript text alone; the released code repository and Zenodo archive were not inspected.
- Table 3-4 model-schema formatting is flattened in the extraction but legible enough to confirm input schemas.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.