Detecting GPT-Assisted Writing Using Interpretable Stylometric Features

Summary

The paper asks whether a compact set of nine interpretable stylometric features, computed solely from submitted text, can separate GPT-assisted from independently authored student writing. It uses the IIITD-BU (Paraphrased) dataset: 90 IIIT Delhi students each answered two LLM-related prompts independently and then by manually retyping and paraphrasing ChatGPT output (copy-paste disabled), yielding 180 documents. From 250-word sliding windows with a 125-word stride (722 windows), the authors compute type-token ratio, hapax ratio, word entropy, non-stopword ratio, four part-of-speech ratios, and sentence-length variability. Training (72 participants) and test (18 participants) partitions are user-disjoint to prevent style leakage. Eight classifiers were tuned with Bayesian optimization under grouped repeated 10-fold cross-validation; Random Forest won validation (ROC-AUC 0.838) and, using median window-probability aggregation, reached ROC-AUC 0.870 and F1 0.842 on the 36 held-out documents, with a 22.2% false positive rate (4/18) and 11.1% false negative rate (2/18). SHAP analysis ranks hapax ratio, non-stopword ratio, noun ratio, adverb ratio, and TTR as most influential. A word-count-only classifier (AUC 0.68) and a moving-average TTR variant control for length confounds. Key limitations: one course, topic, and paraphrase-only assistance mode; a tiny test set (FPR CI 9.0-45.2%); and resampled splits showing mean false negative rates of 23.7-24.8%, far above the headline 11.1%. The authors explicitly advise against standalone use in misconduct decisions.