The Uneven Impact of Generative AI on Student Learning: Examining the Roles of Reliance, Evaluation Literacy, and Course Policy in AI-related Courses
Lydia Manikonda, Mei Si, Sirajam Munira, Oshani Seneviratne, Kristin Bennett
cs.AI · cs.ET · cs.LG
Quick verdict
Survey of 118 students: reliance type, not usage frequency, shapes perceived GenAI benefits—suggestive but cross-sectional, with uncorrected regression tests.
Summary
This cross-sectional survey study examines how generative AI (GenAI) relates to perceived learning among 118 undergraduate and graduate students in 12 AI-related courses at one R1 university. The authors distinguish cognitive reliance (using GenAI for recall, organization, evaluation, and problem-solving) from early reliance (consulting GenAI before independent thinking, search, or instructors) and measure evaluation literacy, task support, attitudinal change, and perceived instructor policy. Four analyses are combined: Ward clustering yields four usage-benefit profiles; FDR-corrected nonparametric comparisons show multi-tool and premium-access users report more support and benefit while single-tool users report more confidence without GenAI; four standardized HC3 regressions link academic benefit to early reliance and task support, positive impact to cognitive reliance, task support, reliability confidence, and policy, and negative impact to early reliance and attitudinal change, moderated by evaluation literacy; and a SHAP analysis associates perceived augmentation with 'improved learning' and stress or critical-thinking harm with its absence. Key limitations: outcomes are self-reported perceptions from a single institution; roughly 11 predictors for N=118 with uncorrected regression p-values and very high adjusted R-squared (.641-.771) suggesting construct overlap; N/A coded as scale midpoint; subjectively validated clusters; and no data or code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GenAI reliance is not unidimensional: cognitive reliance is associated with positive perceived impact (β=.495), whereas early reliance predicts both academic benefit (β=.301) and negative impact (β=.402).
- Access and tool breadth relate to experience: premium-access and multi-tool users report greater perceived support and benefit, while multi-tool users report lower confidence working without GenAI (rrb=+.25 for single-tool users).
- The association between early reliance and negative impact strengthens as evaluation literacy increases (interaction β=.307, p<.001), and is null at one SD below the mean of evaluation literacy.
- Perceived instructor policy matters: discouraged policies are associated with lower positive impact (β=-.138) and lower course-supported GenAI learning (β=-.186) relative to allowed-with-limits.
- Perceived learning benefits track GenAI augmenting students' cognitive capabilities, while stress and perceived critical-thinking harm are associated with lower perceived benefit (SHAP analysis).
Strengths
Useful two-dimensional reliance construct
Separating cognitive reliance (within-task use for organizing, evaluating, problem-solving) from early reliance (pre-thinking consultation) is a genuine conceptual contribution; empirically it pays off — early reliance predicts both academic benefit (β=.301) and negative impact (β=.402), while cognitive reliance predicts positive impact (β=.495).
Robustness-aware, multi-method pipeline
Clustering, FDR-corrected nonparametric comparisons with effect sizes (Table 2 reports rank-biserial up to -.53), HC3 regressions, and course-clustered SEs; the paper transparently reports which effects fail robustness (task support vs positive impact drops to p=.060 under clustering).
Counterintuitive moderation result
Full instrument disclosed
Policy-relevant access findings
Weaknesses
Construct overlap likely inflates regression fit
Inferential asymmetry: no correction in regressions
N/A coded as scale midpoint 2.5
Perception-only outcomes at one institution
Thin validation of cluster and ML components
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did this study measure actual learning outcomes or just student perceptions?
Only self-reported perceptions. The 118 students rated Likert-scale items about perceived benefit, positive and negative impact, and confidence; no grades, test scores, or behavioral measures of learning were collected, so all 'impact on learning' claims concern perceived learning in AI-related courses at one R1 university.
What is the difference between cognitive reliance and early reliance in this paper?
Cognitive reliance means using GenAI during substantive academic work — recalling, organizing, combining, or evaluating information and solving problems. Early reliance means turning to GenAI before attempting independent thinking, using traditional search, or asking an instructor. In the regressions, cognitive reliance tracked positive perceived impact (β=.495), while early reliance predicted both academic benefit (β=.301) and negative impact (β=.402).
Is higher evaluation literacy protective against negative effects of relying on AI?
Surprisingly, the opposite pattern appeared: the association between early reliance and negative impact was near zero for students low in evaluation literacy (simple slope b=.025, n.s. at -1 SD) and strongest for students high in evaluation literacy (b=.688 at +1 SD). The authors suggest critically aware students may notice harms, but the cross-sectional design cannot establish direction or causation.
Can I access the survey data or analysis code for this paper?
The paper publishes the full 60-item survey instrument in a supplementary table and notes IRB approval, but it states no public data or analysis-code availability, so independent re-analysis would currently require requesting the data directly from the authors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The embedded abstract/text is cut off mid-sentence near the end, so a small amount of closing content was unavailable (metadata abstract partially compensates).
- Figures 1-3 (cluster heatmap, interaction plot, SHAP summary) could not be inspected; assessments of them rely on captions and in-text narration.
- Cluster sizes and per-policy-group sample sizes are not reported in the extracted text, limiting assessment of subgroup power.
- Supplementary Table S1 appears complete in the extraction but could not be cross-checked against the original PDF.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.