Designing Against Deskilling: Metacognitive Feedback Reduces Cognitive Offloading to LLM Assistants
Sebastian Maier, Kai Schwabe, Manuel Schneider, Stefan Feuerriegel, Stefan Feuerriegel
cs.HC · cs.AI
Quick verdict
Preregistered 704-participant experiment showing metacognitive feedback cut LLM answer offloading and improved unaided test scores; sound, with minor caveats.
Summary
The paper reports a preregistered online experiment (N = 704, UK Prolific adults) testing whether interaction design can reduce cognitive offloading to LLM assistants without restricting access. Participants practiced ten fraction-arithmetic items with a GPT-OSS-120B assistant that gave complete answers only on explicit request, then took a six-item unaided test. An LLM classifier labeled the highest tier of help requested per item; interventions were metacognitive feedback generated after each item versus an effort-based reward (10/5/1 points by tier), in a 2x2-plus-control design. Metacognitive feedback significantly reduced answer offloading (OR = 0.47) and improved unaided performance (OR = 1.51); the reward changed neither primary outcome, instead reducing overall assistant use (OR = 0.39). Notably, AI access alone did not harm unaided performance versus control (H1 unsupported), which the authors attribute to friction from the explicit-request design. Limitations: short single-session format with no delayed retention test, modest baseline offloading, general Prolific adults rather than students, and LLM-generated feedback whose content varies; headline odds ratios also exceed what the reported condition means imply and deserve verification.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Metacognitive feedback about one's own LLM use reduces answer offloading (OR = 0.47) and improves subsequent unaided test performance (OR = 1.51).
- An effort-based reward scheme did not significantly affect either primary outcome, despite reducing overall assistant use (OR = 0.39).
- Mere access to a request-gated LLM assistant did not impair unaided performance relative to a no-AI control (H1 unsupported).
- The risk of deskilling lies in how much cognitive work users hand over (answer offloading), not in assistant availability per se.
- Designers should make AI-use consequences visible, make complete-answer requests deliberate, and scaffold productive help-seeking rather than price it.
Strengths
Preregistered, adequately powered factorial design
AsPredicted preregistration before collection; a priori power analysis for d = 0.30 yielding 695 (recruited 720, analyzed 704); explicit exclusion rules, honesty check, IRB approval; manipulation check shows 68.6% of AI-only participants used the assistant and 39.4% offloaded at least once (Table 2).
Unusually transparent LLM pipeline validation
Assistant, offloading classifier, and feedback generator all frozen before launch; assistant prompt validated over three full passes on 55 synthetic messages; classifier achieved zero confirmed errors on 198 held-out dialogues (Section 4.2, Appendix C), following a Nature Human Behaviour LLM reporting checklist [20].
Careful outcome measurement and item construction
Mechanistic exploration beyond the headline effect
Reproducibility posture
Weaknesses
Reported effect sizes look larger than the condition descriptives imply
The deskilling premise fails to materialize in this setup
Mechanism not validated despite metacognitive branding
Sample power calculation tied to a hypothesis that came out null
External validity constraints
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether interaction design can reduce cognitive offloading to LLM assistants — and thereby protect skill development — without restricting what the assistant can do. Two interventions are compared in a preregistered 2×2-plus-control design (N = 704): metacognitive feedback generated after each learning item, and an effort-based reward (10/5/1 points by level of assistance requested). Outcomes are answer offloading (proportion of 9 learning items on which the complete answer was requested) and unaided performance (6-item test). The preregistered H1 — that LLM access itself harms unaided performance relative to a no-AI control — was *not* supported; the authors attribute this to their explicit-request design adding friction, citing prior evidence that small frictions suppress offloading. This is honest, but it reframes the contribution: the headline result concerns preventing decline in a regime where decline barely occurs, unlike the high-offloading setups of Bastani et al. [5] and Liu et al. [53] that motivate the work.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did the metacognitive feedback intervention show?
In a preregistered experiment with 704 participants, short AI-generated summaries of how each person had used the LLM assistant — shown after every practice problem — halved the odds of requesting the complete answer (OR = 0.47) and improved scores on a later test without the assistant (OR = 1.51), compared with having the same assistant and no feedback.
Did paying participants to use AI less work?
No. An effort-based reward that gave 10 points for solving with no or guided help, 5 for requesting an intermediate result, and 1 for requesting the full answer did not significantly change either answer offloading or unaided test performance. It did reduce how often participants consulted the assistant at all (OR = 0.39) and was subjectively more effortful.
Does using ChatGPT-style help while learning hurt your exam performance?
In this study, not by itself. Participants with access to an assistant that gave complete answers only when explicitly requested scored statistically indistinguishably on an unassisted test from participants who never had AI access (H1 was not supported). The authors argue the risk comes from how much of the work you hand over — requesting complete answers — rather than from having an assistant available.
Can these results be reproduced?
Largely. The authors released analysis code, anonymized data, prompts, and a deterministic item generator in an anonymized repository; the assistant and classifier ran deterministically (temperature 0) on an open-weight model. The main caveats are the LLM-generated feedback texts (temperature 0.7) and that the human-behavioral effect requires rerunning the experiment rather than a pure re-analysis.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text extraction truncated the middle of the results section (Table 3 confirmatory values are not fully visible), part of the Discussion, and Appendix B; headline ORs are taken from the abstract.
- Figures (Figures 1-5) could not be assessed from text; trajectory and composition claims rest on the textual summaries in Section 5.4.
- Condition means in Table 2 suggest smaller contrasts than the headline ORs; this may reflect interaction terms in the models or an extraction artifact and should be verified against the PDF.
- H1 was tested only against a request-gated assistant with modest offloading rates, so the null harm finding is specific to this design.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.