Who Should Grade My Work? Student Perspectives on Transparent AI-Assisted Writing Assessment in Higher Education
cs.AI
Quick verdict
Qualitative study of 13 Saudi students told ChatGPT graded their writing: feedback accepted, grading authority rejected; honest but small, single-site.
Summary
This qualitative pedagogical study examines how 13 male undergraduate computing students at a Saudi public university responded when explicitly told that ChatGPT, rather than their instructor, had evaluated their handwritten writing and assigned a score that counted toward grades. In-class handwritten submissions were scanned and graded by ChatGPT against a course rubric (clarity, organisation, sentence correctness, conciseness); students then wrote in-class reflections on the feedback, the score, and the appropriateness of AI evaluation. Inductive thematic analysis (Braun & Clarke), primary-coded by the author in NVivo with a confirmatory second pass by Claude Code (Opus 4.7), yielded four themes: universal perceived usefulness of feedback (13/13); awareness of AI's technical and contextual limitations (7 explicit); conditional trust separating feedback utility from evaluative authority (7 explicit); and reasoning about the human instructor's institutional and pedagogical role (5 explicit). The paper proposes the feedback-utility/evaluative-authority distinction as a way to reconcile algorithm-aversion and algorithm-appreciation findings, plus a three-principle framework (transparency, human mediation, reflective practice). Limitations are candidly stated: a small self-selected sample (13 of 19), a single-institution all-male cohort, the instructor-researcher dual role, unverified accuracy of ChatGPT's grading, and no causal isolation of transparency effects against the author's prior blinded study.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Students explicitly told that ChatGPT produced their feedback and score unanimously (13/13) judged the feedback useful for surface-level revision while most rejected AI as the appropriate final grader.
- Students treat feedback utility and evaluative authority as analytically separate judgments rather than opposite ends of a single approval scale.
- Transparency about AI involvement appeared to shift reflection from feedback quality toward authority, trust, and assessment legitimacy, though causality is not established.
- Participants articulated positive reasons for preserving human instructor authority: knowledge of individual students, dialogic assessment, institutional legitimacy of the university, and safeguarding against AI error.
- A three-principle framework — transparency, human mediation, reflective practice — is proposed for ethical GenAI integration in writing assessment.
Strengths
Well-aimed gap: the transparency condition
Most AI-feedback work studies usefulness or performance outcomes; this study discloses the AI evaluator's identity before collecting reflections (RQ1-3, §1.1, §2.7), isolating the interpretive layer — a genuinely under-examined design point.
Unusual dual transparency of method and data
Verbatim transcriptions preserve students' spelling and grammar errors (Figs 2 and 4), Table 1 maps themes per participant, and the OSF deposit (doi 10.17605/OSF.IO/S4HJ6) includes prompts, coding schedule, theme definitions, anonymised transcripts, and the Claude Code session log.
Crisp conceptual contribution
Candid, specific limitations
Disclosure of AI-assisted analysis with sequencing rationale
Weaknesses
No independent human second coder or reliability statistics
Table 1 internal inconsistency
No validity audit of the ChatGPT evaluation itself
Unaddressed tension between student objections and grade consequences
Reference-list hygiene
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This arXiv preprint (2609.05346, cs.AI) reports a qualitative, course-embedded pedagogical inquiry into a deliberately narrow question: what do students say when they *know* an AI graded their writing? Thirteen male second-year computing students (of 19 enrolled) in one Technical Communication section at a Saudi public university had handwritten in-class work scanned, scored, and commented on by ChatGPT against a course rubric; they were then told the source and wrote in-class reflections (§3.2-3.3). The paper claims four themes (universal perceived feedback usefulness; awareness of AI limitations; conditional trust; instructor-role reasoning), a conceptual contribution — the distinction between **feedback utility** and **evaluative authority** — and a three-principle integration framework (transparency, human mediation, reflective practice; Fig 7). To its credit, every generalisation is explicitly bounded ('within the bounded context and sample of this study', §7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did students in the study accept AI grading of their work?
Mostly no. All 13 students found ChatGPT's written feedback useful for spotting surface-level issues like grammar and organisation, but most rejected the idea of AI assigning final grades, insisting a human instructor should hold grading authority and check AI output before it counts.
What is the difference between 'feedback utility' and 'evaluative authority' in this paper?
Feedback utility is students' judgment that AI comments help them improve their writing; evaluative authority is their judgment about who should hold the power to decide grades. The paper's key finding is that students treat these as two separate judgments — they can accept the former while rejecting the latter in the same evaluation event.
How was the study conducted and how big was the sample?
In a technical communication course at a Saudi public university, 13 of 19 enrolled male second-year computing students completed handwritten in-class writing that was scanned and graded by ChatGPT against a course rubric. Students were told AI graded their work, then wrote short in-class reflections that were analysed with inductive thematic analysis.
Is the study's data publicly available?
Yes. The author deposited the reflective-task prompts, coding schedule, theme definitions, anonymised transcribed reflections, and the Claude Code confirmatory-analysis session log on OSF (doi 10.17605/OSF.IO/S4HJ6). Original handwritten pages are withheld because handwriting can identify people even without names.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-7 are images: only their captions appear in the extracted text, so the example ChatGPT feedback (Fig 3), coding map (Fig 5), conceptual model (Fig 6) and framework (Fig 7) were assessed via captions and surrounding prose only.
- Table 1's ●/◐ symbols came through as plain text; the totals-vs-matrix mismatch flagged in this review could partly reflect symbol misalignment in extraction, though the explicit (●) counts reconcile exactly (13, 7, 7, 5).
- No truncation detected: the extraction ends after the final reference (Zhang et al., 2024), which is alphabetically the last entry in the list.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.