greCAPTCHA: Assessing Understanding as Evidence of Research Authorship Under Generative AI
Justin Payan, Bálint Gyevnár, Atoosa Kasirzadeh, Nihar B. Shah
cs.DL · cs.AI · cs.HC
Quick verdict
Proposes LLM-generated proctored exams measuring authors' 'capacity to verify'; strong pilot (AUC 0.90, n=31) but lab-setting evidence overstates authorship detection.
Summary
The paper proposes greCAPTCHA, a proctored assessment testing whether manuscript authors actually understand their own work, motivated by generative-AI-enabled submissions whose authors may lack 'capacity to verify' — the knowledge and reasoning needed to critically evaluate their claimed contributions. A prototype uses GPT-5.6 Sol to generate eight manuscript-specific questions (planted-error multiple choice, unstated rationale, background knowledge, failure mode) with rubrics, and grades free-text responses automatically. In a counterbalanced within-subjects study, 31 Pittsburgh-area researchers took exams on one of their own papers and an assigned unfamiliar paper, then gave semi-structured interviews. Mean scores were 49.7 (own) versus 27.2 (unfamiliar); automated scores separated the conditions with AUC 0.90, rising to 0.934 when the weakly discriminative multiple-choice family (AUC 0.595) was excluded. Scores on in-field versus out-of-field unfamiliar papers did not differ significantly (p=0.437). Ordered-beta regressions with Holm correction confirm own-paper advantages across free-response families. Qualitative analysis surfaced concerns about rigid grading, time pressure, and unintended constructs such as typing speed and English fluency. Key limitations: the evidence is lab-based and non-adversarial — one prepared non-author nearly matched author scores — and all quantitative results rest on an LLM grader never validated against human raters.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A proctored, LLM-generated exam tailored to a manuscript and an author's declared contributions can elicit evidence of 'capacity to verify' — the knowledge and reasoning needed to critically assess one's claimed contributions.
- Automated scores separate participants' own from unfamiliar papers with AUC 0.90, rising to 0.934 when the non-discriminative multiple-choice family is excluded.
- Performance is not explained by general field familiarity: in-field and out-of-field unfamiliar papers show no significant score difference (p=0.437).
- Capacity to verify is necessary but not sufficient for authorship, so low scores alone must never categorically establish a policy violation.
- Researchers broadly support eventual deployment, contingent on contestable grading, human oversight, clearer expectations, and removal of multiple-choice items.
Strengths
Rigorous mixed-methods design
Counterbalanced 2x2 design (authorship within-subjects, field between-subjects; 8/8/8/7 participants), IRB approval (2026_333), four-person pilot, and a documented standardized procedure for unfamiliar-paper selection (Appendix B prompt plus human quality filter), yielding 496 scored responses across 62 tests.
Statistically appropriate analysis with honest robustness checks
Ordered-beta regression for [0,1]-bounded scores, Holm correction across four family contrasts, emmeans with 95% CIs, and an answered-only model (Tables 8-9) that honestly exposes the planted-error MC effect as a skipping artifact (p drops from 0.0122 to 0.7171).
Reports adverse results against its own interest
Careful anti-lookup question engineering
Artifacts released
Weaknesses
Measured construct is not deployed claim
LLM grader never validated
Unmodeled confounds in the own/unfamiliar contrast
Asymmetric contribution statements
Small, homogeneous sample for between-subjects claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: greCAPTCHA (arXiv 2609.20481)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is greCAPTCHA and how does it work?
greCAPTCHA is a proposed proctored exam for research authors. An author submits their manuscript plus a statement of their specific contributions; a system (in this paper, using the GPT-5.6 Sol model) automatically generates eight tailored questions — planted-error detection (multiple choice), unstated rationale, background knowledge, and failure mode — with grading rubrics. The author answers under proctored conditions, responses are auto-graded, and a report is produced for the requesting institution (journal, funder, employer). The goal is to measure whether the author genuinely understands their own paper, rather than to detect AI-generated text.
How accurate is greCAPTCHA at distinguishing authors from non-authors?
In the paper's study with 31 researchers, automated scores separated participants' own papers from unfamiliar papers with an AUC of 0.90 (mean scores 49.7 vs 27.2). Excluding the weak multiple-choice questions (AUC 0.595 alone), the AUC rose to 0.934. However, this was measured in a controlled lab setting with purpose-blind participants and no adversarial preparation; one prepared non-author nearly matched author scores, so the number should not be read as real-world authorship-detection accuracy.
Can someone cheat or game greCAPTCHA?
The study provides only anecdotal evidence in both directions. One participant who had an AI walk them through an unfamiliar paper beforehand scored 1.3 (near zero), suggesting the free-response questions resist superficial prep. But another participant scored 50.6 on an unfamiliar paper versus 53.1 on their own after just five minutes of preparation, scoring 76.0 on its failure-mode questions. The paper itself warns that motivated examinees using LLM assistance were not systematically tested, and that scores must never be used categorically to accuse someone.
Is greCAPTCHA deployed or available for use?
No. It exists as a research prototype evaluated with 31 participants; the paper explicitly states important changes are needed before any deployment. Participants recommended removing multiple-choice questions, stating expected answer detail clearly, making rubrics more flexible, adding contest/appeal mechanisms, and keeping human oversight over any consequential decisions. Code and study data are available at github.com/justinpayan/greCAPTCHA for research purposes.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several body sections were partially truncated in text extraction (parts of Sections 2.1, 4.3, 5, and 6.3); judgments for those gaps rely on the surrounding intact text.
- Figures (score distributions, ROC curves, dashboard screenshots) could not be visually analyzed; all numeric claims are taken from the text and appendix tables, which are complete.
- The GitHub repository (code, anonymized responses) is cited but was not independently inspected — no network access in this review.
- Some Appendix E/H2 robustness detail appears near a truncation boundary; specific robustness figures quoted are from the surviving table text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.