Towards Detecting AI-Assisted Responses in Online Surveys
Qizhou Wang, Bogdan Mamaev, Christopher Leckie
cs.CL · cs.CY
Quick verdict
Persona-grounded LLM agents evade existing survey detectors; SPABD's few-shot behavioural-cue aggregation restores +0.14 AUROC — credible but small-sample.
Summary
The paper introduces ASURRE, a benchmark pairing verified human responses from three pre-2020 surveys (Nature PhD 2019, Springer social-media 2017, OSMI mental-health 2017-18) with LLM completions under three usage strategies: per-question revision, full generation, and persona-grounded agentic completion mimicking entire respondents (Sonnet 4.6 via a Claude Code agent; gpt-oss-120b and Gemini-3-Flash backbones in ablation). Across six existing MGT detectors, full generation is detectable (AUROC 0.79-0.93), revision partially (0.55-0.74), while agentic completion sits near chance (0.50-0.65). The authors propose SPABD, a training-free, few-shot aggregator over four respondent-level behavioural cues (minimum answer length, length coefficient of variation, question-answer similarity, fields answered), normalized by median/MAD against K=5 verified human references and scored by sign-prior-corrected max-z. SPABD lifts mean AUROC from 0.61 to 0.75 across 12 agentic cells (11/12 wins), with TPR@5% FPR of 27.7% vs 12.0% for the best baseline. Ablations cover reference size, simulator backbone, aggregator alternatives (including a supervised 0.856 upper bound), field-count degradation, and reference contamination; a leave-one-survey-out check supports the fixed cue priors. Limitations: small test cells (90-504 respondents), reliance on multiple open-ended fields, English-only surveys, adversarial evasion out of scope, and the agentic generation pipeline withheld for safety.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 4
● low 0
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing MGT detectors detect full AI generation well (AUROC 0.79-0.93) but fall near chance (0.50-0.65) on persona-grounded agentic survey completion.
- Agentic completion leaves respondent-level behavioural traces — a minimum answer-length floor, uniform length CV, over-answering of optional fields, and slightly higher question-answer similarity — that survive per-answer humanisation.
- SPABD, a training-free few-shot median/MAD max-z aggregator over these four cues, improves mean AUROC from 0.61 to 0.75 and TPR@5% FPR from 12.0% to 27.7% across 12 agentic settings.
- Targeted prompting can defeat individual cues but not all simultaneously, except the combined adversarial mode on springer2017 where SPABD (0.56) falls below the best existing detector (0.59).
- The fixed cue priors are not overfit: a blind leave-one-survey-out fitted configuration scores 0.668 versus 0.751 for the unchanged four-cue SPABD.
Strengths
Realistic, graded threat model and respondent-level framing
Sec. 2 distinguishes per-question vs per-respondent detection and orders usage strategies by realism (revision, full generation, agentic completion), showing the most realistic threat is the least detectable — a problem formulation that existing per-answer MGT detectors structurally cannot address.
Provenance-safe benchmark construction
All three source surveys predate 2020 so human text is verifiably AI-free; eligibility filtering (>=80 chars of open text) and exact train/test counts are documented (Tab. 1, 6), and three strategies are crossed with three surveys and multiple LLM families.
Unusually honest and comprehensive ablations
Deployment-relevant metrics and fairness analysis
Traceable, review-compatible detector design
Weaknesses
Small, high-variance test cells
Cue and sign-prior provenance remains partly circular
Baseline aggregation possibly under-tuned
Benchmark reproducibility is compromised by withheld pipeline
Uneven external validity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ASURRE dataset?
ASURRE is a benchmark for detecting AI-assisted participation in online surveys. It pairs verified human responses from three pre-2020 public surveys (a Nature PhD student survey, a Springer Nature social-media survey, and the OSMI Mental Health in Tech survey) with LLM-produced completions under three usage strategies: rewriting a person's own answers, fully generating answers from the questions alone, and an LLM agent mimicking an entire respondent from a persona portrait.
Can current AI-text detectors catch LLM-written survey answers?
Partly. In this paper's evaluation, fully generated answers are detected well (AUROC 0.79-0.93), lightly AI-polished answers only partially (0.55-0.74), but answers produced by a persona-grounded LLM agent that completes the whole survey like a human respondent push existing detectors close to random guessing (0.50-0.65).
What is SPABD and how does it detect AI survey responses?
SPABD is a training-free detector that scores whole respondents rather than single answers. It computes four behavioural cues per respondent (shortest answer length, variability of answer lengths, similarity between each answer and its question, and how many fields were answered), standardizes them against as few as 5 verified human responses using median/MAD, and flags respondents whose most extreme cue deviates in the direction typical of LLM generation. It raised mean AUROC from 0.61 to 0.75 against agent-generated completions.
Do these AI detectors unfairly flag non-native English speakers?
The authors test this directly. Unlike perplexity-based detectors shown to be biased in prior work, SPABD uses no fluency or vocabulary signals, and in their analysis non-native respondents were flagged less often than native respondents on the two larger surveys. They still recommend treating flags as a signal for human review rather than grounds for automatic exclusion.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's heatmap values appear partially garbled in the extracted text, so per-cue AUROC numbers read from that block were not relied upon; judgments rest on the legible tables (3, 4, 5, 11-20).
- Figures 2-4 are described only via captions and axis labels; visual distributional claims (e.g., KDE separation) could not be independently verified.
- The claimed GitHub release (github.com/mike-qz-wang/ASURRE) was not inspected; code/data availability is taken from the paper's own statement.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.