Available but Unclaimed: An Empirical Study of Human-AI Synergy
Robin Welsch, Robin Welsch, Michelle Rausch, Pascal Knierim, Thomas Kosch, Jochen Kuhn, Albrecht Schmidt, Albrecht Schmidt, et al.
cs.HC · cs.AI
Quick verdict
535-participant study finds advice benefits track per-item LLM reliability and only ~half the per-item advantage reaches users; rigorous Bayesian evidence, observational caveats.
Summary
This between-subjects study had 535 Prolific participants solve a 40-item battery of matrix reasoning, mental rotation, syllogisms, and letter-string analogies either unaided (N=187) or with mandatory consultation of one of four LLMs (GPT-5.6-Luna N=179, Claude Opus 4.8 N=61, Gemini 3.6 Flash N=52, Kimi K3 N=56). Each model also answered every item 100 times under matched elicitation, giving per-item reliability. Assisted groups beat the unaided group by .200 (about 8/40 items) but did not exceed their assistant; item-level realized synergy was negative (-.062), resolving positive in only 42 of 160 item-assistant cells. Benefits rose with per-item LLM competence, and about half of the LLM's per-item accuracy advantage passed through to assisted accuracy. Deference followed task boundaries (.751-.947) and within-task competence. Post-advice confidence distinguished correct from incorrect answers less than unaided confidence, and advice adopters were equally confident whether advice was right (78.4 vs 78.0). Retrospective policies estimate up to +.061 (item-level selection) or +.141 (independent-error oracle) of 40. Limitations: between-group confounds from separate recruitments, mandatory consultation only, one interface and battery, and transcript labels from an LLM extractor whose advice-correctness validation is incomplete.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Assisted groups outperform the unaided group (+.200, about 8/40 items) but do not clearly exceed their assistant, and item-wise realized synergy never resolves positive (pooled S = -.062).
- The benefit of assistance rises with item-level LLM competence: over a .25-to-.75 competence contrast, predicted assisted accuracy increases by .263, roughly half the competence change (Table 3, Table 5).
- Deference is selective only coarsely: it varies by task (.751-.947) and increases with within-task competence, but participants followed incorrect advice on .743 of such trials.
- Post-advice confidence separates correct from wrong submissions less than unaided confidence in every assisted group (Table 6), and adopters were equally confident whether the advice was right (78.4 vs 78.0).
- Retrospective reference policies estimate unclaimed gains of up to about six items in forty (item-level selection +.061; independent-error oracle +.141).
Strengths
Scale and matched LLM-alone reference
535 participants across five groups (§3.1) plus a 100-run per-item chat-parity benchmark (§3.5) with matched API settings gives item-level LLM competence with binomial SE ≤.05, directly on the study's own 40 items rather than a public leaderboard.
Statistical transparency
Bayesian hierarchical fits with WAMBS checks (R̂ ≤1.003, bulk ESS ≥1,663, divergences ≤0.49%, prior sensitivity moves ≤.015; §3.7), plus a two-fold held-out policy validation (footnote 4: task .035 [.017,.052], item .058 [.038,.074]) that anticipates the shared-data selection objection.
Honest null and precise vocabulary
Clear behavioral discrimination results
Careful instrument engineering
Weaknesses
Between-group design with separate recruitments
Extractor-dependent transcript labels
Confidence claims are post-advice only
Policy gains are retrospective with a strong oracle
Narrow external validity surface
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper asks a sharper question than the usual "does AI help?": where, at item level, does consulting an LLM pay, and how much of the model's per-item accuracy advantage reaches the user? Five groups of Prolific participants (N=535; unaided 187, GPT-5.6-Luna 179, Claude Opus 4.8 61, Gemini 3.6 Flash 52, Kimi K3 56) worked the same 40-item generated battery (matrix reasoning, mental rotation, syllogisms, letter-string analogies; §3.2), assisted groups under mandatory per-item consultation. Each assistant was benchmarked alone, 100 runs per item, under chat-parity elicitation (§3.5, 15,968 replies). The headline claims: benefits rise with item-level LLM competence (Table 3); realized synergy is never resolved positive (Table 2, pooled S = -.062 [-.118, -.005]); about half of the LLM's per-item advantage passes through to assisted accuracy; and deference/confidence analyses reveal coarse selectivity (.908 deference to correct vs .743 to incorrect advice, Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did human-AI teams beat the AI alone in this study?
No. Assisted participants beat the unaided group by .200 (about 8 of 40 reasoning items), but teams did not clearly exceed the assistant's own accuracy (difference -.006 with an interval spanning zero). Item-by-item realized synergy averaged -.062 and was positive in only 42 of 160 item-assistant cells.
How much of the AI's accuracy advantage actually reached the human users?
In the study's reference comparison, roughly half. Over a per-item competence contrast from .25 to .75, predicted assisted accuracy rose by .263, about .526 of the competence change. How much arrived differed across the four models, and retrospective policies suggest about six more items in forty were attainable with more selective advice use.
Did consulting an AI make people less able to judge their own answers?
The study measured confidence only after consulting the model. Assisted groups' confidence separated correct from incorrect answers less than the unaided group's did (gaps of .053-.062 vs .102), and people who adopted the AI's advice were equally confident whether it was right (78.4 vs 78.0). Whether this reflects a true metacognitive cost of assistance is untested because no pre-advice confidence was collected.
What were the study's main limitations?
Groups were recruited via separate postings (between-group confounds), consultation was mandatory rather than optional, there was one interface and one 40-item generated battery, transcript labels came from an LLM extractor with incomplete validation, and the estimated policy gains are retrospective rather than demonstrated with deployable designs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text has mid-sentence breaks (§3.1 opening, end of §5.1, and parts of Appendices B/C), so a few sub-analyses were not available to this review.
- Figures (1-10) are present only as captions; plotted detail could not be verified against the reported numbers.
- The supplementary notebook with full specifications and sensitivity analyses is referenced but not contained in the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.