GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Them

Summary

GradeTrap is a controlled evaluation of whether vision-language models obey instructions to ignore social annotations rendered inside images. Across 60 synthetic real-world trade-off scenarios, each model first answers five neutral renderings to establish its own preferred answer; items failing a 4-of-5 consistency rule are dropped, leaving 53 stable items for Gemini 3.5 Flash-Lite, 56 for GPT-5.6 Luna, and 52 for Claude Haiku 4.5, with 45 shared by all three. Each stable item is then shown under seven conditions with three fresh-chat repetitions: neutral, a student answer agreeing or conflicting with the model's preference, a generic second-answer card, and the same conflicting answer labeled as peer review, teacher review, or an official answer key — always with an explicit instruction to solve independently and ignore all annotations. On the 45-item common intersection, the conflicting student answer and the generic card raise conflicting-answer selection only from 2.2% to 5.2% and 5.4%; peer review shows no reliable effect, teacher review adds 6.9 points, and the official key adds 19.5 points (pooled 24.9%), with 95% item-cluster bootstrap intervals. Answers are scored by exact match plus an LLM judge audited against a human on 75 samples (98.5% agreement). Limitations include the unvalidated synthetic item bank, only three repetitions per cell with no decoding controls, no code or item-bank release, and the unquantified possibility that the official key is read as evidence rather than deference.