Seeing Is Not Perceiving: When Synthetic Consumers Can and Cannot Pretest Visual Marketing
cs.AI · cs.CY · econ.GN
Quick verdict
Claims LLM synthetic consumers see but misperceive visual marketing cues; preregistered, careful stress test; steering partly = benchmark answer injection.
Summary
The paper stress-tests whether generative-AI 'synthetic consumers' can pretest visual marketing assets by re-running six canonical, preregistered experiments (logo descriptiveness, color saturation, shape, symmetry, and stability effects from JCR/JMR studies) with LLM respondents in a 2x2 design: GPT-4o-mini vs. GPT-5.4-mini, plain text vs. JSON input. All configurations passed manipulation checks (seeing), yet none reproduced more than two of six human effects at zero-shot (perceiving), including one significant reversal (4o-mini, Study 3 comfort, p=.012). Topic modeling of rationales traces failures to missing or competing semantic associations. In-context learning with conceptual (abstract) or empirical (statistics) evidence steers means toward human benchmarks - empirical evidence is more effective, especially for the newer model, which reproduces 6 of 7 cells - but recovered effect sizes stay attenuated, and no configuration reproduces even half of the human response spread (all zero-shot SD ratios <.50), understating consumer heterogeneity. The authors package this into a Calibrate-Intervene-Deploy governance protocol with a case study on novel logos. Limitations include two mini-tier models from one vendor, only six static-image studies, benchmark statistics injected verbatim into prompts (steering success is partly answer provision), and no multiple-comparison correction across many tested cells.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- All tested configurations pass visual manipulation checks yet none reproduces more than two of six human visual-marketing effects at zero-shot, with one significant reversal - seeing is not perceiving.
- In-context evidence steers mean responses toward human benchmarks, with success depending more on model generation than input format (with empirical evidence: 5.4-mini 6/7 cells vs. 4o-mini 3/7).
- Even successful steering recovers less than half the human response spread (all zero-shot SD ratios <.50), so synthetic consumers remain unreliable for segmentation or heterogeneity questions.
- A Calibrate-Intervene-Deploy governance protocol delineates when synthetic pretesting can screen creatives and when human panels remain necessary.
Strengths
Fully preregistered factorial design
Web Appendix A lists 21 aspredicted.org registrations covering every study x model x format x intervention cell plus robustness checks - unusual transparency for LLM-subject research.
Diagnosis beyond binary success
Section 4.1 and Table 6 combine PROCESS mediation with BERTopic rationale analysis, separating main-effect reproduction from the underlying pathway (e.g., 5.4-mini reproduces Study 5's size effect but the attention->size link fails, beta=.022, CI [-.169, .226]).
Distribution-level evaluation
Robustness battery
Deployment-ready governance artifact
Weaknesses
Narrow model and stimulus scope
Empirical-evidence condition is answer injection
No multiplicity control
Degenerate and single-run cells
Contamination cannot be excluded
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether 'synthetic consumers' - LLM agents prompted to answer as human respondents - can validly pretest visual marketing assets. Six canonical, highly cited effects form the benchmark: logo descriptiveness->authenticity (Luffarelli et al. 2019, JMR), saturation->temporal distance (Lee et al. 2017, JCR), logo shape->comfort/durability (Jiang et al. 2016, JCR), asymmetry->excitement (Luffarelli et al. 2019, JMR), saturation->size (Hagtvedt & Brasel 2017, JCR), and logo stability x category->utility (Rahinel & Nelson 2016, JCR). Claims: (1) all configurations see (100% manipulation-check pass) but none perceives reliably (<=2/6 effects at zero-shot, one significant reversal); (2) in-context evidence steers means toward the benchmark, gated by model generation more than input format; (3) steering cannot restore heterogeneity (zero-shot SD ratios all <.50); (4) the Calibrate-Intervene-Deploy protocol operationalizes when to trust the agent. The abstract's quantifiers match the tables.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a 'synthetic consumer' in marketing research?
An AI agent - here a GPT model prompted with a demographic persona - that answers survey or experiment questions as a human respondent would, letting firms pretest logos, packaging, or ads cheaply instead of recruiting a human panel. This paper tests whether that substitution is valid for visual assets.
Did the LLMs in this study actually see the images correctly?
Yes. Every configuration passed 100% of the visual manipulation checks (e.g., correctly rating angular logos as more angular). The failures were in consumer meaning: models saw the visual feature but missed or reversed what it signals to humans, such as circular logos signaling comfort.
Can prompt engineering fix LLMs' visual judgment for ad pretesting?
Only partially. Injecting the original study's theory or - more effectively - its statistics into the prompt pulled average AI ratings toward human results (the newer model matched 6 of 7 effects), but responses stayed far too uniform: no configuration recovered even half of human response variability, so segmentation decisions remain off-limits.
What is the Calibrate-Intervene-Deploy protocol?
A three-step AI governance rule from the paper: (1) Calibrate - validate the AI configuration against a known human benchmark at the original sample size; (2) Intervene - if it fails, steer with in-context evidence; (3) Deploy - use the agent only for average-effect screening; if calibration or steering fails, or the question involves consumer heterogeneity, use a human panel.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text extraction truncates Table 2's lower rows, part of Section 3.2/6.2, and parts of Tables E3; judgments rest on the available text.
- Figures 3-12 are available only as captions with in-text numbers; visual inspection of distributions was not possible.
- Some robustness material (JSON style variants, cell-level SD values) is 'available upon request' and could not be verified.
- Model naming ('GPT-5.4-mini', March 2026 release) and future-dated citations are taken as stated in the manuscript.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.