Synthetic Data in Marketing Research: How to Evaluate and When to Trust
cs.AI
Quick verdict
Taxonomy of twin accuracy measures plus a no-ground-truth R² screen for LLM digital twins; useful but validated in-sample on one survey and one model.
Summary
The paper addresses polarization over LLM-based synthetic respondents in marketing research by reframing the question from 'whether' to 'when' they work. It contributes (1) a typology of synthetic data — ungrounded LLM responses, segment-level personas, individual-level digital twins; (2) a four-family taxonomy of accuracy measures showing that reported accuracy ranging from near-perfect (r=.95 in Dillion et al.) to near-chance largely reflects what is measured rather than method quality, since aggregate measures can look good even when twins carry no individual information (Twin-2K-500: 0.748 with full personas vs 0.734 with none); and (3) an ex-ante answerability diagnostic for 'forgotten questions' — items omitted from a fielded study — based on the out-of-bag R² of a random forest predicting twin outputs from twin inputs. In a leave-one-out test on 108 attitude questions from a nationally representative survey (N=3,063) using GPT-5.4-mini, grounding twins on same-category attitudes raised mean individual-level correlation from 0.08 (demographics only) to 0.57, and screening at R²>0.7 retained 43% of questions while cutting the share with correlation below 0.5 from 25.9% to 4.3%. Embedding similarity and the judgment of 16 practitioners were weaker or comparable alternatives. Limitations: one proprietary survey, one LLM, in-sample threshold selection, no confidence intervals, and no code release.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reported digital-twin accuracy ranging from near-perfect to near-chance largely reflects which accuracy family is measured, not method quality.
- Aggregate accuracy measures can look acceptable with zero respondent-level information: demographics-only twins achieved aggregate MAE 0.39 versus 0.48 in the informed condition while individual correlation was 0.08.
- The out-of-bag R² of a random forest predicting twin outputs from twin inputs screens answerable forgotten questions without ground truth: at R²>0.7, 43% of questions are retained and the share with twin-human correlation below 0.5 falls from 25.9% to 4.3%.
- Embedding similarity is a cheaper but weaker screen, and 16 experienced researchers' confidence tracks the statistical screen (r=0.87) and realized accuracy (r=0.77) on an 18-question subset.
- Segment-level and heterogeneity simulation remains unreliable without individual-level grounding, so human respondents remain necessary to the system.
Strengths
Resolves an apparent contradiction in the literature
The four-family taxonomy (Section 3.1) plus floor/ceiling corrections (Section 3.2) explains why Dillion et al. report r=.95 while Peng et al. report ~1-point individual-level gains; the decomposition of Twin-2K-500's 0.748 (full persona) vs 0.734 (no info) vs 0.746 (14 demographics) shows headline accuracy is available with no individual data.
Ex-ante, ground-truth-free diagnostic with a large empirical effect
The OOB-R² screen (Sections 4.2–4.3) uses only twin inputs and outputs; Table 2 shows monotone improvement — at R²>0.7, 46 of 108 questions retained, mean r=0.65, r<0.5 share cut from 25.9% to 4.3% — and the attainable-R² benchmark cleanly separates question easiness (0.542→0.575) from extraction efficiency (77%→86%).
Unusually honest reporting
Triangulated screening alternatives mapped to decisions
Weaknesses
Single model, single dataset, thin grounding
In-sample threshold selection
No uncertainty quantification or clustered inference
Possible dispersion confound in the screen
Untested extensions and closed materials
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This position-plus-methods paper reframes the polarized debate over LLM-based synthetic respondents in marketing research from 'whether they work' to 'when they work'. The authors — an academic (Netzer, Columbia) and a practitioner (Sambandam, TRC Insights) — make three claims: (1) synthetic data comes in three types (ungrounded LLM responses, segment personas, individual digital twins) that map to different decisions; (2) the wild spread of reported twin accuracy (Dillion et al.'s r=.95 vs Peng et al.'s near-chance individual-level gains) reflects what is measured, not method quality, and their four-family taxonomy (cross-question correspondence; question-level distributions; matched individual accuracy; within-person vs within-question correlations) is the diagnostic lens; (3) the 'forgotten question' — an item omitted from a fielded study — is a tractable twin-augmentation use case whose answerability can be screened ex ante, with no ground truth, via the out-of-bag R² of a random forest predicting twin outputs from twin inputs. Claims 2 and 3 are backed by new empirical work, not just argument.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'forgotten question' problem in this paper?
It is the common situation where a survey has already been fielded and only afterwards does the client realize an important question was never asked. Re-fielding is costly and slow, so the authors propose answering the missing question with LLM digital twins grounded in each original respondent's record — but only after screening whether the question is answerable from that record.
How accurate are LLM digital twins of survey respondents?
It depends on what you measure. Aggregate, cross-question metrics can exceed 90% even when the twin knows nothing about any individual (e.g., moral-judgment correlations of r=.95). Individual-level metrics on bounded scales sit around 72-76% raw accuracy but have high information-free floors — Twin-2K-500 twins with full personas scored 0.748 vs 0.734 with no personal data. Within-question correlations across people are typically modest (mean 0.57 in this paper's informed condition).
What is the R² answerability diagnostic and how well does it work?
For each candidate question, fit a random forest predicting each twin's answer from the data that was given to the twin. A high out-of-bag R² means the twin's answers actually vary with the respondent's record rather than reflecting generic priors. In the paper's test on 108 attitude questions, screening at R²>0.7 kept 43% of questions and cut the share with twin-human correlation below 0.5 from 25.9% to 4.3% — but the threshold was tuned on the same data, so treat it as an upper bound.
Can synthetic respondents replace human survey respondents?
According to this paper, not for recovering heterogeneity, segments, or genuinely new information — demographics-only twins had essentially zero individual-level validity (mean correlation 0.08). Synthetic data is positioned as useful for grounded single-question augmentation of an existing study, cheap pretest steps, and process improvements, with human respondents retained as the source of all preference information.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (R² vs correlation scatter) is referenced but not analyzable from the extracted text; only its caption and the footnote on the two outliers are available.
- The Web Appendix prompt is abbreviated (placeholders rather than full fielded text), so exact prompt engineering cannot be audited.
- Several key comparators (Brand, Israeli, and Ngwe 2026 — whose reference is explicitly incomplete — Park et al. 2026, Peng et al. 2026, Miller 2026) are 2026 journal or working papers whose figures are cited but not verifiable here.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.