Data-Driven Persona-Conditioned Agents for A/B Test Simulation
Ziyad Benomar, Weronika Łajewska, Leonardo Perelli, Saab Mansour
cs.AI
Quick verdict
Data-driven personas let LLM agents predict A/B test directions at 0.75-0.90 accuracy; useful pre-screening tool, but inconsistent numbers hurt trust.
Summary
The paper proposes simulating A/B tests with LLM agents conditioned on data-driven personas that an LLM synthesizes from anonymized e-commerce behavioral signals, replacing the screenshot-derived synthetic personas of prior work (SimAB). Agents view control/treatment screenshots under four question formats and score variants; per-persona scores are aggregated into a predicted effect distribution and evaluated with sign-based metrics. On a proprietary benchmark of 40 e-commerce tests (20 CTR, 20 subscriptions), filtered to decisive effects, pairwise rating with Claude Sonnet 4.5 achieves 0.75 (CTR) and 0.80 (subscriptions) directional accuracy. Personas from public e-commerce data rival platform personas (0.90 vs 0.80 subscription accuracy); behaviorally deep personas beat representative ones only on CTR; subsampling 500 of 935 personas preserves accuracy within about one point, halving cost. Ablations show behavioral profiles and population diversity drive the gains: demographics-only collapses on subscriptions (0.30), and a single generic persona matches no-persona. Findings replicate qualitatively across Claude Haiku and Opus. Main limitations: the benchmark is proprietary and filtered toward unambiguous effects; numbers for identical configurations differ across main-text and appendix tables (e.g., 0.75 vs 0.70 CTR); statistics rely on uncorrected one-sided t-tests with n=20 per metric; and only Anthropic models were tested.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Pairwise rating is the most effective question format, reaching 0.75 (CTR) and 0.80 (subscriptions) directional accuracy on their 40-test benchmark (Table 1).
- Domain alignment of persona source data is the primary driver of quality; public e-commerce personas rival proprietary platform personas (0.90 vs 0.80 subscription accuracy, Table 2).
- Population diversity can fully compensate reduced per-persona behavioral depth on subscription tests, while depth wins on CTR accuracy (Table 3).
- Subsampling to 500 of 935 personas preserves accuracy within about one point, enabling roughly 2x inference cost reduction (Table 4).
- Behavioral persona conditioning lifts accuracy +25-30pp (CTR) and +15-20pp (subscriptions) over an unconditioned LLM, while a single generic persona adds nothing over no persona (Table 5).
Strengths
Personas grounded in real behavioral data, independent of test stimuli
Unlike SimAB's screenshot-generated synthetic personas (§2), personas here are LLM-synthesized from months of anonymized activity (§3.4, App. I.2-I.3), enabling clean questions about behavioral depth (RQ3) that prior work could not ask.
Informative ablation design with useful negative results
Table 5/11: demographics-only collapses subscription accuracy to 0.30 vs 0.80; a single generic shopper persona (0.40 CTR) does not beat no-persona (0.45), isolating population diversity — not persona framing — as the driver.
Multi-model robustness check
Reproducibility path on public data
Bias controls in the simulation loop
Weaknesses
Internal numerical inconsistencies for identical configurations
Benchmark selection toward favorable regimes
Statistical reporting below bar
Single-vendor confound
Overstated framing of the headline format finding
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.01038) proposes simulating e-commerce A/B tests with persona-conditioned LLM agents. Personas are synthesized by Claude Sonnet 4.5 from anonymized, aggregated behavioral signals (§3.4, App. I.2-I.3) rather than from screenshots (SimAB) or session replays (SimGym). Agents score control/treatment screenshots under four question formats (§3.3); scores aggregate into a normal effect distribution evaluated by sign-based metrics (Acc, SignOv, SignBC; §3.6). Four research questions structure the study: question format, persona source and domain alignment, depth-diversity trade-off, and subsampling. Central claims: pairwise rating is the best format (0.75 CTR / 0.80 subscriptions, Table 1); domain alignment dominates, with public e-commerce personas rivaling proprietary ones (0.90 vs 0.80 subscriptions, Table 2); demographic diversity compensates for shallow personas on subscriptions (Table 3); 500-of-935 subsampling costs ~1 point (Table 4); ablations attribute gains to behavioral profiles and population diversity (Table 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can LLM agents accurately predict A/B test outcomes?
On this paper's benchmark of 40 real e-commerce A/B tests, the best configuration — persona-conditioned Claude Sonnet 4.5 agents answering a pairwise rating question — predicts the winning direction 75-80% of the time (up to 90% on subscription tests when personas come from public e-commerce data). Accuracy is highest for large effects and drops near zero-effect cases, so the authors position the method as a pre-screening tool rather than a replacement for live experiments.
What are the data-driven personas in this paper?
Structured user profiles (inferred demographics, shopping habits, engagement patterns, behavioral narratives) that an LLM synthesizes from anonymized, aggregated behavioral records of real e-commerce users. Unlike prior work's personas invented from product screenshots, these are grounded in months of actual activity — though the demographics are still LLM-inferred, not self-reported, and individual personas are not meant to represent specific people.
Can I reproduce this A/B test simulation approach with public data?
Only partially. The 40-test benchmark and its ground-truth labels are proprietary and not released, and no code is published. Appendix E shows how to rebuild an evaluation from public preference datasets (MovieLens, Book Crossing, Jester): treat item pairs as control/treatment and rating differences as ground-truth effect. The paper reports only qualitative results there — accuracy rises monotonically with effect size — not headline numbers comparable to the main tables.
How much cheaper is simulating an A/B test than running one?
The paper claims a full simulation completes in hours via batched LLM inference at a fraction of a multi-week experiment's cost, and that subsampling 500 of 935 personas preserves accuracy within about one percentage point, halving simulation inference cost. However, no absolute cost figures comparing simulation against a live experiment are provided, and real experiments still involve engineering effort the simulation does not remove.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Mathematical notation is partially garbled by PDF extraction (the SignBC formula in §3.6 and App. C is missing square-root signs), so formula-level judgments should be treated cautiously.
- Appendix Figures 4-6 are referenced but their plotted values are only partially recoverable from axis labels in the extracted text; trend claims rely on the surrounding prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.