Augur: A Synthetic Decision Lab for Rehearsing Reactions to Product and Policy Changes
Rahul Khedar, Mayank Malhotra, Avinash Karn
cs.AI · cs.CL · cs.MA
Quick verdict
Decision-simulation lab; on 50 real cases it argues frontier-vs-open gaps are mostly prompt artifacts, not capability — credible, honest, but small-n.
Summary
Augur is a five-stage pipeline that turns product- or policy-change documents into a typed knowledge graph, a graph-grounded persona population, an OASIS-style multi-agent simulation, and an auditable memo recommending one of five release actions (ship, revise, delay, segment, mitigate). The authors contribute Gold-50: fifty real episodes whose realized outcomes were adjudicated against the public record in a single author pass assisted by Claude Opus 4.8, balanced ten per class. The central result is methodological and negative. In a matched 2x2 ablation, appending a taxonomy-defining prompt block lifts every frontier model by +24 to +34 points (Claude Sonnet 5: 46 to 80%); under the under-specified prompt the offline LoRA-SFT Qwen3-32B significantly beats all three frontier models (exact McNemar, Holm-corrected), and under the fair prompt no significant difference is detectable. Holding weights fixed, one system swings from 0% to 73% across prompt envelopes (an 11-case subset); teacher-agreement climbs to 70% without accuracy moving; the full pipeline amplifies an over-doom bias (per-agent auto: 14%, below the 20% chance rate). Blind judges find synthetic reactions anticipate 67-90% of concerns the real public raised; a pre-registered simulation ablation averages +5.5pp, missing its own +6pp bar. Limitations: n=50 with single-pass author labels, model-family confounds in the pipeline comparison, documented gateway seed instability, and no public code or data release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On Gold-50, most of the measured frontier-vs-offline gap is a prompt artifact: appending a taxonomy-definition block lifts every frontier model +24 to +34 points with no change in weights.
- Under the under-specified prompt the offline Qwen3-32B LoRA-SFT significantly beats all three frontier models (paired McNemar, Holm-corrected); under the fair prompt no significant difference from any of them is detected at n=50.
- Agreement with the distillation teacher climbs from 10% to 70% across SFT rounds without outcome accuracy moving, as predicted by a formal argument that distillation collapses ensemble independence.
- The full five-stage pipeline amplifies an over-doom (mitigate) bias rather than improving verdicts, while the report-stage reaction signal alone is net-neutral-to-helpful (pre-registered +5.5pp, below the authors' +6pp bar).
- Blind cross-family judges find the synthetic reaction anticipates 67-90% of the concerns the real public actually raised.
Strengths
Matched 2x2 design with correct paired statistics
Section 5.3 and Table 3 vary one prompt block with weights, cases and scorer fixed; exact McNemar with Holm correction shows sft_v3 beating all three frontier models under the baseline prompt (p=0.039/0.012/0.002) and no detectable difference under the debias prompt, with an explicit disclaimer that non-significance is not equivalence.
Formal theory that maps onto observed failure modes
Prop 2 (distillation collapses judge independence) predicts the Section 5.7 result (teacher-agreement 10% to 70% with flat accuracy) and Prop 3 (schema-compliance ceiling) predicts Section 5.2; both are stated with proofs and matched to measurements.
Pre-registration honored even when inconvenient
Transparency about circularity and data hygiene
Full case ledger and label convention published in-paper
Weaknesses
Small sample undercuts several headline numbers
Gold-50 label quality is unverifiable
Model-family confound in the flagship pipeline comparison
Envelope advantage is baked into the comparison
No public code, data or artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Augur?
Augur, from PayPal AI researchers, builds a typed knowledge graph from product or policy change documents, synthesizes graph-grounded personas, runs an OASIS-style multi-agent social simulation, and emits an auditable memo recommending one of five release actions: ship, revise, delay, segment, or mitigate.
What is the Gold-50 benchmark?
Gold-50 is a benchmark of 50 real product and policy decisions (for example the Amazon Prime price increase, the D&D Open Game License reversal, Helldivers 2 account linking) whose actual outcomes are documented in the public record, balanced at 10 cases per action class; systems must predict the realized disposition from pre-decision documents only.
What is the paper's main finding?
Mostly a methodological negative: when the prompt defines the decision taxonomy, frontier models gain +24 to +34 points, and a fine-tuned offline Qwen3-32B becomes statistically indistinguishable from frontier models on these 50 cases. The paper argues leaderboard gaps were largely measurement artifacts, and that a full simulation pipeline amplified a pessimism bias rather than improving verdicts.
Does the simulation actually improve the verdict?
Not reliably. The complete pipeline scored at or below chance (one default routing at 14% against a 20% chance rate) with heavy over-prediction of 'mitigate', and the pre-registered matched ablation of the reaction signal averaged +5.5 points, just under the authors' own +6-point threshold for calling it helpful. The synthetic reactions themselves were rated faithful on content (67-90% concern recall by blind judges).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are extracted as axis labels and values only; visual content (loss curves, confusion heatmaps) could not be inspected beyond printed numbers.
- Minor extraction artifacts around tables (stray unparseable-count markers in Table 10; Table 5 header rendered inline) suggest slight parse loss near tables.
- Some appendix detail (exact frontier API settings, per-case rationales in realworld_gold_50.jsonl) is summarized rather than quoted in full.
- No public artifact URL appears in the text; availability rests on the sentence 'available from the authors', which could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.