From Narrative to Auditable Forecasts: A Structured Scaffold for Agentic Forecasting
Yuanpu Cao, Yongkang Du, Yurui Chang, Lu Lin, Jinghui Chen
cs.CL · cs.LG
Quick verdict
AuditForecast decomposes LLM forecasts into a model base rate plus odds-space factor updates; wins across live benchmarks, but no significance testing.
Summary
The paper identifies 'narrative aggregation' as a structural weakness of LLM-agent forecasting: agents weigh evidence in prose and attach a probability without an explicit derivation. AuditForecast imposes a structured decomposition: the agent commits to a quantitative baseline model (e.g., a zero-drift lognormal for stock-threshold questions), retrieves data guided by the model's input requirements to compute a base probability p0, then identifies situational factors outside p0's scope, validates each for scope and conditional independence, and combines them with p0 via likelihood-ratio multipliers in odds space. Evaluated live on Polymarket, Kalshi (via ProphetArena), and FutureX with GPT-5.4, GPT-5.4-mini, and Gemini-3-flash (831 and 709 questions), AuditForecast consistently lowers Brier scores versus Think&Search (e.g., 0.1225 vs 0.1342 Polymarket overall), surpasses market-implied references on Polymarket for all three backbones, beats o3/o4-mini/smolagents deep-research agents at lower cost on 142 May-2026 questions, and outperforms Chronos-2/TimesFM on 369 value-threshold questions. Ablations (n=37) support model-guided retrieval, scope/independence checks, and odds-space aggregation; Section 5's audit finds 73% of factor directions correct, 66% well-calibrated magnitudes with an overstatement asymmetry, and 26% missed obtainable factors. Main limitations: no statistical significance testing or confidence intervals despite small subsamples (FutureX n=49/67, ablations n=37), differing question pools across backbones, and LLM-judge-based auditing sharing the model family with the audited agent. Code and prompts are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing agentic forecasting collapses into 'narrative aggregation' lacking an explicit evidence-to-probability pathway; AuditForecast replaces it with a schema-enforced decomposition of a model-derived base probability plus odds-space likelihood-ratio factor updates.
- Across three live benchmarks and three frontier backbones (831/709 questions), AuditForecast consistently lowers Brier scores versus Think&Search, e.g., 0.1225 vs 0.1342 on Polymarket overall with GPT-5.4.
- AuditForecast surpasses market-implied references on Polymarket for all three backbones (on Kalshi only GPT-5.4 does) and is the unique Pareto-dominant method against o3/o4-mini/smolagents deep-research agents on 142 May-2026 questions.
- The scaffold's components — model-guided retrieval, scope/independence checking, odds-space aggregation — each contribute measurably per ablations on 37 Polymarket events (Table 5: full pipeline 0.0764 vs 0.0807-0.0824 for variants).
- The auditable report enables post hoc audits revealing 73% correct factor directions, an overstatement asymmetry (28% overstated vs 6% understated), and 26% missed obtainable factors.
Strengths
Principled, executable decomposition
Bayesian odds-space update (Sec. 3.2, App. A) enforced at schema level (Pydantic, App. B.2); worked examples are arithmetically verifiable end-to-end (Fig. 2: 0.448×0.85×0.45×1.20 → p=0.171; App. C AAPL: 0.308×0.85×0.78×1.12 → p=0.186).
Unusually broad live evaluation with strong references
Three benchmarks (Polymarket/Kalshi via ProphetArena, FutureX), three backbones, market-implied reference, three deep-research agents with cost/Pareto analysis (Fig. 5, n=142), two TSFMs (Table 3, n=369), and a generic-structured-scaffold control (D.4) that isolates the decomposition from mere JSON output (GPT-5.4: 0.1151/0.1462 vs 0.1241/0.1596 on Polymarket/Kalshi).
Component ablations plus mechanism checks
Auditing yields novel empirical findings
Effort- and size-robustness results
Weaknesses
No uncertainty quantification anywhere
Likelihood ratios are elicited, not estimated
Judge circularity in Section 5 audits
Non-identical evaluation sets across backbones
Illustrative example internally inconsistent
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: AuditForecast (arXiv 2609.05905)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does AuditForecast solve in LLM forecasting?
LLM agents that forecast real events typically retrieve evidence and then state a probability in prose, with no explicit path from evidence to number — the paper calls this 'narrative aggregation'. AuditForecast forces a structured pipeline: a quantitative base probability (e.g., from a zero-drift lognormal price model), then independent situational factors applied as likelihood-ratio multipliers in odds space, so every step of the derivation is recorded and can be audited after the event resolves.
Does AuditForecast beat prediction markets?
Partially. On Polymarket, all three tested backbones (GPT-5.4, GPT-5.4-mini, Gemini-3-flash) achieve lower Brier scores than the market-implied reference at forecast time. On Kalshi, only GPT-5.4 does (0.0870 vs 0.1060); GPT-5.4-mini (0.1074) and Gemini-3-flash (0.1284) trail their market references. The abstract accordingly claims wins 'in several settings', not universally.
Is the code and data available for AuditForecast?
Code, system prompts, and the structured output schema are released at github.com/CaoYuanpu/AuditForecast. Because evaluation questions are live (April-May 2026, resolving within a week), exact benchmark runs cannot be repeated; the paper does not mention releasing a frozen snapshot of the questions, forecasts, and resolution labels, so third parties cannot directly recompute the reported Brier scores.
What are the main weaknesses of the AuditForecast paper?
Three stand out: (1) no statistical significance tests or confidence intervals anywhere, with several key comparisons on small samples (ablations n=37, FutureX n=49/67, deep-research comparison n=142); (2) evaluation pools differ across backbones (831 vs 709 questions at different times), confounding cross-model comparisons; (3) the Section 5 audits rely on LLM judges from the same model family as the audited agent, with human validation mentioned but not quantified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 1's sub-column headers are jumbled in the extraction; per-category numbers cited in this review assume the natural column order (Polymarket: Stocks/Crypto/Sports/Overall, then Kalshi: Finance/Entertain./Sports/Overall). The Overall columns themselves are unambiguous.
- Axis labels and legends of Figures 3-5 are interleaved with body text in the extraction; only in-text descriptions of those figures were usable — no plotted values were cited from them.
- Figure 1 and 2 embedded text is partially garbled/interleaved; the worked-example numbers cited here come from the legible report panels and were checked for internal arithmetic consistency.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.