Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents
Liqin Ye, Haorui Wang, Fardin Ahmed, Rongzhi Zhang, Yuan He, Ziyuan Lin, Yanbin Yin, Jing Peng, et al.
cs.AI · cs.LG
Quick verdict
工程构建良好的可回溯预测基准；研究结果（研究工具对全部 12 个模型都有帮助，但全部逊于市场）是可信的，部分源于架构设计。
Summary
这篇论文介绍了 Forecast-Dojo，一个用于 LLM 预测代理（forecasting agents）基准测试和训练的可回溯环境。它将 1,568 个已结算的 Polymarket 事件（按时间分为 1,338 个训练事件和 230 个评估事件）与 1,880 万篇有日期的 CC-News 文章相结合，因此每个事件都可以重放为固定的历史预测日期序列；代理只能检索该日期之前发布的信息，而实际结果仅用于评分，且对代理隐藏。在 230 个留出事件（held-out events）上评估了 12 个模型（每个日期进行四次 rollout），相对于没有工具的情况，使用研究工具降低了全部 12 个模型的 Brier 分数，并且随着事件的展开，预测效果有所提升，在出现更多新证据的步骤中收益最大；基于笔记本的证据捕获度量与改进相关（Spearman ρ=0.74）。尽管如此，所有模型仍然落后于历史市场预测（最佳 Brier 0.546 对比市场 0.498），并且在日期之间携带的信念笔记本（belief notebook）降低了研究成本（在专有模型中中位数降低了 24%），但未能持续提高质量。在记录的轨迹上进行监督微调（supervised fine-tuning）改进了较小的 Qwen 学生模型。局限性：日期选择偏向于新闻活动频繁的日子，证据索引依赖于自我报告，并且没有评估信息匹配的非 LLM 基准。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A replayable environment with aligned historical forecast dates enables controlled, immediately scorable comparison of LLM forecasting agents and yields reusable training trajectories.
- Research tools lower Brier scores for all 12 evaluated models, and forecasts improve as events unfold with the largest gains at steps where more newly dated evidence appears.
- Every evaluated model still trails historical market forecasts in Brier score, accuracy, and Information-α.
- A belief notebook carried between dates cuts research cost (median 24% among proprietary models) but does not consistently improve forecast quality.
- Supervised fine-tuning on Forecast-Dojo trajectories improves a smaller student model's held-out forecasting (Brier 0.924→0.749, accuracy 34.8%→42.8%).
Strengths
Aligned longitudinal replay design
Fixed, pre-specified forecast dates shared across all models (Section 3, Table 1) enable matched trajectory comparison that live benchmarks (ForecastBench, FutureX) and single-cutoff historical benchmarks cannot provide; outcome stays hidden from agents while remaining immediately scorable.
Temporal-integrity engineering is unusually thorough
Publication/modification metadata cascades with crawl-timestamp fallback (§A.1, only 13.1% of articles use crawl time), UTC-day filtering applied before ranking, full-text recheck at read time, plus a content-level leakage screen layered on the metadata boundary (§A.2). Leakage is treated as a first-class threat, not an afterthought.
Honest baseline framing and failure reporting
Memory analysis separates aggregate from paired effects
End-to-end training loop demonstrated
Weaknesses
Evidence-tracking finding is partly built into task construction
Self-reported, small-n evidence measures
LLM-in-the-loop filtering with no information-matched non-LLM baseline
SFT proof of concept lacks a control arm
Pooled Brier mixes event types on the unnormalized [0,2] scale
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Forecast-Dojo (arXiv 2609.28876) proposes a replayable benchmark-and-training environment for LLM forecasting agents. Each resolved Polymarket event is replayed as a fixed sequence of 3-10 dated forecast steps; at each step the agent researches a temporally filtered 18.8M-article CC-News corpus and reports probabilities, while the realized outcome stays hidden and is used only for scoring. Central claims: (1) aligned historical replay enables controlled, immediately scorable comparison plus reusable training trajectories; (2) research tools lower Brier for all 12 evaluated models; (3) improvement concentrates where new dated evidence appears; (4) every model trails historical market forecasts; (5) a belief notebook cuts cost but not consistently quality; (6) SFT on trajectories improves a student model. The claims are, for the most part, matched by the evidence presented.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Forecast-Dojo?
A replayable benchmark environment that reconstructs resolved Polymarket events as fixed sequences of dated forecast steps, pairing each step with a news archive restricted to articles published by that date, so that LLM forecasting agents can be compared under identical historical information conditions and trained on the logged interactions.
Do LLMs beat prediction markets in this benchmark?
No. All 12 evaluated models trail the historical market forecasts: the best model reaches Brier 0.546 versus the market's 0.498, best accuracy is 57.69% versus the market's 64.55%, and Information-α is negative for every model in every setting, meaning all have worse log scores than the market.
Does giving the forecasting agent memory between dates help?
It clearly cuts research cost — a median 24% reduction among the five proprietary models — but improves Brier for only 6 of 12 models, so the authors conclude that cost reduction, not forecast quality, is memory's clearest benefit in the evaluated protocols.
Can Forecast-Dojo data be used to train forecasting agents?
Yes, per the paper's proof of concept: trajectories collected from a large Qwen teacher model on the training split were used to supervised-fine-tune Qwen3-30B, improving held-out Brier from 0.924 to 0.749 and accuracy from 34.8% to 42.8% on 230 unseen events.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are only partially recoverable from extracted text (axis labels, legends); figure-level analyses were judged from captions, in-text numbers, and §B.6-B.7 definitions.
- The claim that code and data are publicly available is taken from the abstract; the repository itself was not verifiable from the provided text.
- Model names and knowledge cutoffs (Table 9, e.g., GPT-5.6 Sol, Opus 4.8) are reported as stated and not independently verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.