Dutch Books for Language Models
Isaiah Andrews, Suproteem Sarkar
econ.GN · cs.AI · cs.CL · cs.LG
Quick verdict
Proposes label-free LLM forecast coherence test via de Finetti Dutch-book LPs; finds sizable incoherence; sound but single-domain, protocol-sensitive.
Summary
The paper turns de Finetti's Dutch-book theorem into a label-free coherence evaluation for language-model probabilistic forecasts. Given stated probabilities for a set of events with known logical structure (bins of variance-normalized stock returns, plus unions, complements, and cross-day/cross-asset conjunctions), a linear program computes the maximum profit an arbitrageur can guarantee per unit stake; by minimax duality this equals the sup-norm distance from the stated vector to the nearest coherent distribution, and is zero exactly under coherence. The authors elicit 365,100 forecasts from 15 models on 100 stock-days (CRSP returns with Refinitiv headlines, anchors Aug–Dec 2025). The open-weight workhorse GPT-OSS-120B shows mean arbitrage profit 0.00207 per unit stake (95% CI [0.00147, 0.00274]); profit varies roughly 100-fold across models (0.0021–0.199) while Brier scores span only 0.197–0.216, with Spearman rank correlation 0.914. Incoherence grows with joint events linking assets or days, and elicitation details matter enormously: asking all 14 events in one query nearly eliminates incoherence, while irrelevant contextual notes can raise it by an order of magnitude. Limitations: a single forecasting domain, elicitation experiments confined to the most coherent model, horizon comparisons confounded by prompt preambles, proprietary data, and no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Language model probabilistic forecasts over logically structured event sets are frequently incoherent, with mean arbitrage profit of 0.00207 per unit stake for GPT-OSS-120B and up to 0.199 for the least coherent of 15 models.
- Label-free coherence discriminates between models far more than outcome-based accuracy: arbitrage profit varies ~100-fold across models while Brier scores span only 0.197–0.216 (Spearman rank correlation 0.914).
- Incoherence increases when queried events embed richer logical dependencies, whether added as joint events or compared at fixed event-set size (Appendix D linked vs dispersed panels).
- Measured coherence is highly protocol-dependent: asking the full 14-event panel in one query nearly eliminates arbitrage profit, while irrelevant contextual notes can increase it by an order of magnitude.
- Coherence is likely improvable through training strategies that encourage computation across related events, with the Dutch-book LP usable as an exact set-level reward.
Strengths
Mathematically clean, label-free coherence measure with exact duality
LP (1) with the ‖b‖₁ ≤ 1 cap, plus the Appendix A minimax argument, yields ℓ∞(p) = min_π ‖Mπ − p‖∞ — an exact zero-coherence characterization that needs no realized outcomes; demonstrated on unresolved 2/5/10-year return forecasts (Appendix B).
Experimental hygiene above the field's norm
Anchor window (Aug 15–Dec 23, 2025) postdates GPT-OSS-120B's release to limit lookahead; four-decimal reporting bounds mechanical profit at 5×10⁻⁵; 95% of answers are identical across five passes, so incoherence is not response randomness; inference uses 20,000-replicate date-clustered bootstrap.
Well-identified contrasts, including a fixed-set-size design
Near-complete disclosure enabling replication
Honest self-flagging of confounds
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper operationalizes de Finetti's Dutch-book theorem as a label-free coherence metric for LLM probabilistic forecasts (§2). Given stated probabilities p for n events with known logical structure, LP (1) maximizes the worst-case profit of an arbitrageur betting at those prices under a unit gross-stake cap; Appendix A's minimax duality shows this equals min_π ‖Mπ − p‖∞, the sup-norm distance to the coherent simplex — zero exactly when p admits a probability extension. The empirical claims (§4) are that LLM forecasts are substantially incoherent, that incoherence grows with logical richness (§4.3, Appendix D), and that elicitation details move it by orders of magnitude (§4.4). Section 5 proposes training remedies (process reward models, group-level RL with the LP as exact set-level reward).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a Dutch book evaluation of an LLM?
A Dutch book is a set of bets that wins regardless of the outcome. Treating a model's stated probabilities as betting prices, the authors solve a linear program for the largest guaranteed profit an arbitrageur can extract per unit staked. By de Finetti's theorem (proved here via minimax duality), zero guaranteed profit holds exactly when the probabilities are coherent — i.e., consistent with some probability distribution over the underlying outcomes.
Does the coherence test require knowing the actual outcomes?
No — that is its key advantage over accuracy or calibration metrics. It uses only the stated probabilities plus the known logical relationships among the queried events, so it applies to unresolved or unobservable events. The paper demonstrates this on 2-, 5-, and 10-year-ahead stock-return forecasts, finding mean incoherence around 0.007–0.008 per unit stake.
Which models were tested and how incoherent were their forecasts?
Fifteen models including GPT-OSS-120B/20B, GPT-5.6 variants, Claude Opus/Sonnet/Fable 5, Gemini 3.5 Flash, DeepSeek V4 Flash, Llama-4 Maverick, Qwen3-235B, and Mistral Small 3.2. Mean arbitrage profit per unit stake ranged from 0.0021 (GPT-OSS-120B) to 0.199 (Mistral Small 3.2) — roughly a 100-fold spread — while Brier accuracy scores varied only between 0.197 and 0.216.
How could LLM forecast coherence be improved?
The authors show models are nearly perfectly coherent when all 14 related events are elicited in a single query, suggesting post-training that encourages reasoning across related events — e.g., process reward models or group-level RL using the Dutch-book LP as an exact set-level reward — could improve coherence for ordinary single-question deployments.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 are images in the PDF: their exact panel values are not in the extracted text, so magnitudes not quoted in prose (e.g., individual red-herring arm values in Fig 4D) rest on axis ranges and in-text claims only.
- No code or data release is mentioned anywhere in the provided text; if one exists in a part of the PDF not rendered as text, this review's codeAvailable=false would need revision.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.