Competence-Gated Pooling of Language Models and Priors for Event Forecasting
Aditi Tiwari, Aashrith Bandaru, Heng Ji
cs.AI
Quick verdict
Credible study: outcome-estimated domain gates give modest, significant Brier gains over global pooling, defer on strong markets; stacker parity caps novelty.
Summary
The paper reframes hybrid event forecasting as a relative-competence problem: given an existing external forecast (market, crowd, or statistical prior), decide whether language-model forecasts add marginal value. Under Brier loss the authors characterize when model disagreement improves an external forecast (gain equals squared covariance of disagreement and external error, normalized by disagreement dispersion) and prove an exact identity (Theorem 1) for the gain of domain-specific over global pooling weights. They propose a competence gate: per-domain affine source weights estimated by ridge-constrained least squares from resolved training outcomes, shrunk toward a global weight, with isotonic recalibration. On 2,357 resolved ForecastBench questions across eight sources and five LLMs, the gate lowers Brier from 0.0771 to 0.0732 (p=0.001), beating global two-source and simplex pools; the identity's predicted gain (0.0035) is close to the realized gain (0.0027). Gains persist under leakage controls and against an ARIMA prior on the 172 structured questions (0.172→0.160), but the gate shows no significant gain on the 1,294-question live-market subset and is statistically indistinguishable from a domain-aware logistic stacker (p=0.53). Verbal confidence in four Qwen models fails to predict when the model beats the external source (AUCs 0.389–0.501), while outcome-estimated competence improves abstention. Limitations: small effect sizes, coarse source-as-domain labels, contamination risk, and self-assessment results confined to one model family.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under Brier loss, an LLM forecast improves an external forecast only when its disagreement correlates with the external forecast's errors, with gain E[de]²/A_k (Section 4, Eq. 9).
- Theorem 1 gives an exact identity for the gain of domain-specific over global affine pooling, and its training-fold estimate (0.0035) closely matches the realized held-out gain (0.0027).
- Across 2,357 resolved questions and five LLMs, the competence gate improves the main external baseline from 0.0771 to 0.0732 Brier (p=0.001) and significantly beats global two-source and simplex pools, with gains persisting under leakage controls and against an ARIMA prior on the pooled structured set and FRED.
- On the official ForecastBench live-market subset (1,294 questions) the gate yields no significant improvement and largely defers to the market.
- Verbal confidence in four Qwen models does not reliably predict when the model beats the external forecast (AUC 0.389–0.501), whereas outcome-estimated competence supports better abstention (risk-coverage area 0.019 vs 0.054).
Strengths
Exact theory-practice bridge
Theorem 1 gives an exact identity for the gain of domain-specific over global affine pooling, and the training-fold routing term (0.0035) is directly compared against the realized held-out gain over the global simplex pool (0.0027, Section 7) — a falsifiable consistency check that is rare in applied forecasting work.
Serious leakage discipline
Structured-source priors are built from training-fold outcomes only; the ARIMA prior (Eq. 16) refits at each forecast date using vintage data available at t_i; a leakage-controlled subset (2,103 questions resolving ≥ June 1, 2025) is evaluated separately; models are question-only and blind to the external forecast g, following the evaluation pitfalls of Paleka et al. (2025).
Honest negative results and narrow claims
Informative ablation set
Useful conceptual distinctions
Weaknesses
Small absolute effect sizes
No accuracy gain over the strongest baseline
Coarse domain definition
Self-assessment external validity is narrow
Contamination cannot be excluded
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a deployment decision in hybrid forecasting: given an existing external forecast (prediction market, crowd, or statistical prior), when should a language model's probability be used at all? The authors formalize this as *relative competence* — marginal Brier improvement beyond the external source — explicitly separated from standalone accuracy (Section 3, Eq. 2 vs Eq. 5). Three claims anchor the work: (1) under Brier loss, model disagreement d = f−g helps only when correlated with external error e = g−y, with gain E[de]²/A_k (Section 4, Eq. 9); (2) an exact identity (Theorem 1) quantifies the gain of domain-specific over optimal global affine pooling as Σ π_k A_k (λ*_k − λ*_glob)²; (3) a competence gate operationalizing this — per-domain ridge-constrained affine weights estimated from resolved outcomes, shrunk toward a global vector, with isotonic recalibration — yields significant improvement on 2,357 ForecastBench questions, while verbal confidence fails as a competence signal.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'relative competence' mean in this paper?
It is the marginal value of a language model's forecast beyond an already-available external forecast (a prediction market, crowd, or statistical prior), measured under Brier loss — as opposed to the model's standalone accuracy. A model can be worse than the market on its own yet still improve a pooled forecast if its disagreements correct the market's errors.
Does the proposed gate beat prediction markets?
No. On the official ForecastBench live-market subset (1,294 questions with genuine market probabilities) the gate shows no statistically significant improvement (p=0.42) and assigns little weight to the models, effectively deferring to the market. Gains appear where the available external forecast is weaker: mixed crowd/statistical baselines and structured-data sources.
How large is the improvement reported?
On 2,357 resolved questions across eight sources and five LLMs, the Brier score improves from 0.0771 (external baseline) to 0.0732 (gate), a gain of 0.0039 with p=0.001, and by 0.0027 over the best global pooling baseline (p=0.010). Statistically significant, but modest in absolute terms — and the gate is statistically indistinguishable from a logistic-stacking baseline (p=0.53).
Can LLMs tell you when they are more reliable than a market or crowd?
In this paper, no: asking four Qwen models for verbal confidence yielded AUCs of 0.389–0.501 at predicting whether the model's forecast beat the external source — at or below chance. Signals estimated from resolved outcomes (competence rankings) worked much better for deciding when the model should abstain (risk-coverage area 0.019 vs 0.054).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–3 are present only as captions and in-text numbers; underlying panel-level values beyond those quoted in the text cannot be inspected from this extraction.
- The supplementary material (multi-source identity derivation, hyperparameters, elicitation prompts, ARIMA series-matching details, per-domain contributions) is referenced but not included in the provided text; judgments relying on it are limited accordingly.
- The metadata-supplied abstract is cut mid-sentence, but the abstract embedded in the paper body is complete — there is no truncation in the main text, which runs through the full reference list.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.