LEAP: Likelihood Elicitation and Aggregation for LLM-based Probabilistic Forecasting
Yufei Chen, Yiran Zhao, Xiaogang Xu, Qipeng Xie, Jiafei Wu, Zhe Liu
cs.AI
Quick verdict
LEAP replaces holistic LLM forecasting with per-evidence likelihoods and Bayesian aggregation; gains broad but rest on hand-set constants and 60-task analyses.
Summary
The paper targets the final prediction stage of LLM forecasting agents, where all gathered evidence is read in one opaque call — the authors' 'Monolithic Prediction.' LEAP instead elicits, in isolated LLM calls, a prior and per-evidence likelihood parameters, then combines them with a closed-form tempered Bayesian update (Gaussian, categorical, or Bernoulli conjugate families) into a posterior, with safeguards: dependency clustering, reliability sampling, and 4-sigma outlier rejection. Evaluation uses a 347-task benchmark built from FutureX, GAIA, and BrowseComp, with a shared evidence set per task across methods. Across five base models on the authors' ReAct-style loop, LEAP improves the FutureX composite by 3.6-18.1 points, and accuracy, spherical score, and NCRPS improve in every cell, while Brier improves in only three of five models. As a plug-in skill on four external agent CLI frameworks it gains 7.0-13.8 FutureX points. On a 60-task diagnostic subset, ECE roughly halves (0.184 to 0.088) and gains widen with horizon. Main limitations: likelihood parameters come largely from hand-set calibration tables, the diagnostics rest on the small subset with one model, no formal significance tests are reported, and the closest related systems (BIRD, Bayesian Linguistic Forecaster) are discussed but not empirically compared.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Given an identical fixed evidence set, LEAP's per-evidence likelihood elicitation plus closed-form Bayesian aggregation improves most prediction and calibration metrics over Monolithic Prediction across five base models and four external agent CLI frameworks (Tables 1-2).
- LEAP roughly halves expected calibration error (0.1840 to 0.0876) and overconfidence (0.3167 to 0.1500) on the diagnostic subset, with advantages widening as forecast horizon grows from 7 to 60 days (Table 6, Figure 3).
- The explicit prior is the most critical component: removing it drops FutureX below Monolithic (0.6427 vs 0.6512, Table 3).
- LEAP works as a plug-in probability skill on unmodified traces from DeerFlow, Hermes, OpenClaw, and MiroFlow without retraining, gaining 7.0-13.8 FutureX points (Table 2).
- Because aggregation is deterministic and closed-form, per-evidence leave-one-out contributions are exactly reproducible, providing an audit signal unavailable from free-text LLM rationales.
Strengths
Strictly controlled comparison protocol
Sections 3.3/5.1 hold the evidence set fixed per task across both methods, restrict retrieval to pre-freeze material, and constrain models by knowledge cutoff (Table 12), so score differences attribute to the prediction stage rather than retrieval; D.3 reports a temporal-leakage audit with zero violations.
Confounds addressed, costs disclosed
Table 4 runs prior-matched and token-matched Monolithic controls (5,733 vs 11,508 tokens/task, about 2x), and Table 5 shows structured conjugate updating (0.7284 FutureX) beats a linear opinion pool (0.6808); the p95 latency disadvantage from the slowest per-evidence call is admitted.
Calibration gains with horizon analysis
Genuinely reproducible audit trail
Above-average variance reporting
Weaknesses
Elicitation is largely a fixed lookup, not learned probabilistic judgment
All component diagnostics rest on 60 tasks and one model
Brier is not uniformly improved
No empirical comparison to nearest related systems
Benchmark is self-constructed with author-authored distractors
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
LEAP: Likelihood Elicitation and Aggregation for LLM-based Probabilistic Forecasting — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'Monolithic Prediction' in LLM forecasting?
It is the common design where an LLM agent first collects evidence (e.g., via web search) and then, in one final LLM call, reads the entire evidence bundle at once and outputs the forecast. The LEAP paper argues this design is opaque — you cannot isolate which evidence item drove the answer — and tends to collapse uncertainty into overconfident single answers.
Does LEAP require training or fine-tuning a model?
No. LEAP uses the base LLM through prompt-only, isolated per-evidence calls that return structured likelihood judgments, then combines them with a closed-form Bayesian update in fixed code. It was also applied to four external agent frameworks (DeerFlow, Hermes, OpenClaw, MiroFlow) as a pure post-processing 'probability skill' over their completed traces, with no retraining or framework modification.
How much extra compute does LEAP cost compared to a single LLM forecast?
Per the paper's Table 4, roughly twice the tokens per task (11,508 vs 5,733 on the diagnostic setup), with similar median latency but higher tail (p95) latency, because per-evidence calls run in parallel and completion waits for the slowest one. This cost is explicitly listed as the main practical tradeoff.
What benchmarks and models were used to evaluate LEAP?
A 347-task benchmark built from FutureX, GAIA, and BrowseComp (266 single-choice, 29 multi-choice, 52 continuous tasks), evaluated with five LLMs (DeepSeek-V3.2, Gemini-3.1-Flash-Lite, Claude-Haiku-4.5, GPT-5.4-mini, Grok-4.20-Fast) on the authors' ReAct-style agent loop, and as a plug-in on four external agent CLI frameworks, with five random seeds per configuration.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction linearizes math and figure-embedded labels (Figures 1-3, Tables 7-8 equations); symbols were reconstructed from surrounding context, but no numerical table content appeared to be missing.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.