Forecast Workflow Bench: Evaluating Language-Model Decisions with Budgeted Forecast Tools
Shunya Nagashima, Shunya Nagashima
cs.LG · cs.AI
Quick verdict
Claims budgeted-forecast agent benchmark; GPT-6 Astra wins at 2.5% budget, but all local models trail a free heuristic and Astra's repeat lacks significance.
Summary
FWBench is a benchmark that scores language-model agents on whether buying time-series forecasts actually improves operational decisions, not just forecast accuracy. Across 1,251 cases (US electricity demand from EIA-930 and London cycle-hire departures), agents observe 336-hour histories, choose forecast models and horizons from fixed tariffed tools (Chronos-Bolt Tiny, Chronos-2, TimesFM 2.5, plus free classical methods) under a credit budget, and submit capacity plans in eight 3-hour blocks scored by a loss-cost objective normalized against a hindsight optimum. The paper compares two hosted and eight local quantized configurations with paired cluster-bootstrap statistics. GPT-6 Astra achieved the best score (S=0.3964, loss 0.3044), spending about 2.5% of its budget on cheap short-horizon Bolt Tiny forecasts while matching fixed Chronos-2's loss at a small fraction of its credits. All eight local configurations scored worse than a free hour-of-day heuristic, and TSFM access yielded no significant benefit for any local model — it significantly hurt Gemma 4 12B. Instructing agents to minimize the combined score cut Astra's forecast spending by 98.5% with 1.6% higher loss. Main limitations: two domains over 16 dates, simulated contracts, no multiple-testing correction, and a 100-case repeat that failed to confirm Astra's statistical superiority over the best fixed policy.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- FWBench scores language-model agents on a combined decision-loss and forecast-cost objective (Eq. 2) over 1,251 electricity and cycle-hire cases using fixed, tariffed forecast tools.
- GPT-6 Astra achieved the best score (S=0.3964) by selectively buying cheap short-horizon Bolt Tiny forecasts with about 2.5% of the credit budget, matching fixed Chronos-2's decision loss (0.3044 vs 0.3050) at a small fraction of its credits.
- All eight local configurations scored worse than the free hour-of-day policy, and TSFM access provided no statistically significant score benefit to any local model.
- The stated objective causally shaped purchasing: instructing Astra to minimize the combined score cut credits by 98.5% with only 1.6% higher loss on the 16-case test (§4.4).
Strengths
Decision-quality-plus-cost scoring design
Eq. (2) normalizes excess loss against a hindsight optimum F_i and adds a credit-share term, cleanly separating forecast skill from decision value; a shared DP optimizer converts any forecast into a feasible plan, isolating selection behavior from plan optimization (§3.2–3.4).
Careful paired statistics and honest negative results
Paired 95% cluster-bootstrap CIs back the headline claim (§4.2); the paper itself reports that Astra's 100-case repeat CI included zero (§4.5) and that TSFM access gave no local configuration a significant benefit (Table 4, §4.3).
Ablations beyond leaderboard rows
Reproducibility infrastructure
Transparent failure accounting
Weaknesses
Hosted-vs-local comparison is confounded
Headline superiority not seed-stable
Economic interpretation of the budget is partial
Non-completion dominates several local rows
Minor numeric inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FWBench (Nagashima, Neurogica Inc.; arXiv 2609.27385) is a compact benchmark paper asking whether language-model agents improve capacity decisions by selectively purchasing time-series forecasts. Central claims: (1) evaluation should measure decision quality plus forecast cost, not accuracy alone; (2) GPT-6 Astra achieves the best score (S=0.3964, Table 2) by buying cheap short-horizon Chronos-Bolt Tiny forecasts with about 2.5% of the 113,726-credit budget, matching fixed Chronos-2's loss (0.3044 vs 0.3050) at a fraction of its credits; (3) all eight local configurations score worse than the free hour-of-day policy; (4) TSFM access yields no significant score benefit for any local model (Table 4). Claims are mostly tied to specific paired tests — a discipline many leaderboard papers lack.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FWBench?
FWBench is a benchmark that evaluates whether language-model agents can improve operational capacity decisions by selecting and paying for time-series forecasts under a fixed credit budget. Agents observe demand histories, buy forecasts from fixed tools (Chronos-Bolt Tiny, Chronos-2, TimesFM 2.5, plus free classical methods), and submit capacity plans scored by a combined decision-loss and forecast-cost metric.
Which model performed best on FWBench?
GPT-6 Astra achieved the best score (S=0.3964). It selectively bought cheap short-horizon forecasts using about 2.5% of its credit budget and matched the decision loss of always using Chronos-2 at a small fraction of that policy's credit cost. All eight local (open-weight) configurations scored worse than a free hour-of-day heuristic.
Do small local models benefit from access to forecasting tools?
According to the paper, no. None of the eight local configurations showed a statistically significant score benefit from time-series foundation model access, and access significantly worsened results for Gemma 4 12B (without thinking) and Qwen3.6 35B-A3B (with thinking).
Is FWBench's code and data public?
Yes. The paper links a public GitHub repository (Neurogica/forecast-workflow-bench) and a Hugging Face dataset (Neurogica/forecast-workflow-bench). Fixed-policy baselines and oracle references are deterministically reproducible on CPU; runs of hosted models such as GPT-6 Astra are not exactly reproducible because they depend on proprietary APIs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are available only as caption text; the loss–cost trade-off plot (Fig. 3) and per-case capacity trajectories could not be independently verified.
- Abstract states 2.5% of budget while §4.2 states 2.6% for the same 2,802/113,726 credits (arithmetically 2.46%); noted as a minor inconsistency.
- Hosted-model results depend on proprietary API versions and sampling; exact replication of agent rows is not possible from the paper alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.