E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning
Xiaoya Wang, Yutong Xu, Junjie Wang
cs.CL
Quick verdict
New benchmark scores chart-evidence-to-action reliability in 20 VLMs via NDR; design and validation are strong, but high UCR and single runs warrant caution.
Summary
The paper introduces E2A-Bench, a 969-query benchmark (323 HS300 constituents × text-only, chart-only, and multimodal inputs) that tests whether vision-language models turn candlestick-chart evidence into reliable trading recommendations. Instead of future returns or human action labels, each 30-trading-day OHLCV window yields ten deterministic evidence anchors (3,230 facts), against which the protocol audits model claims (UCR), rationale–action consistency (RCI), high-confidence action under weak evidence (ECI), and a coverage-weighted net decision reliability (NDR). Evaluating 20 VLMs shows scalar hallucination scores can mislead: Gemma-4-E2B-It has the lowest UCRall (39.9%) yet only 6.4% directional coverage and NDR 5.1%, while Qwen2.5-VL-3B tops NDR (32.1%) at 82.5% coverage; proprietary models trail open-weight ones. Chart-only interventions suggest oracle-aided verification can collapse coverage (negative mean ΔNDR), and three strict base–fine-tuned pairs show consistent BUY:SELL amplification (×4.21–4.68) despite heterogeneous NDR changes. Validation is nontrivial: four industry annotators (Gwet's AC1 0.71–0.82), scorer F1 ≥85.9%, and NDR correlates best with holistic human judgment (ρ=0.466, n=30), only marginally above coverage alone (ρ=0.453). Main limitations: a single HS300 snapshot in Simplified Chinese, single-run inference without leaderboard confidence intervals, and the verification-collapse finding not evidenced by any table in the provided text.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Scalar claim-level hallucination scores misrank decision reliability: Gemma-4-E2B-It has the lowest UCRall (39.9%) yet ranks 19/20 by NDR (5.1%) because its directional coverage is only 6.4%.
- Oracle-aided verification can sharply reduce unsupported claims while collapsing actionable coverage, yielding negative mean ΔNDR in chart-only interventions.
- Financial fine-tuning consistently amplifies the BUY:SELL ratio by 4.21-4.68× across three strict base-fine-tuned pairs (1.41→6.10, 0.85→4.00, 1.84→7.73), despite inconsistent NDR changes.
- The three local diagnostics are largely non-redundant (pairwise Spearman ≤0.26 across 20 models), supporting a chain-level rather than scalar evaluation.
- NDR correlates most strongly with holistic human reliability judgments (ρ=0.466, p=0.009) among compared metrics in a blinded 30-response pilot.
Strengths
Chain-level evaluation design with demonstrated non-redundancy
Four metrics cover grounding (UCR), reasoning-action consistency (RCI), calibration (ECI) and coverage (NDR); Fig 11 shows pairwise Spearman correlations among UCRdir, RCI and ECI are all ≤0.26 across 20 models, so the diagnostics capture distinct failure modes rather than re-measuring one quantity.
Deterministic, fully specified evidence anchors instead of subjective action labels
Table 7 specifies all ten OHLCV anchors with exact rules and thresholds (e.g., trend labeling at 0.15%/day vs strong-signal counting at 0.30%/day; volume heavy at >1.5× the 20-day mean), making the ground truth auditable and reproducible without human trading judgment.
Above-average scorer validation effort
Metric robustness analyses pre-empt arbitrariness objections
Hedged causal language on the fine-tuning finding
Weaknesses
UCR support rule likely conflates 'false' with 'off-ontology', inflating error rates
Second headline finding lacks visible supporting results
No uncertainty on the main leaderboard; single-run inference
Human-alignment evidence for NDR is thin
Arbitrary ECI confidence cutoff, single-market single-snapshot scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
E2A-Bench (arXiv 2609.14302) is a benchmark paper addressing what the authors call coverage-blind and chain-blind evaluation of financial VLMs: claim-centric hallucination scores reward models that avoid commitments, while task-centric scores never audit the evidence behind an action. The benchmark comprises 323 HS300 constituents × three input modalities (text-only, chart-only, multimodal) = 969 queries, each a 30-trading-day OHLCV window annotated with ten deterministic evidence anchors (3,230 facts; full spec in Table 7). Models emit an analysis, rationale, action (BUY/SELL/HOLD/UNCERTAIN) and 1-5 confidence; the protocol scores UCR (Eq. 6), RCI (Eq. 7), ECI (Eq. 8) and NDR (Eq. 11) = Covdir × (1−UCRdir)(1−RCI)(1−ECI). Three headline claims: (1) scalar hallucination scores misrank reliability — Gemma-4-E2B-It has the lowest UCRall (39.9%) but NDR 5.1% at 6.4% coverage; (2) oracle-aided verification trades unsupported claims for coverage collapse (negative mean ΔNDR); (3) financial fine-tuning amplifies BUY:SELL ratios ×4.21-4.68 across three strict base-fine-tuned pairs.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is E2A-Bench?
E2A-Bench is a 969-query benchmark (323 HS300 stocks × three input modalities: text-only, chart-only, multimodal) that tests whether vision-language models turn candlestick chart evidence into trading recommendations whose claims, rationale, confidence and final action all hold up against deterministic OHLCV-derived facts. It scores four links: UCR (unsupported claims), RCI (reasoning-action inconsistency), ECI (overconfidence under weak evidence) and NDR (coverage-aware net decision reliability).
What does NDR mean in E2A-Bench and how is it computed?
NDR (Net Decision Reliability) is directional coverage — the fraction of responses committing to BUY or SELL — multiplied by (1−UCR)(1−RCI)(1−ECI) computed on directional responses. It rewards models that actually make directional recommendations while keeping their evidence chain sound; a model that mostly answers HOLD or UNCERTAIN gets near-zero NDR even with a low hallucination rate, which is the 'coverage trap' the benchmark is designed to expose.
Which models performed best and worst on E2A-Bench?
In the paper's 20-model evaluation, open-weight Qwen2.5-VL-3B-Instruct ranked first by NDR (32.1% at 82.5% directional coverage), followed by Qwen2.5-VL-7B (25.0%) and PyFi-QwenVL-3B (25.3%). The best proprietary model, Gemini-3.1-Pro, reached only 16.8%. Gemma-4-E2B-It had the lowest unsupported-claim rate (39.9%) but ranked near the bottom by NDR (5.1%) because it rarely committed to directional actions (6.4% coverage).
Does a high E2A-Bench score mean a model is good at trading?
No. The authors explicitly state that NDR measures evidence-to-action reliability under their evaluation protocol, not realized trading returns, profitability or investment quality. Rankings on E2A-Bench should not be used as the sole basis for selecting or deploying systems in financial applications.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text ends with a complete sentence after Appendix C.6; any appendix content beyond C.6 would be invisible to this review, so the absence of an oracle-intervention results section is judged on the provided extraction only.
- Values embedded in Figures 4, 5, 7 and 10 (e.g., the Qwen3.5 scaling series 24.1/12.6/18.1/13.1) were read from OCR-extracted numerals; the exact figure-to-value alignment cannot be fully verified from text extraction.
- A.7 states artifacts will be released 'upon acceptance'; the actual contents of the stated GitHub repository could not be checked in this offline review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.