StableEval Arena: A Cost-Aware Agentic Benchmark for Stablecoin Price Stability Prediction
Sean Wan, Dongping Liu, Luyao Zhang
cs.LG · cs.AI
Quick verdict
Well-instrumented benchmark: six LLM agent frameworks emit valid JSON cheaply yet miss 19/21 stablecoin stress events; rankings lack statistical testing.
Summary
The paper introduces StableEval Arena, a historical-replay benchmark evaluating six LLM-backed agent frameworks (Nanobot, LangGraph, Hermes AI, EvoAgentX natively; OpenClaw and NanoClaw via adapters) and four non-agent baselines on stablecoin peg-risk prediction. Each case gives a frozen 30-day hourly lookback for USDT, USDC, or DAI (with BTC/ETH context) and requires a strict-JSON 7-day-ahead assessment: stable/watch/stress label, depeg probability, maximum deviation, direction, confidence, rationale. All systems share the qwen-2.5-72b-instruct backend via OpenRouter at temperature 0, a frozen calibration rule, and identical metrics (Macro-F1, balanced accuracy, stress recall, missed-depeg rate, Brier, deviation MAE, valid-JSON rate, latency, token cost). Two blocks are reported: a 120-case stress-enriched validation and a 507-case natural-distribution pool (441 stable/45 watch/21 stress). Key results: calibrated agents beat the historical-persistence baseline on Macro-F1 only in the enriched block (best 0.351 vs 0.285); on the natural pool no agent beats the simple ML/logistic baseline (0.402); all agents miss 19 of 21 stress cases and every agent has missed-depeg rate 1.0 (only 3 sustained-depeg events exist pool-wide). Calibration and verifier ablations yield modest gains at up to doubled cost. Main limitations: no significance testing on rankings, a single LLM backend, two of six agents evaluated only as adapters, and partially unspecified label thresholds.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under one frozen protocol, calibrated agent frameworks beat the historical-persistence baseline on Macro-F1 in the 120-case stress-enriched block (0.351 vs 0.285) but no agent beats the simple ML/logistic baseline (0.402) on the 507-case natural pool.
- All six LLM-backed systems achieved 100% valid structured outputs and zero execution failures across both blocks, at estimated per-case inference costs of $0.003-$0.010.
- Rare-event detection fails uniformly: agents catch only 2 of 21 stress cases (recall at most 0.095) and every agent has a missed-depeg rate of 1.0 on the 3 strict sustained-depeg events, while historical persistence reaches 0.333 stress recall and 0.667 MDR.
- A verifier (second-pass risk auditor) raises best stress recall from 0.095 to 0.143 but roughly doubles mean cost (Table 12: $0.628 to $1.260) with slightly lower best Macro-F1 (0.349 vs 0.351).
- Trustworthiness of agentic systems should be evaluated jointly over forecast quality, operational reliability, and cost; the central empirical finding is the gap between protocol-following reliability and financial-risk reliability.
Strengths
Foregrounds a falsifiable negative result
Sections 4.3 and 5.2 report that all six agents miss 19/21 stress cases and that no agent beats the ML baseline on natural-distribution Macro-F1, positioning the benchmark as diagnostic rather than agent-promotional; the always-stable baseline (0.870 accuracy, zero stress recall, Table 2) makes the accuracy trap explicit.
Leakage-safe, auditable design
Cases use frozen 720-hour lookbacks with hidden labels excluded from prompt packets (Section 3.3); the release ships prompt-packet hashes, freeze manifests, calibration rules, and no-API recomputation scripts (Table 23, Appendix E).
Execution-mode transparency
Multi-dimensional trustworthiness metrics with cost logging
Two-block design isolates distribution effects (RQ3)
Weaknesses
No uncertainty quantification on any reported ranking
Headline missed-depeg claim rests on 3 events
Single shared backend confounds platform with model
ML baseline training protocol unspecified
Frozen calibration rule appears mismatched to natural distribution
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is StableEval Arena and what task does it benchmark?
StableEval Arena is an open benchmark that tests whether LLM-based AI agent frameworks can predict stablecoin peg risk. Each case provides 30 days of hourly price/volume history for USDT, USDC, or DAI (plus Bitcoin/Ethereum context) and asks for a structured 7-day-ahead forecast - a stable/watch/stress label, depeg probability, and maximum expected deviation - in strict JSON, scored against what actually happened in hidden historical windows.
What did the benchmark find about AI agents and stablecoin risk prediction?
The evaluated agents followed the output format perfectly (100% valid JSON) at low cost (roughly $0.003-$0.01 per case), but they missed 19 of 21 real stress/depeg events, and under natural market conditions a simple logistic-regression baseline outperformed every agent on Macro-F1. The key takeaway is that reliably following instructions does not equal reliably detecting rare financial risk.
Which AI agent frameworks were compared in the study?
Six systems were evaluated: Nanobot, LangGraph, Hermes AI, and EvoAgentX as native frameworks, plus OpenClaw and NanoClaw via adapter execution. All six ran the same Qwen2.5-72B-Instruct model through OpenRouter, and were compared against always-stable, historical-persistence, threshold-rule, and machine-learning baselines under an identical prompt and evaluation protocol.
Where can I access the StableEval Arena dataset and code?
According to the paper, the dataset is released on Hugging Face (SeanWan05/stableeval-arena) and the source code on GitHub (StableTradeAtlas/StableEval-Arena). The release includes frozen case lists, prompt packets, calibration rules, cost logs, and scripts that recompute all reported metrics from saved outputs without new API calls.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3-8 appear only as axis-label fragments in the extracted text; visual claims such as the shape of cost-performance frontiers could not be verified.
- Exact numeric label thresholds (theta_watch, theta_stress, h_watch) are not stated in the text beyond Table 5's qualitative definitions.
- Per-case raw outputs and the appendix CSV artifacts referenced in Appendix E are not included in the extracted text, so table-level numbers could not be independently recomputed.
- All cost figures are estimates from backend token prices; OpenRouter price changes since the runs may affect cost reproduction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.