What LLM Trading Agents Actually Do in Production: A Six-Month, Population-Scale Record from Two Fleets
T. J. Barton, Chris Constantakis, Patti Hauseman, Annie Mous, Alaska Hoffman, Brian Bergeron, Hunter Goodreau
cs.AI · cs.CE · cs.MA
Quick verdict
Paper: the operating layer, not model reasoning, drives LLM trading agents. Candid and rigorous, but paper-engine fills and self-citation temper trust.
Summary
This paper assembles a roughly six-month measurement record of autonomous LLM trading agents across two production systems sharing one design lineage: DX Terminal Pro (February–March 2026; 3,505 user-funded vaults trading real ETH in Base memecoin markets for 21 days) and the DXAP fleet (June–August 2026; 500–599 user-created agents, mostly $10,000 paper accounts, trading Hyperliquid perpetuals). The record spans 7.5M single-model invocations, ~300K onchain actions, 231,638 multi-tool turns, and 14,596 fills. Four findings carry the paper: (1) the operating layer — sliders, rendered candidate lists, order-path mechanics — determines behavior more than strategy text (risk slider maps to leverage at +0.425x per level; a leaderboard render boundary routes selection 1.75x by regression discontinuity); (2) sizing is volatility-blind (median leverage stays 5.0x in every volatility sextile while realized returns degrade from -10.6 to -98.2 bps, and 62% of 205 liquidations sit in one slider cell); (3) agents fail to capture favorable excursion (43.2% of positions reach +300 bps within 24h, yet 49.3% of those close negative; a mechanical 2%/4% bracket adds +39.0 bps per position); (4) neither fleet shows directional edge (-$217K cumulative P&L restated at a common fee; 41% vs 50% win rate versus a matched retail benchmark). Claims carry FIRM/PROVISIONAL evidence classes and three earlier results were self-retracted. Main limitations: most fills come from a paper engine with zero slippage, zero funding, and placeholder margining that liquidates roughly 2x later than real Hyperliquid; evidence is largely self-referential; raw telemetry is not public.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The operating layer (slider constraints, rendered candidate lists, order-path mechanics) determines agent behavior more than strategy text or model choice, with a risk slider mapping to leverage at +0.425x per level and a render boundary routing selection 1.75x at the top-3 cut.
- Sizing is volatility-blind: median chosen leverage is 5.0x in every volatility sextile across a 5.7x volatility spread while median realized returns degrade from -10.6 to -98.2 bps, and one posture-by-frequency cell holds 62% of the 205 liquidations.
- Agents fail to capture the upside they reach: 43.2% of 6,400 closed positions saw at least +300 bps of favorable excursion within 24h, 49.3% of those closed negative, and a mechanical 2%/4% bracket recovers +39.0 bps per position.
- Neither fleet shows a directional edge: the DXAP fleet's cumulative realized P&L is -$217K at the common 5.5 bps fee rate and it trails a matched Hyperliquid retail benchmark 41% to 50% in roundtrip win rate.
- Frontier models are statistically indistinguishable in decision quality at this horizon under paired replay (regret 263.3–264.4 bps, minimum Holm-adjusted p = 0.46), differing sharply only in choice stability and inference cost.
Strengths
Explicit evidence-class and retraction discipline
Section 2.4 grades every claim FIRM/PROVISIONAL/WEAK-RETRACTED and documents three self-retractions (a HIP-3 timestamp artifact and two posture/edition claims killed by calendar-footprint checks); canon rule 15 even reports a rolling p-value of 0.0067 → 0.19 → 0.0277 before it was called — rare transparency that raises trust in the surviving claims.
One genuinely clean causal identification
The render-boundary regression discontinuity (Section 4.2, Figure 3) gives 1.75x [1.49, 2.06] at the rank-3/rank-4 cut, cross-checked against the raw count ratio 1.81x (290/160), on 2,339 entry observations — with the over-selection baseline (46.5% vs 8.9% availability, 5.2x) reported separately.
Honest fee restatement and null-result census
A transferable, failure-derived methodology canon
Careful scoping of the preregistered probe
Weaknesses
Headline risk results rest on an artificial paper engine
Load-bearing references are all internal and unreviewed
The fleet-vs-retail comparison lacks a test
The 'six-month record' framing stretches two disjoint windows
Raw data and code are not available for recomputation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement record, not a performance paper — the authors state this commitment explicitly and keep it. The evidence covers two production systems with a shared design lineage (five-slider config, one-action-per-turn loop, Go-template prompts): DX Terminal Pro (Feb 26–Mar 18, 2026; 3,505 funded vaults, 3,454 active; Qwen3-235B at temp 0.6; 7.5M invocations, ~300K onchain actions, real ETH) and the DXAP fleet (Jun 8–Aug 15, 2026; 500–599 agents all-history, 91–117 concurrently active; 231,638 finalized turns; 14,596 fills, mostly $10K paper accounts plus 5,035 flagged real-money fills). Four headline claims, all marked FIRM: operating-layer dominance (Section 4), volatility-blind sizing (Section 5.1), the capture gap and bracket result (Section 6), and the no-edge null (Section 3.2, Section 7). The claim inventory is internally consistent — the same figures recur in abstract, body, and conclusion.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did the LLM trading agents in this study make money?
No. The paper explicitly reports no directional edge for either fleet. The DXAP fleet's cumulative realized P&L is -$217K when restated at a common 5.5 bps fee, only 15% of week-active agents were net-positive (vs 53% of a matched Hyperliquid retail sample), and in DX Terminal Pro the median vault return was 0.492x with 16.2% of vaults finishing profitable.
Was real money traded in these experiments?
Partly. DX Terminal Pro traded real ETH: 4,984 ETH deposited across 3,505 user-funded vaults, with 92% withdrawn. In the DXAP fleet, most agents ran $10,000 paper accounts against live prices, with a small real-capital book of 5,035 real-money fills from 6 of 19 users (notionals $14-$2,000), which the authors flag but analyze within the single fleet population.
What is the main practical takeaway of the paper?
That agent behavior is set by the machinery around the model, not the prompt: attach stop/target brackets at the order path (+39.0 bps per position in their paired backtest), scale leverage to realized volatility at the tool rather than letting a slider fix it at 5.0x, and A/B test what the UI renders, since a leaderboard render boundary alone moved symbol selection 1.75x.
Which AI models ran the trading agents?
DX Terminal Pro used a single frozen model, Qwen3-235B-A22B-Thinking-2507, served via SGLang. The DXAP fleet ran an OpenRouter mix dominated by qwen3.7-plus (86 of 91 active agents on July 22). The June 2026 replay league compared qwen3.7-max, qwen3.7-plus, and claude-fable-5 and found no measurable decision-quality separation among them.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are extracted as flattened caption text; all cited numbers appear in captions or body text, but plot-internal details (e.g., exact bar heights in Figure 4) could only be read from the caption values.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.