DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis
Abhinav Rajeev Kumar, Harshit Arora, Varun Singh, Manikandan Nanjappan
cs.LG · cs.SE
Quick verdict
Claims LLM DeFi workflows execute unsafely; its repair layer zeroes unsafe trades in local simulation, but small team-authored data limits generality.
Summary
The paper introduces DeFiFlowBench, 207 team-authored prompts (120 development, 87 held-out) for natural-language DeFi workflow synthesis, scored by three nested static metrics — graph coverage, configuration completeness, and declared safety predicates — plus an execution check that replays supported trades on a local constant-product AMM in an in-process EVM. Its central distinction: a slippage bound derived from a quote does not cap the order's own price impact. Direct, constrained, and few-shot prompting of Gemini 3.1 Flash Lite and GPT-5.4 mini each yields 14–19 unsafe held-out executions under a fixed 5% impact cap, while the best baseline (safety-instructed Gemini) scores only 0.33 on the static safety proxy. The proposed Koan-Safe — prompt-only intent parser, replaceable generator, and a repair layer injecting default safety parameters — reaches 0.67 with its hybrid variant (Gemini) and records zero unsafe executions; a matched-candidate ablation produces 14–17 unsafe executions when enforcement is disabled. A 34-pair metamorphic suite and a 36-case policy grid show permissive pre-existing thresholds still authorize unsafe trades; a post-hoc 3% policy cap fixes this diagnostically. Main limitations: prompts, gold annotations, and method share one team; single temperature-zero generations; simulation neither traverses the generated DAG nor covers bridges; the policy cap was designed after inspecting results.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Direct, constrained, and few-shot LLM prompting produce 14–19 unsafe held-out executions per configuration under a fixed 5% price-impact cap, because a slippage bound derived from a quote cannot limit an order's own price impact.
- Koan-Safe's generator-agnostic enforcement layer raises the static safety proxy from 0.33 (best baseline) to 0.67 (hybrid, Gemini) on 75 held-out workflow prompts and eliminates unsafe executions in matched replay.
- Default safety injection has a boundary: pre-existing permissive thresholds still authorize unsafe trades, which a post-hoc policy cap (3% impact, 1% slippage) fixes on a 36-case diagnostic grid.
- Static declared-safety scores and execution outcomes measure different failures; neither a high static score nor zero observed unsafe trades establishes a general safety guarantee.
Strengths
Execution-based evaluation targeting a real failure mode
The slippage-vs-own-price-impact distinction (Section II-e) is operationally meaningful: Table I shows direct/constrained/few-shot configurations mining 14–19 trades above the fixed 5% cap while their static safety proxies sit at 0.00–0.13, a gap presence-based scoring alone would not expose.
Clean matched-candidate ablation design
Table II reconstructs both enforcement conditions from the same saved candidate, requiring the frozen layer to reproduce the saved workflow exactly — 14–17 unsafe executions off vs zero on across rules/LLM/hybrid generators and both model families, with no new API calls (Section VI-C).
Explicit epistemic discipline
Calibration and artifact hygiene
Weaknesses
Authorship circularity across benchmark and method
Single temperature-zero generation per configuration
Narrow execution semantics
Limited model coverage with strong model-specificity
Headline framing slightly ahead of the evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
DeFiFlowBench (arXiv 2609.11504) bundles two contributions: a benchmark of 207 team-authored prompts (120 development across swaps, limit orders, cross-chain transfers, and compositional tasks; 87 held-out including out-of-vocabulary tokens and structures outside the proposed method's presets) plus 34 metamorphic prompt pairs; and Koan-Safe, a synthesis pipeline whose enforcement layer repairs workflow structure and injects default safety parameters. The central conceptual claim is that a slippage bound derived from a quote cannot cap an order's own price impact, so safety must be measured by execution, not by presence of declared protections. Headline numbers: the best baseline reaches 0.33 on the static safety proxy (safety-instructed Gemini, 95% Wilson CI [0.24, 0.45]) versus 0.67 for Koan-Safe hybrid with Gemini (CI [0.55, 0.76]) on 75 held-out workflow prompts; direct, constrained, and few-shot prompting produce 14–19 unsafe executions per configuration under a fixed 5% impact cap, while every frozen Koan-Safe configuration records zero.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DeFiFlowBench?
A benchmark of 207 human-written prompts (120 development, 87 held-out) for generating DeFi workflows from natural language. It scores generated workflows on graph coverage, configuration completeness, and declared safety predicates, and additionally executes supported trades against a simulated constant-product AMM on a local EVM to check whether they respect a 5% price-impact cap.
What is the difference between slippage and price impact in this paper?
A slippage bound is a minimum output computed from an already impact-adjusted quote; it protects against price movement between quote and execution. It does not prevent a large order from moving the price against itself. A separate price-impact gate blocks trades whose own footprint exceeds a threshold.
What is Koan-Safe and how well does it perform?
A three-stage pipeline (prompt-only intent parser, replaceable generator, safety-enforcement layer that repairs structure and injects default safety parameters). Its hybrid variant with Gemini scores 0.67 on the static safety proxy versus 0.33 for the best baseline, with zero unsafe executions in the local checker; an ablation shows 14–17 unsafe executions when enforcement is disabled.
Does Koan-Safe guarantee safe DeFi trades?
No. The evaluation uses mock tokens and one simplified constant-product AMM, does not traverse the generated workflow graph, and excludes bridges and adversarial risks like sandwich attacks or oracle manipulation. Zero observed unsafe executions in this simulation is not a real-world safety guarantee, as the authors themselves state.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 and Figure 2 values extract as unformatted number streams, so per-segment bar values were not independently verified; all cited numbers come from the running text and Tables I–III, which extracted cleanly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.