EDGEGEN: Improving Tool-Calling Agents Beyond Happy Paths with Synthetic Edge Case Generation
Harshavardhan Abichandani, Penny Chong, Jiyuan Shen, Gunraj Singh, Ashutosh Hathidara, Marcus Duigan Xing Yu, Jane Lo, Atin Ghosh, et al.
cs.AI
Quick verdict
Automated edge-case task generation lifts tool-agent finetuning +2–42% consistently; but tiny 10–15-scenario evals and judge-error rates temper claims.
Summary
EDGEGEN is a synthetic task-generation framework for tool-calling LLM agents. It extracts atomic compliance rules from an agent's policy document (top five rules, at most 32 rule subsets), enumerates violation scenarios over these combinations, samples tool workflows from a policy-conditioned tool graph, grounds tasks in database states via a SQL agent, and admits tasks only after data-grounding and feasibility verification. Combined with an anonymized synthetic database generator (under review), it forms a closed loop that turns GPT-5.4 expert trajectories into finetuning or harness-optimization data without human annotation. Across tau2-bench airline and retail and ToolSandbox, finetuning six open models on EDGEGEN data improves mean progress in all 18 model-benchmark pairs (relative gains from +2% to +42%), whereas human-curated, naive, TaskBench, and FuncBenchGen baselines often degrade. Harness optimization on EDGEGEN data gives +30% and +10% relative mean-progress gains over the base and human-curated harnesses for Gemma-4-e4b on airline with fewer tokens, but trails the human-curated harness for GPT-5.4. Key limitations: only 10-15 evaluation scenarios per benchmark with no significance testing; a human study confirms only 6/10 (airline) and 4/10 (ToolSandbox) of reported successes; gains concentrate on smaller models; complexity and data-scaling effects are non-monotonic (two-violation training drops below base for Qwen2.5-3b).
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Extracting atomic compliance rules from an agent specification and enumerating their violations yields database-grounded edge-case tasks that generic synthetic generators systematically miss.
- Supervised finetuning on EDGEGEN data improves mean progress in all 18 model-benchmark pairs, with relative gains of +2% to +42%, while human-curated, naive, TaskBench, and FuncBenchGen data often degrade performance.
- Harness optimization on EDGEGEN tasks beats the base and human-curated harnesses for Gemma-4-e4b on tau2-bench airline (+30%/+10% relative mean progress) using fewer tokens and tool calls.
- The entire pipeline operates without human annotation or production logs, forming a closed loop with an anonymized synthetic database generator.
- Training on happy paths alone is insufficient and can be harmful; single-violation data helps most, but multi-violation data can hurt (non-monotonic complexity effect).
Strengths
Verified, database-grounded generation attacks a documented failure mode
Sec. 3.1-3.2 define two admission checks - data grounding (identifiers must exist in D) and scenario feasibility (compliant response must be executable, including denials). Appendix A.3 shows baseline samples with fabricated reservation IDs (HATHAT reused across unrelated samples), non-existent airports, and natural-language fragments passed as API arguments; the verify-then-admit loop is the real contribution.
Uniform finetuning gains where baselines degrade
Table 1: mean progress improves in all 18 model-benchmark pairs (100% win rate vs base, Table 10). On tau2-bench airline, human-curated finetuning regresses 3/6 models and TaskBench 4/6, directly supporting the coverage hypothesis that happy-path-only data can be actively harmful.
Self-auditing honesty uncommon in synthetic-data papers
Mechanistic analysis of why edge data helps
Reproducibility scaffolding
Weaknesses
Underpowered, significance-free evaluation
Circular evaluation-signal validity
Headline scope narrower than abstract
Key dependency unpublished and rule space unexamined
Generation knobs are unreliable dials
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EDGEGEN (SAP, arXiv 2609.24115) targets a real production pain point: tool-calling agents are trained and evaluated largely on happy-path interactions, while field failures cluster at policy boundaries - refund eligibility, loyalty-tier baggage rules, identity verification before writes. The paper claims three things: (1) a violation-driven task generator that extracts atomic compliance rules from an agent specification, enumerates their combinations, and grounds tasks in executable database states (Sec. 3); (2) a fully automated closed loop - synthetic database, generated tasks, GPT-5.4 expert trajectories - requiring no human annotation (Contribution 2); (3) empirical gains: finetuning improves mean progress on all 18 model-benchmark pairs (+2% to +42% relative, Table 1), and harness optimization yields +30%/+10% relative mean-progress improvements over base/human-curated harnesses for Gemma-4-e4b on tau2-bench airline (Table 2). Within the paper's own small-sample regime these claims are mostly supported, and the authors are unusually candid about where they do not extend (GPT-5.4, non-monotonic scaling).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EDGEGEN?
EDGEGEN is a framework from SAP researchers that automatically generates edge-case training and evaluation tasks for tool-calling LLM agents. It extracts business-policy rules from the agent's specification, enumerates combinations of rule violations, grounds each task in a real database state via a SQL agent, and verifies feasibility before admitting the task - all without human annotation.
How much does EDGEGEN improve agent performance?
In the paper's experiments on tau2-bench (airline and retail domains) and ToolSandbox, finetuning six open-source models on EDGEGEN data improved mean progress in all 18 model-benchmark pairs, with relative gains from +2% to +42% (largest for Qwen2.5-3b on the airline domain). For harness optimization, the Gemma-4-e4b agent improved +30% relative mean progress over the base harness and +10% over a harness tuned on human-curated data.
Does EDGEGEN help large or already-strong models?
Gains concentrate on smaller or weaker models. For GPT-5.4, the EDGEGEN-optimized harness scored 0.798 mean progress versus 0.831 for the human-curated-optimized harness, though it used fewer tokens and tool calls. The authors note strong models have little headroom because the human-curated test tasks are relatively simple.
Is EDGEGEN code and data available?
The paper states the EDGEGEN code and generated datasets are released under Apache 2.0, with prompts and LoRA hyperparameters in the appendix. One pipeline component - the synthetic database generator - is a separate anonymous submission under review, so full pipeline reproduction may require that companion work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is only textually described; the visual pipeline layout could not be verified from the extraction.
- Table 1 was extracted in a dense max|mean notation with inline deltas; minor transcription risk when citing individual cells.
- The bibliographic record (arXiv 2609.24115, several 2026-dated references) appears very recent/provisional; citation metadata could not be cross-checked.
- The Anonymous 2026 database-generator component is not inspectable from the provided text, so closed-loop claims rest on the authors' description.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.