Failure-Guided Co-Evolution of Prompts and Training Data
cs.SE · cs.AI
Quick verdict
FORGE co-evolves prompts with failure-guided synthetic data, +16.52 pts over baseline on 8 benchmarks; promising but single-seed, no significance testing.
Summary
The paper introduces FORGE, an automatic prompt optimization (APO) framework that treats training data as mutable optimization state. FORGE runs a GEPA-style reflective prompt search whose imperfect executions are grouped by an LM assigner into reusable, decontextualized failure modes; when validation performance stalls, active modes guide data synthesis through four mutation intents (positive, negative, boundary, stress), and a tool-capable verifier admits only valid and faithful instances. Across eight benchmarks (HotPotQA, HoVer, IFBench, FiNER-139, LawBench, AIME-2025, MMLU-Pro, AppWorld) with Qwen3.5-35B-A3B as task model and GPT-5.5 as optimizer, FORGE reaches an aggregate of 72.18 versus 55.66 for the unoptimized baseline (+16.52 points) and beats MIPROv2 (63.36), ACE (67.26), and GEPA (65.33); results largely hold on Qwen3-8B (+14.10 over baseline, +5.38 over GEPA). A cumulative ablation shows random synthesis hurts (62.38) while failure memory and mutation guidance are both needed (68.24, 72.18). In transfer studies, FORGE-augmented data improve all nine APO comparisons by 2-9 points and all three GRPO comparisons by 4-8 points under matched budgets. Main limitations: single-seed runs without variance estimates on small test sets (30-294 examples), budget alignment that excludes optimizer-side LM calls, and generator/verifier sharing the same GPT-5.5 model.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Treating training data as mutable APO state, with failure-mode-guided synthesis, yields +16.52 aggregate points over the unoptimized baseline across eight benchmarks with Qwen3.5-35B-A3B (Table 1: 72.18 vs 55.66).
- FORGE outperforms MIPROv2 (63.36), ACE (67.26), and GEPA (65.33) by 4.92-8.82 aggregate points, and leads GEPA by 5.38 points on Qwen3-8B (Table 3).
- Random synthesis on expanded data is harmful (64.17→62.38), while failure memory and mutation guidance contribute further conditional gains to 72.18 (Table 2).
- FORGE-synthesized instances transfer: all nine prompt-optimizer comparisons improve 2-9 points and all three GRPO comparisons improve 4-8 points under matched budgets (Table 4).
- A two-stage audit finds no semantic overlap between synthesized instances and validation/test partitions (Appendix D.2).
Strengths
Well-formalized problem statement
Eq. 2 defines adaptive-data APO as a strict generalization of fixed-data APO (A≡∅ recovers the special case), with append-only data updates and a validation set that never supplies training or synthesis instances — a clean contract that makes the claim testable.
Ablation directly tests the core hypothesis
Table 2 shows random synthesis on expanded data hurts (64.17→62.38), while failure memory (+5.86) and mutation guidance (+3.94) recover to 72.18 — isolating failure-guidance from a mere more-data effect, which is the paper's central claim.
Transfer study goes beyond the home optimizer
Unusual implementation transparency
Honest failure accounting and robustness checks
Weaknesses
No variance or significance reporting
Compute alignment is asymmetric
Generator and verifier share one model
Cumulative ablation conflates components
Some benchmarks limit informativeness
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FORGE (arXiv 2609.15209, cs.SE/cs.AI) is a method paper on automatic prompt optimization (APO). Its thesis: every failed execution is a dual signal — it should drive both a prompt revision and the synthesis of new training evidence. The paper formalizes adaptive-data APO (Eq. 2) as append-only training-data growth A(H) layered on a GEPA-style search U, with the validation set supplying selection signal only and fixed-data APO recovered when A≡∅. Four claims are staked: (1) +16.52 aggregate points over the unoptimized baseline across eight benchmarks (Table 1: 72.18 vs 55.66); (2) superiority over MIPROv2 (63.36), ACE (67.26), and GEPA (65.33) by 4.92–8.82 points; (3) failure-guided rather than random synthesis is necessary (Table 2); (4) synthesized instances transfer to other APO methods (2–9 points over nine comparisons) and to GRPO (4–8 points over three comparisons) under matched budgets (Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FORGE in prompt optimization?
FORGE (Failure-Guided Co-Evolution of Prompts and Training Data) is a framework that simultaneously optimizes an LLM program's prompts and grows its training set. It abstracts failed executions into reusable failure modes and, when validation progress stalls, synthesizes new training examples via positive, negative, boundary, and stress mutations, admitting only instances that a separate tool-using verifier agent judges valid and faithful.
How much does FORGE improve over baselines?
On eight benchmarks with Qwen3.5-35B-A3B as the task model, FORGE's aggregate score is 72.18 versus 55.66 for the unoptimized baseline (+16.52 points), ahead of MIPROv2 (63.36), ACE (67.26), and GEPA (65.33). Its synthetic data also lift nine prompt-optimizer comparisons by 2-9 points and three GRPO weight-optimization runs by 4-8 points under matched budgets.
Does FORGE's synthetic data leak test questions?
The authors report a two-stage audit: GPT-5.5 screened every generated example against validation and test partitions for semantic equivalence, and a random 10% was manually checked; no matches were found. This audit is self-reported in the paper and was not independently verified in this review.
Is code or data available for FORGE?
The reviewed manuscript does not announce a code or dataset release. However, the appendix discloses full agent prompts, exact benchmark splits with seeds, synthesis configurations, and complete GRPO hyperparameters, which would substantially aid reimplementation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 13-16 (candidate lineage graph, synthetic-example cards) and Figure 4's GRPO trajectory shapes are only partially recoverable from extracted text; GRPO endpoints were taken from Table 4.
- Figure 1 values are interleaved with body text in the extraction; per-benchmark numbers were cross-checked against Tables 1 and 3, but figure-only details could not be visually verified.
- Review covers arXiv v1 (14 Sep 2026) as extracted; no supplementary files beyond the in-paper appendix were visible.
- Benchmarks named AIME-2025, Qwen3.5-35B-A3B, GPT-5.5, DeepSeek-V4-Pro, and several 2026-dated cited works could not be verified against external sources under the no-network constraint.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.