AgentFactory: Towards Automated Agentic System Design and Optimization
Enci Zhang, Haofeng Wang, Yuesheng Zhu, Xiaole Cui, Guibo Luo
cs.AI
Quick verdict
Auto joint model+workflow design claims +9.1 pts over ADAS/AFlow on 8 benchmarks; promising, but no validation split, no latency eval, no code release.
Summary
AgentFactory (Peking University Shenzhen) targets automated design of LLM-based agentic systems, arguing prior automated approaches (ADAS, AFlow) optimize workflows only and score single metrics. It formalizes agent design as multi-objective optimization over a joint space of fine-tuned models (base model x dataset x tuning method x hyperparameters) and code-based workflows, using a powerful LLM as optimizer within a three-stage pipeline (planning, fine-tuning configuration, workflow code generation), terminating when accuracy and cost targets are met or after 20 iterations. On eight benchmarks across five domains (MMLU, DROP, HumanEval, MBPP, GSM8K, MATH-L5, MedQA, FinEval) in two settings (Llama-3.1-8B-Instruct; GPT-4o-mini, with AgentFactory allowed both models), it reports +9.1 average points over automated baselines and +19.6/+18.7 points on MedQA/FinEval (Llama setting) at lower inference cost than ADAS/AFlow; optimizer choice matters, with GPT-4o best. Main limitations: no validation/test separation is described for the 20-step search, raising adaptive-overfitting concerns; latency is claimed but never measured; baselines get a restricted action space; Table 1's reported averages do not match the arithmetic means of the listed per-benchmark scores; three repeats are mentioned without variance or significance tests; optimization-side costs (optimizer calls, fine-tuning compute) are excluded; and no code is released.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Jointly optimizing fine-tuned models and workflow structures outperforms workflow-only automated design (ADAS, AFlow) by 9.1 points on average across eight benchmarks.
- Incorporating fine-tuning into the automated search yields large domain-specific gains: +19.6 points on MedQA and +18.7 on FinEval in the Llama-3.1-8B setting.
- The multi-objective search maintains or lowers inference cost relative to automated baselines (e.g., $0.68 vs $2.13 for AFlow in the GPT-4o-mini setting, a 68% reduction).
- Optimization quality correlates with optimizer LLM capability but is robust across comparable frontier models (GPT-4o 68.7 vs Claude-3.5-Sonnet 67.1 average).
Strengths
Joint model+workflow search is a genuine step beyond workflow-only automation, with ablation support
Sections 3.1-3.3 formalize a search space S = M x W where M spans base model, dataset, tuning method and hyperparameters; the ablation (Section 5.3, Fig. 5) reports the full pipeline beating fine-tuning-only by 12.7 and workflow-only by 8.2 average points, and the domain gains (MedQA/FinEval) land exactly where fine-tuning should matter.
Multi-objective formulation with explicit termination targets is practically relevant
Algorithm 1 uses component-wise target thresholds (Eq. 6) and scalarization (Eq. 7); the Section 5.2 FinEval case study shows a concrete trade-off trajectory: accuracy 62.3 -> 66.7 -> 72.5 -> 74.6 -> above the 75% target while cost was cut from $1.14 to $0.60, terminating at iteration 17.
Unusually broad empirical coverage for this literature
Above-average implementation transparency
Weaknesses
No validation/test separation is described for the search loop
Latency is claimed as an objective but never measured
Asymmetric action space vs baselines
Table arithmetic and statistical reporting are unreliable
Optimization costs excluded and no code release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AgentFactory (Peking University Shenzhen Graduate School; arXiv 2609.01045, cs.AI) is a method paper on automating the design of LLM-based agentic systems. It positions itself against workflow-only automation (ADAS, AFlow, GPTSwarm) and prompt-only optimization (PromptBreeder, TextGrad), and makes three claims: (1) jointly searching fine-tuned models and workflow structures beats workflow-only search; (2) optimizing multiple objectives (accuracy, cost, latency) matters for real deployment; (3) empirically, +9.1 average points over automated baselines across eight benchmarks, +19.6 on MedQA and +18.7 on FinEval (Llama-3.1-8B setting), at lower inference cost. The formalization (Section 3.1, Eqs. 1-7) is clean: a solution space S = M x W, an LLM optimizer generating candidates from the solution-evaluation trajectory, component-wise termination targets, and scalarization for the final pick.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AgentFactory?
A framework from Peking University researchers that automatically designs LLM-based agentic systems: a powerful LLM acts as an optimizer that iteratively (up to 20 steps) selects and fine-tunes foundation models and generates workflow code, trying to meet user-set accuracy and cost thresholds through a three-stage pipeline of planning, tuning configuration, and workflow code generation.
How much does AgentFactory improve over prior methods?
The paper reports an average gain of 9.1 points over automated design methods (ADAS, AFlow) across eight benchmarks, with the largest gains on domain tasks in the Llama-3.1-8B setting (+19.6 on MedQA, +18.7 on FinEval), while its inference cost ($0.26-0.68 per evaluation) is lower than AFlow's ($0.29-2.13). These are the authors' reported figures under their own evaluation setup.
Is AgentFactory's code available?
No. The paper (arXiv 2609.01045) contains no code-release statement or repository link. It does publish the optimizer prompt template (Fig. 2), the algorithm, and the fine-tuning dataset list, but not the search harness, workflow code templates, or evaluation scaffolding.
What benchmarks does AgentFactory evaluate on?
Eight public benchmarks across five domains: MMLU and DROP (general knowledge/reading, 1,000 samples each), HumanEval and MBPP (coding, full test sets), GSM8K and a 617-problem MATH level-5 subset (mathematics), MedQA (medicine, full test set), and FinEval (finance, full test set).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided extraction ends inside reference 39 (the GPTSwarm citation lacks its year/pages); any appendix beyond the references is not present in the text.
- Figures 4 and 5 are visual: exact per-benchmark ablation values and optimization-curve data are not extractable, so ablation judgments rely on the Section 5.3 prose (12.7 and 8.2 average-point margins).
- Tables 1 and 2 were extracted as flat number runs; individual digit/column assignments could be garbled, so the reported Avg-vs-computed-mean discrepancies should be re-verified against the PDF before being treated as confirmed errors.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.