GraphSkillEvo: Evolutionary Optimization of Graph-Structured Agent Skills
Rui Sun, Zhi Zheng, Zhenkun Wang, Zhichao Lu
cs.LG
Quick verdict
Graph-structured agent skills + population-based evolution; beats SkillOpt in 13/14 benchmark settings, but gains are modest with mixed significance.
Summary
The paper argues that LLM agent skills should be represented as graph-structured natural-language artifacts: global guidance, reusable nodes (execution steps with instructions), and edges induced by conditional workflows. Building on this, GraphSkillEvo is a population-based evolutionary framework that evolves four skills over five generations using four operators — global-guidance and graph-structure mutation (driven by failed-trajectory reflection) plus global-guidance and graph-structure crossover — with rank-based parent selection, full-validation fitness, and a schema validator that discards malformed skills. Across five benchmarks (SearchQA, SpreadsheetBench, DocVQA, LiveMath, ALFWorld), two LLMs (GPT-5.4, GPT-5.4-nano) and two harnesses, it ranks best in 13 of 14 settings, beating SkillOpt by 4.01% average accuracy (GPT-5.4-nano, no harness), 1.76% (GPT-5.4, no harness) and 1.33% (GPT-5.4, Codex harness), while SkillOpt consumes 1.31-1.36x more optimization tokens. Ablations (three benchmarks) attribute the gains mostly to graph structure (-7.44 average without it) and mutation (-17.02), with crossover contributing -4.93; skills transfer from GPT-5.4-nano to GPT-5.4, notably on SpreadsheetBench. Main limitations: SkillOpt is the only optimization baseline; Welch's t-tests reach p<0.05 on only two of four tested benchmarks (SearchQA 0.069, DocVQA 0.089, no multiplicity correction); the five-run significance means differ from the three-run Table 1 numbers; validation sets are tiny (18-53 instances), making fitness estimates noisy.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Representing skills as graph-structured artifacts (global guidance + step nodes + conditional workflows) improves agent execution versus unstructured counterparts: removing structure costs 0.75 to 4.52 points across five benchmarks (Table 3).
- Population-based evolution with four structure-aware operators beats SkillOpt, ranking best in 13 of 14 model-harness-benchmark settings with average gains of 4.01% (GPT-5.4-nano), 1.76% (GPT-5.4) and 1.33% (Codex harness).
- GraphSkillEvo is more token-efficient: SkillOpt consumes 1.31-1.36x more optimization tokens for worse results (Table 2).
- Optimized skills transfer from GPT-5.4-nano to GPT-5.4, with transferred GraphSkillEvo skills beating transferred SkillOpt skills, most notably on SpreadsheetBench (71.78 vs 53.21, Table 5).
Strengths
Search-space integrity is enforced, not just assumed
A schema validator (App B.3) discards and regenerates malformed skills, keeping the structured search space well-formed; operator prompts (E.1), initialization prompt (E.2), a full optimized skill (E.3) and its unstructured counterpart (E.4) are printed in full, which few competing papers do.
Broad evaluation grid with an efficiency result
Table 1 covers 5 benchmarks x 2 LLMs x 2 harnesses with GraphSkillEvo best in 13 of 14 settings; Table 2 shows SkillOpt needs 1.31-1.36x more optimization tokens (81.08M vs 61.94M on GPT-5.4; 103.54M vs 75.94M on nano) for worse results, and Fig 3 plots convergence against token spend.
Cleanly factored ablations
Honest reporting of negative and limiting results
Weaknesses
Single optimization baseline
Statistical evidence is thin and internally inconsistent
Selection on very small validation sets
The graph representation is shallower than the framing
Token accounting omits optimization overhead items
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GraphSkillEvo (arXiv 2609.21749, cs.LG) addresses skill optimization for LLM agents: iteratively improving a reusable natural-language skill artifact attached to a fixed LLM plus execution harness. The thesis has two parts: (1) unstructured skills lack explicit workflow guidance and make the optimization search space large and redundant; (2) representing a skill as global guidance h_s plus a directed graph — nodes are execution steps with self-contained instructions, edges are induced by M workflows (an applicability condition plus an ordered node path) — fixes both, and a population-based evolutionary framework with four structure-aware operators (guidance/graph mutation, guidance/graph crossover) then outperforms SkillOpt's single-skill LLM self-refinement. Key quantitative claims: best result in 13 of 14 settings across five benchmarks (SearchQA, SpreadsheetBench, DocVQA, LiveMath, ALFWorld), two models (GPT-5.4, GPT-5.4-nano), two harnesses (none, Codex; ALFWorld excluded under Codex because the adapter lacks persistent environments); average gains over SkillOpt of +4.01% (nano, no harness), +1.76% (GPT-5.4, no harness), +1.33% (GPT-5.4, Codex); SkillOpt consumes 1.31-1.36x more optimization tokens (Table 2); skills transfer from nano to GPT-5.4 (Table 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is GraphSkillEvo?
It is a framework that represents LLM-agent skills as graph-structured documents — global guidance, reusable execution-step nodes, and conditional workflow paths — and optimizes them with a population-based evolutionary algorithm using mutation and crossover operators. It was evaluated on five agent benchmarks with GPT-5.4 and GPT-5.4-nano.
How much does GraphSkillEvo improve over SkillOpt?
On average across five benchmarks it adds +4.01% accuracy on GPT-5.4-nano (no harness), +1.76% on GPT-5.4 (no harness) and +1.33% on GPT-5.4 with the Codex harness. It is the best method in 13 of 14 tested settings, though SkillOpt wins on LiveMath with GPT-5.4-nano.
Are the improvements statistically significant?
Only partially: one-sided Welch's t-tests over five optimization runs give p<0.05 for SpreadsheetBench (0.016) and ALFWorld (0.038), but SearchQA (0.069) and DocVQA (0.089) do not reach significance, and no multiple-comparison correction is reported.
Is the code available?
The paper states code is available at github.com/ruisun7/GraphSkillEvo. Note that all experiments depend on proprietary GPT-5.4 and GPT-5.4-nano APIs, which may affect long-term reproducibility.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 appear only as garbled inline text; figure layouts (e.g., Fig 3 curve shapes) could not be independently analyzed, only their captions.
- Appendix C.2 case-study trajectories are partially garbled but recoverable in structure (three task types).
- The stated GitHub repository (ruisun7/GraphSkillEvo) could not be verified from the extracted text alone.
- Table 7 formatting is mangled in extraction (method names run together), though all numeric values are legible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.