RestoreBench: Can AI Agents Restore Power Flow Convergence?
Riccardo Mansutti, Andrea Pomarico, Robert Jakob, Qian Zhang, Alberto Berizzi, Kevin O'Sullivan
cs.AI · eess.SY
Quick verdict
RestoreBench: a well-specified agentic benchmark for restoring non-convergent power flows; rigorous design, but n=1 runs per config weaken comparative claims.
Summary
This paper introduces RestoreBench, a benchmark asking whether LLM-based AI agents can diagnose and repair non-convergent AC power flow cases caused by insufficient reactive-power support. The environment is deterministic: pandapower 3.4.0 with a locked Newton-Raphson policy, a success contract excluding bus voltages, and an atomic action space of generator setpoint steps, shunt switching, and tap changes. Datasets of 46 cases per grid are generated for the IEEE 118-bus and PEGASE 89-bus systems via a pipeline that stresses electrical pockets beyond a measured convergence boundary, admits only reactive-deficit cases, and certifies an offline ground-truth solution of at most 10 maneuvers for each. Seven LLMs (GPT-5.6 Sol, Claude Haiku/Sonnet/Opus 5, DeepSeek V3.2, Kimi K2.5, GLM-5) are evaluated as chatbots, single agents, and Analyst-Executor-Orchestrator multi-agent systems using four metrics: success rate, cost, runtime, and in-band buses. Across both grids, chatbots perform worst, tool-bearing agents improve success, and the multi-agent design raises cost and runtime without systematically improving over a single agent; DeepSeek V3.2 suffers structured-output failures. The chief limitation, acknowledged by the authors, is a single run per configuration (about $978 total), so reported differences carry no variance estimates.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Tool-augmented single- and multi-agent architectures restore convergence on more cases than a no-tool chatbot across both test grids.
- The multi-agent Analyst-Executor-Orchestrator design increases cost and runtime without systematically improving success rate over a single agent.
- Non-convergence from reactive-power deficit can be formalized as a certifiable agentic task: every benchmark case has a verified ≤10-maneuver solution within the allowed action space.
- Failure modes are strongly model-specific — DeepSeek V3.2's tool-architecture losses trace to empty structured terminal outputs, not bad electrical decisions.
Strengths
Fully deterministic, contract-driven environment
Appendix A locks the entire solver policy (pandapower 3.4.0, Newton-Raphson, enforce_q_lims PV→PQ switching, DC init, dual tolerances 10⁻⁸/10⁻⁶ MVA) and every terminal verdict comes from the simulator, not an LLM (A.1-A.2).
Certified recoverability of every case
Appendix B.4.1 runs an offline search restricted to the agent's own action space and retains only cases solvable within ≤10 maneuvers, with the solution stored as private ground truth — no impossible tasks and a direct/sequential difficulty taxonomy.
Unusually rigorous dataset construction
Honest self-assessment and failure decomposition
Cross-network replication of findings
Weaknesses
Single stochastic run per configuration
Environment does much of the search — no no-LLM baseline
Narrow task and grid coverage
Voltage quality outside the success contract
Model versions and budgets not fully pinned
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RestoreBench targets a real gap: prior agentic power-system benchmarks (PowerAgentBench-SS/Dyn [7,8], PowerChain [9]) assume solvable simulations, while non-convergent cases still demand iterative diagnosis. §1 claims three contributions: a general agentic benchmark formulation, an evaluation of seven LLMs, and a comparison of chatbot, single-agent, and multi-agent architectures. The scope is deliberately narrow — only reactive-deficit non-convergence (§2.1), three atomic action families, two small grids (IEEE 118-bus, PEGASE 89-bus), 46 cases each. Headline claims: tool-bearing agents beat chatbots; multi-agent coordination adds cost without systematic gains.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RestoreBench?
RestoreBench is a benchmark that measures whether LLM-powered AI agents can diagnose why an AC power flow simulation fails to converge and restore convergence using limited grid controls — generator voltage setpoint steps, shunt switching, and transformer tap changes — across 46 cases on each of two standard test grids (IEEE 118-bus and PEGASE 89-bus).
Which AI models were tested in RestoreBench?
Seven LLMs — GPT-5.6 Sol, Claude Haiku 4.5, Claude Sonnet 5, Claude Opus 5, DeepSeek V3.2, Kimi K2.5, and GLM-5 — each run in three setups: a no-tool chatbot, a single tool-using agent, and a three-role (Analyst-Executor-Orchestrator) multi-agent system.
What is RestoreBench's main finding?
Tool-using agents restore convergence on more cases than chatbots, but the single-agent setup offers the best cost-performance trade-off: the multi-agent architecture roughly doubles cost without consistently improving success. DeepSeek V3.2 often fails the structured-output protocol required by the tool interface rather than making bad electrical decisions.
Is the RestoreBench code and data public?
Yes. The paper states that the exact prompts, tool descriptions, datasets, and implementation are released at github.com/Mansutti081/RestoreBench, and because the environment is deterministic (pandapower-based), the benchmark cases and ground-truth solutions can be independently re-verified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Numeric success rates, in-band bus percentages, and runtime statistics for the IEEE 118-bus runs appear only inside figures (Figs. 2–3, 6–9); their values are not in the extracted text, so exact per-model numbers can be quoted only for PEGASE 89 (Appendix F) and the cost table.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.