RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents
Aziz Ben Amor, Drish Mali, Mann Acharya, Vijayasri Iyer, Sébastien Bratières
cs.CL · cs.AI
Quick verdict
Open-source harness for repo-scale refactoring agents; useful controlled S1/S2/S3 design, but single-run, one-model ablations weaken headline claims.
Summary
The paper introduces RefactorPlatform, an open-source evaluation harness for repository-scale refactoring agents that holds the environment fixed while varying four axes: model backbone (via OpenRouter or GitHub Copilot CLI), execution regime, prompt specificity, and target dataset. Three cumulative regimes are compared: S1 baseline terminal assistant, S2 retrieval-augmented assistant (CocoIndex AST-aware chunking, nomic-embed-code embeddings, hybrid BM25/vector search fused by RRF), and S3 native sub-agent delegation. Verification is strict: AST-based unit tests for Python (RefactorBench, 100 tasks) and extraction-compilation-test plus RefactoringMiner gates for Java (SWE-Refactor). On qwen3.6-flash, AST chunking beats naive token-window chunking by 25-30 percentage points across prompt modes, and naive retrieval falls below the retrieval-free baseline (57% vs 73% in Descriptive mode). A lean S2 agent reaches 86% versus 66% for S3 delegation on matched tasks, with 20 discordant tasks and none reversed; cost per successful refactoring is unchanged by retrieval (~$0.13 for qwen3.6-flash). Cross-model S2 gains average +9.5pp over four model families. Limitations: single-run campaigns without uncertainty estimates, mid-tier models only, S3 exercised on one model with agent-initiated delegation (81/100 tasks), and Java results from small non-matched subsets (gpt-5-mini n=33/75), where a compiler-feedback loop lifted pass rate from 20% to 85%.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On qwen3.6-flash, AST-aware chunking beats naive token-window chunking by 25-30 percentage points across all three prompt modes (86/74/56 vs 57/44/31).
- A retrieval-augmented single agent (S2, 86%) beats sub-agent delegation (S3, 66%) on the same 100 Descriptive-prompt RefactorBench tasks, with 20 tasks passing only under retrieval and none the reverse.
- Retrieval's token overhead is absorbed by accuracy gains: cost per successful refactoring shifts by at most $0.01 when moving from S1 to S2 for all four model families (Table 3).
- Incomplete reference propagation dominates failures: partial completion plus missing imports account for 50.9% of RefactorBench failures, and compilation failures account for 73.5% of SWE-Refactor failures (Table 6).
- In the Java appendix, feeding compiler diagnostics back to the model lifts gpt-5-mini pass rate from 20% (n=75) to 85% (n=33) on non-matched subsets (Table 5).
Strengths
Honest S3 regime accounting
Because delegation is agent-initiated, the platform proves delegation from the event stream and marks runs conformant vs setup_not_exercised; Section 3.1 reports both the overall 66% and the conditional 70.4% on the 81/100 tasks where delegation actually fired, plus the 47.4% rate on the 19 non-delegated runs.
Behavior-preserving verification gates
Section 2.5 uses conjunctive gates: AST unit tests for Python; extraction + compilation + full test pass + RefactoringMiner for Java, explicitly to reject compile-only stubs; CodeBLEU is logged as diagnostic but excluded from pass/fail, avoiding a proxy-metric trap.
Actionable negative result on naive retrieval
Deployment-relevant cost accounting
Telemetry-grounded failure analysis
Weaknesses
No statistical uncertainty anywhere
S3 evidence rests on one model and one run
Java appendix comparisons are not matched
Cross-model table is Descriptive-only
Novelty asserted, not demonstrated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RefactorPlatform (arXiv 2609.04898) is an open-source harness for evaluating repository-scale refactoring agents under controlled variation of four axes: model backbone (OpenRouter BYOK or GitHub Copilot CLI), execution regime (S1 baseline terminal assistant, S2 retrieval-augmented, S3 native sub-agent delegation), prompt specificity (Descriptive/Base/Lazy, following RefactorBench), and target dataset (RefactorBench 100 Python tasks; SWE-Refactor Java in the appendix). Claimed contributions are the infrastructure itself, an AST-verification pipeline, and a 100-task demonstration campaign across four model families (qwen3.6-flash, minimax-m3, kimi-k2.6, deepseek-v4-pro). Headline empirical claims: AST-aware chunking beats naive token-window chunking by 25-30pp in every prompt mode; naive retrieval drops below the retrieval-free baseline; a lean S2 single agent (86%) beats S3 delegation (66%) with one-sided discordance (20-0); cost per success is unchanged by retrieval; cross-model S2 gain averages +9.5pp.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RefactorPlatform?
An open-source evaluation harness for LLM coding agents on repository-scale refactoring tasks. It runs each task in an isolated workspace and lets researchers vary the model backbone (via OpenRouter or GitHub Copilot CLI), the execution regime (baseline terminal agent, retrieval-augmented agent, or multi-agent sub-agent delegation), and prompt specificity, while logging tokens, diffs, and transcripts and verifying results with AST-based tests.
What are the paper's headline results?
On 100 RefactorBench Python tasks: AST-aware chunking beats naive token-window chunking by 25-30 percentage points; naive retrieval underperforms running with no retrieval at all (57% vs 73% in Descriptive mode on qwen3.6-flash); a retrieval-augmented single agent scores 86% vs 66% for sub-agent delegation; and cost per successful refactoring is essentially unchanged by retrieval (~$0.13 for qwen3.6-flash). Across four model families, retrieval adds about 9.5 percentage points on average.
Is the code and data available?
Yes. The platform is open-sourced at github.com/PiSchool/refactor-platform, and it evaluates the public RefactorBench (Python) and SWE-Refactor (Java) benchmarks. Because reported results are single runs, exact replication also depends on API-side model availability and pricing.
What are the main limitations of the study?
Three main ones, acknowledged by the authors: all campaigns are single runs with no uncertainty estimates; only mid-tier (non-frontier) models were tested; and the multi-agent (S3) comparison was run on just one model where the agent itself decided when to delegate (it did so on 81 of 100 tasks). Additionally, the Java results come from small, non-matched task subsets (e.g., n=33 vs n=75 for the gpt-5-mini comparison).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (dashboard screenshot) cannot be assessed from the extracted text; UI claims rest on the caption and Section 2.4 prose alone.
- Token pricing sources behind Table 3's $/pass figures are not stated in the text, so those numbers cannot be independently verified from the paper alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.