DeltaSelect: Affordable A/B Testing for Coding Agents
cs.SE · cs.AI · cs.LG
Quick verdict
Budget-constrained task subsets enable cheap coding-agent A/B tests; honest paper, but validation is in-sample and no trivial-selector baseline is run.
Summary
DeltaSelect is a method for making frequent baseline-versus-candidate comparisons of coding agents affordable. Instead of rerunning an expensive benchmark, it mines DeepSWE's published trial matrix (113 tasks, 50 model-effort configurations, four runs each) to rank tasks by the fifth percentile of the resampled Pearson correlation between one-run results and the full-benchmark score; only 22 of 113 tasks reach 0.50. It then maps fractional fail-to-pass (F2P) verifier results onto the full-score scale with per-task linear regression, weights tasks by inverse calibration error, and greedily selects ranked tasks fitting a dollar budget, freezing the specification before use. A published-data example projects that nine tasks ($113.08) can separate Claude Opus 5 medium from Gemini 3.6 Flash high, 27.1x cheaper than the full 898-trial protocol. Harness validation shows published mini-swe-agent estimates fail to transfer to Codex (score gaps of 21.0-26.1 points; costs 1.5-3.5x higher), motivating a mandatory target-harness baseline. A 13-evaluation case study revising Agent Layer's skills cut recorded cost 58.1% ($4.18 to $1.75, p=0.008) while the calibrated score rose 5.9 points (p=0.326, not significant, using variance transferred from another harness). Key limitations: selection, calibration, and validation share one in-sample snapshot; the selector is never compared with random or cheapest-first alternatives; endpoints rest on one run per task; and the case study covers one model, one harness, and the author's own tool.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Only 19.5% of DeepSWE tasks (22/113) provide reliable one-run signal, defined as a fifth-percentile Pearson r of at least 0.50 with the full-benchmark score.
- Repeated public trials suffice to build a small, fixed, budget-constrained task set for repeated baseline-versus-candidate comparisons during development.
- Published benchmark scores and prices do not transfer across harnesses (21.0-26.1-point score gaps; 1.5-3.5x costs), so a target-harness baseline is mandatory.
- In the Agent Layer case study, the adopted skills configuration cost 58.1% less ($1.75 vs $4.18; p=0.008) with a higher but statistically non-significant calibrated score (42.36% vs 36.46%; p=0.326).
- A nine-task prefix ($113.08) is projected to distinguish two configurations 20.3 points apart, 27.1x cheaper than the full 898-trial protocol.
Strengths
Unusually disciplined claim scoping
The paper repeatedly fences its statistics: the fifth percentile is 'not a 90% confidence bound' (Sec 3.2), the two-arm test is 'retrospective... not preregistered' (Sec 3.6), and the case-study score gain is reported as not significant (p=0.326, Sec 5.2). This honesty extends to the limitations section and raises trust.
Harness-transfer evidence is quantified and decision-relevant
Section 5.1/Figure 5: same model, tasks, and effort shifted scores +21.0/+26.1/+3.0 points and costs 1.5-3.5x between mini-swe-agent and Codex; an unchanged repeat scored 14.84% vs 20.43%. This empirically justifies the mandatory target-harness baseline.
Reproducibility engineering
Design choices are evidence-backed within the snapshot
Transparent end-to-end cost accounting
Weaknesses
Entirely in-sample evidence for the headline claims
Selector value never tested against trivial baselines
Fragile endpoint inference
Unaddressed self-component circularity
Single-case, self-referential evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
DeltaSelect (arXiv 2609.19607, N. Conn) addresses a narrow, well-posed measurement problem: developers iterating on coding agents need cheap, repeatable baseline-versus-candidate evidence, while full suites such as DeepSWE cost thousands of dollars per protocol and run in a harness that differs from production. The paper claims that (1) most DeepSWE tasks are noisy one-run proxies — only 22 of 113 tasks reach a fifth-percentile Pearson r of 0.50 against the full score, with a median of 0.321 (Section 1, Figure 1); (2) a deterministic pipeline — resampled-correlation ranking, per-task OLS calibration of fractional F2P onto the full-score scale, inverse-residual weighting, and a greedy dollar-budget scan with a frozen specification — yields a usable fixed task set (Section 3); (3) published scores and prices do not transfer across harnesses (Section 5.1); and (4) a real 13-evaluation iteration cut cost 58.1% with a non-significant score gain (Sections 4.2, 5.2). The claims are deliberately fenced: no leaderboard, no model ranking, no unseen-task generalization. This restraint is genuine and consistently enforced, which is rarer than it should be.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DeltaSelect?
DeltaSelect is an open-source method that chooses a small, fixed subset of benchmark tasks within a dollar budget so developers can repeatedly A/B test changes to a coding agent (skills, prompts, models) at a fraction of full-benchmark cost. It ranks tasks by how consistently a single run tracks the full-benchmark score and calibrates their scores onto one common scale.
How much cheaper is DeltaSelect than running the full benchmark?
In the paper's published-data example, nine selected tasks costing $113.08 across two configurations were projected to detect a known 20.3-point score difference that the full 898-trial DeepSWE protocol ($3,063.63) clearly shows — a 27.1x reduction in scored-trial cost. This is a projection from published data, not a newly run experiment.
Does DeltaSelect replace full benchmarks or produce model rankings?
No. The authors explicitly state it is not a leaderboard, not a universal model ranking, and not a replacement for comprehensive evaluation. It only supports repeated comparisons of two versions of the same agent system on a frozen task set, and it requires running a baseline in the actual target harness first.
What did the Agent Layer case study actually show?
Across 13 evaluations of Agent Layer's skills on eight fixed tasks with gpt-5.6-luna, the adopted configuration cost 58.1% less ($1.75 vs $4.18; statistically significant, p=0.008) and scored higher (42.36% vs 36.46%), but the score gain was not statistically significant (p=0.326) and relied on variance estimates transferred from a different harness.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only through captions in the extracted text; point-level figure content could not be independently checked.
- External artifacts (the DeepSWE v1.1 snapshot, agent-layer.dev, the GitHub repository) were not verifiable from the text; availability claims are taken as stated.
- Minor extraction artifacts exist (e.g., stray footnote markers merged into sentences), but content appears complete through the references.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.