SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
Jennifer Williams, Dave Farris, Jeff Farris, Jiantao Jiao
cs.AI · cs.SE
Quick verdict
SWE-Serve benchmarks agents on SGLang production-inference tasks; rigorous pipeline and E2E gap, but matched-control CI includes zero, tempering causality.
Summary
SWE-Serve is a benchmark of 53 repository-grounded tasks for evaluating coding agents on production inference engineering, built from 83 merged SGLang changes since December 2025 across six families (model enablement, speculative decoding, kernels/quantization, caching/runtime state, distributed execution, serving APIs). Tasks ship a containerized repo at a base commit, an instruction, hidden fail-to-pass/pass-to-pass tests, and, for 19 tasks, 276 model-serving end-to-end (E2E) tests; performance gates appear in only 3 tasks; 41 tasks need one H100, 12 are CPU-only. Validity rests on no-op/oracle controls, agent-assisted adversarial probing, and human adjudication (53 of 156 candidates admitted). Under closed-book execution with trajectory audits, 11 models in 31 model-effort configurations were run three times with mini-SWE-agent; Claude Opus 5 (max) and GPT-5.6 Sol (max) lead at 75% mean pass@1, a 40-point spread. The core finding: on the 19 E2E-covered tasks, pass rate rises from 45.9% to 69.4% (+23.4pp) when E2E tests are removed from scoring, but a matched test-removal control yields only a 2.1-point differential with a 95% interval of −0.5 to 4.3, crossing zero. Limitations: a single codebase (SGLang), K=3 runs, single-GPU scope, and no private held-out set against training-data exposure.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SWE-Serve is the first benchmark targeting repository-scale production inference engineering, with 53 tasks from 83 merged SGLang PRs across six engineering families.
- Under closed-book execution with a shared harness, the best configurations (Claude Opus 5 max, GPT-5.6 Sol max) reach 75% mean pass@1, with a 40-point spread across 11 models.
- Removing model-serving E2E tests from scoring raises pass rate from 45.9% to 69.4% (+23.4pp) on the 19 E2E-covered tasks, with increases in all 11 top-per-model configurations.
- E2E test removal produces 2.0× as many fail-to-pass transitions as matched non-E2E removal (16.1 vs 8.0 across 10,000 randomized pairs, 92.2% favoring E2E).
- Tasks whose verifiers test multi-domain behavior, persistent state, or concurrent coordination show 21.3, 20.8, and 29.0pp lower mean pass rates, respectively.
Strengths
Disciplined task-admission pipeline with full non-admission accounting
786 source records → 203 candidates → 156 task candidates → 53 admitted (34%); the 103 rejections are itemized by reason (Figure 6, Appendix A.1.5), and one review round flagged 27 candidates for verifier changes. No-op rejection and oracle acceptance hold for 53/53 tasks (Table 6).
Pre-specified task characterization before any model outcomes
Runtime-domain breadth, persistent state, and concurrent-concurrency labels were assigned from task instructions and verifier tests only, before examining results (A.1.4), with a four-domain taxonomy grounded in documented components of vLLM, SGLang, TensorRT-LLM, and Triton (Table 4).
Empirically motivated evaluation-integrity design
The E2E-gap claim is tested against a matched control, not just asserted
Resource-aware leaderboard and harness sensitivity
Weaknesses
Headline gap rests on a control whose differential interval includes zero
Single-codebase scope with vendor proximity
E2E-only tasks mechanically inflate the raw gap
Thin statistical power for a leaderboard paper
Subgroup analyses are descriptive only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SWE-Serve (NVIDIA / UC Berkeley) is a benchmark paper for agentic coding evaluation in production inference engineering. It makes three claims: (1) existing benchmarks leave a coverage hole — repository-level SWE suites (SWE-bench Verified/Pro, DeepSWE) and terminal-agent benchmarks (Terminal-Bench 3: only 3/74 inference tasks) do not target inference, while kernel/perf benchmarks (KernelBench, SOL-ExecBench, ISO-Bench, InferenceBench) stop at optimization; (2) SWE-Serve fills it with 53 tasks from 83 merged SGLang PRs across six families, executable on CPU or one H100; (3) agents show a measurable production correctness gap. The headline number is +23.4pp (45.9%→69.4%) when model-serving E2E tests are removed from scoring on 19 tasks, rising in all 11 top-per-model configurations.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SWE-Serve?
SWE-Serve is a benchmark of 53 repository-level coding tasks built from real merged changes (83 pull requests) to SGLang, an open-source LLM inference-serving system. It tests whether AI coding agents can implement production inference features — model support, decoding, caching, serving APIs — correctly enough to pass hidden functional, regression, and end-to-end serving tests.
Which models score best on SWE-Serve?
Claude Opus 5 at max reasoning effort and GPT-5.6 Sol at max effort lead with 75% mean pass@1 (three runs each, mini-SWE-agent harness, closed-book). The lowest is Inkling S at 35%. Among four configurations tied at 64%, per-task cost varies 7.6× ($0.95–$7.24).
What is the production correctness gap the paper reports?
Many agent patches pass all local tests but fail end-to-end tests that exercise a live SGLang server through its public interfaces. On the 19 tasks with such E2E tests, removing the E2E tests from scoring lifts pass rate from 45.9% to 69.4% (+23.4pp). A matched control removing equal numbers of non-E2E tests shows a much smaller E2E-specific differential (2.1 points, 95% interval −0.5 to 4.3), so the raw gap partly reflects test count and difficulty.
Is SWE-Serve publicly available and can it be reproduced?
The authors state that the benchmark, execution environments (Harbor 0.13.1 containers), and verifiers are released, with pinned harness versions and randomized seeds. Reproducing the full leaderboard requires a CPU or one NVIDIA H100 per task plus paid API access to frontier models (roughly $0.33–$17.40 per task attempt); the 12 CPU-only tasks are the cheapest entry point.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (1, 2, 4–18) are available only as extracted text labels; plot-level claims were checked against stated numbers, not the rendered plots.
- Wide tables (Table 5 task characterization, Table 10 task-level outcomes) may have row/column misalignment from PDF extraction; per-task cells were not individually verified.
- Numbers are taken from the provided extraction without external verification against the released artifacts.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.