FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration
Weihang Ding, Junfei Zhan, Yueting Li, Qirong Guo
cs.SE · cs.AI
Quick verdict
Rigorous replay-graded benchmark for LLM deployment agents with strong anti-gaming gates; single-episode cells and n=1 human baseline limit statistical power.
Summary
FDE-Bench is a benchmark of 136 hand-authored deployment-configuration tasks, drawn from a 149-card blueprint corpus, spanning Docker images, multi-service Compose stacks, and Kubernetes in greenfield and diagnose-and-repair modes. Agents receive a ticket, a machine-readable specification, and a workspace, then submit declarative artifacts that are rebuilt and redeployed in a pristine sandbox and scored by four gated binary check layers — build, readiness, behavior, conformance, each weighted 0.25 — with no LLM judge. A four-arm release gate requires the reference solution to resolve while do-nothing, spec-transcription, and generic-stub submissions must fail; adversarial strategies resolve none of the 135 tasks they cover. Under a shared four-tool scaffold, seven models resolve 52.9–75.0 percent of tasks, led by Gemini-3.6-Flash; the ensemble union reaches 92.6 percent and ten tasks resist all agents. Repair tasks average 30.7 percentage points higher resolution than greenfield across all models, and readiness is the largest failure stage (110 of 313 unresolved episodes). The validity work is unusually thorough, including a spec-leak audit that found 58 of 61 constants exposed. Key limitations: one episode per task–model cell with no run-to-run variance estimates, an n=1 human baseline (92 vs 72 percent, p=0.125) on a richer scaffold, and timing-sensitive checks across heterogeneous hosts.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Replay grading through four gated binary programmatic layers (build/readiness/behavior/conformance) measures deployment capability without LLM-judge noise and grants no credit to imperative live state.
- The four-arm release gate plus adversarial arms demonstrate that no tested shortcut resolves any task: all zero-intelligence floors resolve 0 of 136 with mean DS between 0.217 and 0.438.
- Under a shared four-tool scaffold and fixed budgets, seven models resolve 52.9–75.0% of the 136 tasks; the ensemble union reaches 92.6% and ten tasks resist all seven agents.
- Repair tasks are uniformly easier: macro-averaged RR is 30.7 percentage points higher than greenfield, with a positive gap for every model.
- Readiness is the dominant failure stage, accounting for 110 of 313 unresolved episodes, refuting the hypothesis that health monitoring is hardest.
Strengths
Anti-gameability is engineered into task admission, not bolted on
The four-arm release gate (§3.2) requires the hidden reference and a second solution (resolved on 132/136 frozen tasks) to pass while do-nothing, spec-transcription, and generic-stub arms must fail; Figure 3 shows distinct floor lattices (stub modal 0.50 on 100 tasks, do-nothing 0 on 71) and no floor ever reaches 1.0. Three adversarial strategies resolve 0 of 135 covered tasks.
Replay grading removes the two classic benchmark confounds
Submission collects only graded-artifact globs, tears down live state, and redeploys over pristine fixtures (Eq. 1), so hand-started containers earn nothing; scoring is fully programmatic (Eq. 2) with a common binary lattice {0, 0.25, 0.50, 0.75, 1.00} and no LLM judge (§3.3).
Unusual defect transparency with quantified residue
Statistical honesty above field norm
Failure attribution is check-grounded, not narrative
Weaknesses
One episode per cell; no run-to-run variance anywhere
Human baseline is a single participant on a richer scaffold
Difficulty calibration missed its target and headroom is shrinking
Partial DS is interpretable only with the resolution lens
Hand-authored tasks with author-solved references
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FDE-Bench (arXiv 2609.27571) evaluates forward-deployed engineering: whether an LLM agent can turn fixed application code plus a machine-readable deployment specification into artifacts that build, become ready, behave correctly, and satisfy the specification. The grid is 136 tasks from a 149-card hand-authored blueprint corpus — 37 single-image, 43 Compose, 56 Kubernetes; 66 greenfield, 70 repair; axes C 49 / O 48 / H 34 / F 5 (Tables 2–3). Central claims: (1) replay grading with four gated binary layers measures deployment competence without LLM-judge noise; (2) the four-arm release gate makes shortcuts measurable — do-nothing, spec-transcription, and generic-stub arms resolve 0/136 with mean DS 0.217–0.438 (Fig 3); (3) under a shared four-tool scaffold, Gemini-3.6-Flash resolves 75.0% (Table 1), the ensemble union 92.6%, and ten tasks resist all seven models; (4) repair exceeds greenfield by 30.7pp macro-average for every model; (5) readiness is the modal failure (110 of 313 unresolved episodes). Claims are consistently scoped to the scaffold and the frozen v1.1 grid — a discipline the paper mostly maintains, with the human-baseline sentence in the abstract as the notable exception.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FDE-Bench?
FDE-Bench is a benchmark of 136 hand-authored deployment-configuration tasks (from a 149-card blueprint corpus) that tests whether LLM agents can produce Docker, Compose, and Kubernetes artifacts that build, become healthy, behave correctly, and match a deployment specification. It covers both building deployments from scratch (greenfield) and repairing deliberately broken ones.
How are submissions scored in FDE-Bench?
Submitted declarative files are rebuilt and redeployed from scratch in a pristine sandbox, then scored by four gated binary check layers — build, readiness, behavior, and spec conformance — each worth 0.25, using only programmatic checks with no LLM judge. A task counts as resolved only when all four layers pass; hand-started live containers earn no credit.
Which models perform best on FDE-Bench?
On the 136-task grid, Gemini-3.6-Flash leads with 75.0% resolution and Claude-Sonnet-5 follows at 72.8%; the lowest is Gemini-3.5-Flash-Lite at 52.9%. The paper's statistical tests do not support a significant difference between the top two (p=0.59), and the union of all models' solved tasks reaches 92.6%.
Why do repair tasks have higher scores than greenfield tasks?
Repair tasks start from a complete but deliberately broken environment, giving the agent a concrete failing system to diagnose, whereas greenfield tasks require authoring all configuration from source and a specification. Average resolution is 30.7 percentage points higher on repair (82.0% vs 51.3% macro-average), a gap that is positive for every one of the seven models tested.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are available only as embedded text descriptions; visual details (e.g., per-model failure mixes in Fig 2b) were assessed from captions and tables.
- The extraction appears complete through Appendix N, but supplementary artifacts (full prompts, check implementations) were not independently inspectable.
- Statements about the released repository's contents rest on the paper's reproducibility statement; the anonymous repository itself was not verified in this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.