CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Engine
Shutong Wu, Kevin Calderone, Andy Tsen
cs.AI
Quick verdict
Solid no-LLM-judge Unreal agent benchmark; C++ beats Blueprint by 30-43pp; single-attempt results lack significance tests and MCP2 comparison is confounded.
Summary
The paper introduces CraftBench-UE, a harness that evaluates coding agents on bounded gameplay and editor tasks in Unreal Engine 5.8. Agents author changes in an isolated project; the harness replays only saved, permitted changes into a fresh project and grades them with deterministic build (L1), runtime (L2, fixed-timestep Play-in-Editor assertions via Unreal Functional Tests), and asset-structure (L2I) checks, with no LLM judge. The benchmark has 70 tasks: 33 C++, 25 Blueprint, 12 editor-scripting, including 10 gameplay pairs where C++ and Blueprint versions share the starting project and runtime tests. Seven models were run once per task under two editor-MCP configurations plus a file/shell baseline on C++ (pass@1, 40-minute budget). C++ completion reached 86.1%/87.9% under MCP1/MCP2 versus 77.9% baseline; Blueprint 62.9%. On paired tasks C++ exceeded Blueprint by 30.0/42.9 percentage points, and among on-time Blueprint submissions passing asset checks, 42.2% (19/45) and 50.0% (20/40) failed runtime assertions. Trajectory analysis shows accepted graph revisions and asset readbacks often fail to predict correct gameplay. Limitations: one attempt per run with no significance testing, tool-permission and driver-version differences confounding the MCP comparison (the authors' proprietary MCP2), tasks selected for checkability, and a prompt-level one-playtest cap. Release of harness, tasks, verifiers, and trajectories is promised.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Deterministic build, asset, and runtime checks — without an LLM judge — can grade agent work on bounded Unreal Engine gameplay tasks (§3.3, §3.5).
- On 10 matched gameplay pairs, C++ completion exceeds Blueprint by 30.0pp (MCP1) and 42.9pp (MCP2) within the 40-minute budget, with every model completing more C++ than BP (§5.1, Tables 3/7).
- The Blueprint deficit is not explained by asset checks: counting runtime passes only leaves all four completion counts unchanged (Table 9).
- Among on-time asset-passing Blueprint submissions, 42.2% (19/45, MCP1) and 50.0% (20/40, MCP2) fail explicit runtime assertions (§5.2, Table 14).
- Local editing success — accepted graph revisions, asset readbacks, compilations — does not predict completed gameplay (§5.3, Tables 11-12).
Strengths
Judge-free deterministic grading with validity safeguards
§3.3 defines three auditable check levels (L1 build, L2 fixed-1/60-timestep PIE assertions via Unreal Functional Tests, L2I headless asset inspection); §3.5 admission requires a reference solution to PASS and the unmodified project to FAIL, and Appendix B.1 monitors withheld verifier paths with run-exclusion on exposure — directly addressing the test-validity concerns SWE-bench Verified raised.
Paired C++/Blueprint design isolates the deliverable variable
The 10 gameplay pairs (Table 17) share starting project and runtime verifier, differing only in required deliverable. Table 9 is the sharpest result: counting runtime passes only leaves all four completion counts unchanged (within deadline no BP submission passes runtime but fails assets), cleanly showing the BP deficit is in gameplay implementation, not asset checks.
Unusually candid attribution boundaries
Trajectory analysis tied to graded outcomes
Strong reproducibility commitments
Weaknesses
No statistical inference anywhere
MCP configuration comparison is confounded and conflicts with author interest
Task selection opacity
Budget- and instruction-dependent headline numbers
Unexplained per-model anomalies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CraftBench-UE is both a harness and a 70-task benchmark (33 C++, 25 Blueprint, 12 editor-scripting; 10 C++/BP gameplay pairs) that evaluates coding agents inside Unreal Engine 5.8 on Windows via Claude Code. Its claims: (1) deterministic, LLM-judge-free grading via build (L1), runtime (L2, fixed-1/60-timestep PIE assertions through Unreal Functional Tests), and asset (L2I) checks is feasible for engine tasks; (2) on 10 matched gameplay pairs, C++ completion beats Blueprint by 30.0pp (MCP1: 46/70 vs 25/70) and 42.9pp (MCP2: 50/70 vs 20/70); (3) the BP deficit is not an artifact of asset checks — runtime-only counting leaves all four counts unchanged (Table 9); (4) 42.2% (19/45) and 50.0% (20/40) of on-time, asset-passing BP submissions fail runtime assertions. Positioning against GameEngineBench (LLM judge) and AutoUE (LLM-judged) is fair and documented in Appendix G.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CraftBench-UE?
A benchmark and harness that evaluate AI coding agents on bounded gameplay and editor tasks inside Unreal Engine 5.8. An agent's saved changes are reconstructed in a fresh project and graded with deterministic build, asset, and runtime checks — no LLM judge — across 70 tasks: 33 C++, 25 Blueprint, and 12 editor-scripting tasks.
Are AI coding agents better at Unreal C++ or Blueprints?
According to this paper, C++ is substantially easier for the tested agents: on 10 matched gameplay tasks, C++ completion exceeded Blueprint by 30.0 percentage points (Unreal MCP config) and 42.9 points (Aura MCP config) within a 40-minute budget, and 42-50% of on-time asset-passing Blueprint submissions still failed runtime gameplay assertions.
Which AI models were evaluated in CraftBench-UE?
Seven models, each run once per task under two editor-MCP configurations plus a file/shell baseline on C++ tasks: Claude Sonnet 5, DeepSeek V4 Pro-0813, Gemini 3.7 Flash, GPT-5.6 Luna, GPT-5.6 Sol, Grok 4.6, and GLM-5.3 Flash, all driven through Claude Code on Windows.
Is CraftBench-UE open source?
The authors state the harness, 70-task benchmark, verifiers, reference implementations, and agent trajectories will be released at github.com/ramenvr/craftbench-ue upon publication. Users must supply their own Unreal Engine installation; the proprietary Aura MCP (MCP2) internals and engine binaries are not redistributed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-7 appear only as extracted text labels; visual plot details (e.g., Figure 4's checkpoint traces, Figure 7's duration distributions) could not be independently verified.
- Some figure-embedded numbers (Figure 5/6 labels) were cross-checked against Tables 6 and Appendix text; minor misalignment from extraction is possible.
- The embedded text appears complete for main text and appendices A-G; no truncation affecting headline numbers was detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.