GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development
Xiuhui Zhang, Yi Chen, Shusheng Xu, Fan Li, Huan Wang, Tongkai Yang, Binhang Yuan
cs.AI · cs.SE
Quick verdict
Game-generation benchmark requiring pre-declared behavioral checks; sound design, but single-run results and an unenforced N/A loophole weaken several claims.
Summary
The paper introduces GameASG-Bench, a benchmark of 47 browser-native game-generation tasks (12 genres; 32 in 2D, 15 in 3D) that makes behavioral testability part of autonomous software generation. Before generation, task authors fix an evaluation interface specification - window.__gameTest with reset, loadScenario, input, and getSnapshot - plus legal scenarios, action semantics, rejection behavior, and invariants; coding agents implement this interface alongside the game, letting fixed checks access heterogeneous private implementations. Evaluation layers 336 static L1 checks and 885 browser-executed L2 checks (102 P0, 534 P1, 249 P2) in headless Chromium, combining semantic snapshots, real mouse and touch input, and runtime evidence. Strict task success requires passing every L1 and every applicable L2 P0/P1 check; all 47 independently verified reference implementations pass acceptance. Across nine agent stacks, mean L1 pass rates reach 97.7-99.6% and the best mean L2 pass rate is 93.2% (GPT-6-Astra with Codex CLI), yet the best strict task success is only 26/47 (55.3%). For DeepSeek-V4-Flash, full tool access yields 18 successes versus 6-9 restricted; raising the turn budget from 30 to 120 lifts successes from 7 to 18, mostly by letting more tasks complete evaluation; success is non-monotonic in reasoning effort (7, 19, 18 at low, high, maximum). Both harnesses reach 18 successes but overlap on only ten tasks. Limitations include single runs per configuration with no variance estimates, an unenforced NOT_APPLICABLE loophole for P1 checks, and reliance on agent-implemented snapshots.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Declaring an evaluation interface specification (window.__gameTest: reset/loadScenario/input/getSnapshot) before generation lets fixed checks consistently exercise heterogeneous generated games (Section 2).
- High mean check pass rates mask task-level noncompliance: across nine agent stacks, L1 pass rates of 97.7-99.6% and a best mean L2 of 93.2% coexist with a best strict task success of only 26/47 (55.3%) (Table 3).
- For DeepSeek-V4-Flash, tool access dominates (18 vs 6-9 strict successes) and larger turn budgets raise success from 7/47 to 18/47, mainly by increasing evaluation completion (Tables 5-6).
- Strict success is non-monotonic in reasoning effort, with high effort yielding 19/47 versus 18/47 at maximum while using 26.9% fewer reasoning tokens (Table 7).
- Two harnesses with identical success counts (18/47 each) succeed on largely different task sets, overlapping on only ten tasks (Table 8, Section 4.5).
Strengths
Pre-declared evaluation interface cleanly solves check-to-artifact coupling
Section 2.2 fixes window.__gameTest (reset/loadScenario/input/getSnapshot) plus scenario, action, rejection and invariant semantics before generation, so identical fixed checks drive heterogeneous private implementations - a genuine alternative to GameGen-Verifier's runtime state injection and GameXpert-Bench's post-hoc rubric pooling (Section 5.3).
Strict metric exposes what mean pass rates hide
Equation 1 and Table 3: GPT-6-Astra hits 93.2% mean L2 pass yet only 55.3% strict success; GPT-5.6-Sol outranks Claude-Opus-5 on mean L2 (91.3% vs 90.4%) but wins fewer strict tasks (21 vs 24) - concrete evidence the dual reporting is necessary.
Multi-signal behavioral evidence instead of LLM-judge scores
Positive-control quality control
Unusually candid negative reporting
Weaknesses
Single run per cell, no variance estimates
Acknowledged but unquantified integrity loopholes
Model-harness confound in the headline comparison
Turn-budget effect is largely a completion effect
Small, imbalanced corpus with no contamination discussion
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GameASG-Bench (arXiv 2609.21293; Ant Group / Beihang / HKUST) asks one narrow, well-posed question: can coding agents deliver a complete browser game whose runtime behavior satisfies a written specification, rather than code that merely looks right? It claims three things: (1) pre-declaring an evaluation interface specification (window.__gameTest with reset, loadScenario, input, getSnapshot; Section 2.2) lets fixed checks drive heterogeneous generated implementations; (2) the resulting 47-task corpus with 336 L1 plus 885 L2 checks separates high average pass rates from task-level compliance; and (3) agent performance is strongly configuration-sensitive (tool access, turn budget, reasoning effort, harness). The headline empirical claim is the contrast between a 93.2% best mean L2 pass rate and a 55.3% (26/47) best strict success rate (Table 3, GPT-6-Astra with Codex CLI). The claims are carefully scoped and mostly supported by the presented data, with the caveats below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is GameASG-Bench?
It is a benchmark of 47 browser-game generation tasks in which coding agents must build a complete playable game that also implements a declared test interface (reset, loadScenario, input, getSnapshot). Evaluation runs 336 static source checks and 885 checks in a headless Chromium browser; strict task success requires every required check to pass, not just most of them.
What are the main results of the paper?
Across nine agent stacks, source-level (L1) pass rates are 97.7-99.6% and the best mean browser-check (L2) pass rate is 93.2%, yet the best strict task success is only 26 of 47 tasks (55.3%), achieved by GPT-6-Astra with Codex CLI - showing that high average check pass rates hide individual failed requirements.
Where can I get the GameASG-Bench code and tasks?
The paper points to a public repository at github.com/areal-project/GameASG-Bench, which per the text contains the task packages, executable checks and evaluation runners; the generation container deliberately excludes the checks so agents cannot read the grading code.
How is it different from WebGameBench or GameGen-Verifier?
Those evaluate generated games via browser interaction or injected runtime state after generation. GameASG-Bench instead fixes the evaluation interface and executable checks before generation, so the agent implements the test hooks along with the game and the exact same checks run across different private implementations (Section 5.3).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract is truncated mid-sentence in the extracted text ('...yield more strict task successes, w'); RQ2 conclusions were reconstructed from Section 4.3.
- Figure 1 composition is only available as extracted legend text; the rendered figure layout was not verifiable.
- Tables were reflowed during extraction (e.g., a stray parenthesis in Table 6's '120 (47/47' row), suggesting minor OCR noise, though all cited numbers cross-check across sections.
- Whether any appendix content beyond A.3 and B exists could not be confirmed from the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.