GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions
Xinyu Che, Yunfei Ge, Shihao Li, Yanchen Liu, Hang Yan, Xinping Lei, Xinping Lei, Yanghai Wang, et al.
cs.SE · cs.AI · cs.CL
Quick verdict
GameLogicBench: a rigorously validated, deterministic tick-level Godot coding-agent benchmark; credible results but single-run, Godot-only, scaffold-dependent.
Summary
The paper introduces GameLogicBench, a benchmark of 72 gameplay-logic tasks for coding agents in Godot 4.4 projects, evaluated by a deterministic judge that checks game-state assertions at every simulation tick across 403 hand-designed scenarios and their 1,451 seeded test cases. Tasks span three integration tiers — Atom (21 single-mechanic tasks), Combo (28 multi-mechanic games), and Repo (23 real open-source projects) — with a three-layer information boundary separating disclosed requirements from hidden scored scenarios and judge code. Tasks pass human screening by three annotators, agent-assisted construction, calibration against proper solutions, naive solutions, single-capability mutants, and behavior-preserving controls, plus independent agent and human review. Across 20 model-scaffold configurations (12 models under Claude Code, 4 each under Codex and OpenCode), the best run — Claude-Opus-5 with Claude Code — solves 52.78%, and solve rates fall from Atom to Repo for nearly all configurations. Ablations show terminal-state checks let 34 of 36 mutants escape and preview-only evaluation all 36, while criteria validated without mutants had 24 missing checks across 19 of 36 audit tasks, flipping 3 task outcomes. An open-network audit shows upstream-source retrieval inflating Repo-tier scores (Kimi-K3 with Claude Code: 3 to 12 tasks). Limitations: one run per configuration without statistical uncertainty, Godot/GDScript only, and largely agent-authored tasks.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GameLogicBench is the first game-development benchmark combining tick-level assertions, multiple judge-selected scenarios, and deterministic verdicts across all tasks (Table 1).
- The best of 20 model–scaffold configurations, Claude-Opus-5 with Claude Code, solves 52.78% of the 72-task library, and solve rates decline from Atom through Combo to Repo for nearly all configurations.
- Both tick-level assertions and multiple scenarios are necessary for reliable scoring: terminal-only scoring lets 34/36 mutants escape; preview-only evaluation lets 36/36 escape and passes 418/488 (85.7%) previously failing solutions.
- Criterion validation must use mutants: criteria validated only on correct solutions left 24 missing checks across 19 of 36 audit tasks and changed 3 benchmark outcomes.
- Open-network source retrieval inflates Repo-tier solve rates for all four retrieving configurations among five audited (up to +9 tasks for Kimi-K3 with Claude Code).
Strengths
Evaluation design fills a demonstrable gap
Table 1 shows GameLogicBench is the only compared benchmark combining deterministic scoring, tick-level assertions on all tasks, and 2–12 scenarios per task; existing game benchmarks rely on VLM/agent judges (GameCraft-Bench, WebGameBench), single scenarios (GameDevBench, JAMER), or final-state checks.
Mutant-based criterion validation with quantified ablation
Section 4.6: control criteria without mutant validation let 127 of 666 mutants pass, exposing 24 missing checks across 19 of 36 audit tasks; repair flipped 3 task outcomes (PASS→FAIL) for 3 of 9 tested models while proper solutions and controls still passed. Most benchmark papers never measure judge sensitivity this way.
Scoring-protocol ablation is close to conclusive
Transparent contamination reporting
Reproducibility discipline
Weaknesses
No statistical uncertainty
Single engine and language
Agent-heavy task construction without agreement statistics
Repo tier is small and pretraining contamination unbounded
Cost claims are time-sensitive list-price arithmetic
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GameLogicBench (arXiv 2609.21562v2) is a benchmark paper for coding agents: 72 gameplay-logic tasks inside Godot 4.4 projects, graded by a deterministic judge that applies state assertions at every simulation tick rather than at run end. The library holds 403 hand-designed scenarios expanded by seeded parameter variation into 1,451 test cases, organized into three integration tiers — Atom (21 single-mechanic tasks), Combo (28 multi-mechanic interactions), and Repo (23 real open-source projects). The authors claim three things: (1) no prior game-development benchmark combines multiple judge-selected scenarios, tick-level assertions, and deterministic verdicts across all tasks (Table 1); (2) current agents remain weak at runtime game logic — the best of 20 model–scaffold configurations, Claude-Opus-5 with Claude Code, solves 52.78%, with solve rates declining Atom→Combo→Repo for nearly every configuration (Table 3); (3) reliable evaluation requires both tick-level multi-scenario scoring and mutant-based criterion validation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is GameLogicBench?
A benchmark of 72 gameplay-logic tasks for coding agents inside Godot 4.4 projects. A deterministic judge — no language model in the scoring path — checks game-state assertions at every simulation tick across 403 hand-designed scenarios, which seeded variations expand into 1,451 test cases. Tasks come in three tiers: single mechanics (Atom, 21), interacting mechanics (Combo, 28), and real open-source repositories (Repo, 23).
Which AI models perform best on GameLogicBench?
Across 20 model–scaffold combinations, Claude-Opus-5 with the Claude Code scaffold is best, solving 52.78% of tasks; Qwen-3.8-Max with Claude Code follows at 44.44%. Solve rates drop as tasks grow from isolated mechanics (best over 60%) to whole-repository features (best 43.48%). All headline scores come from network-sealed runs.
Why check the game at every tick instead of just the final state?
Because a game can end in a valid state despite breaking its rules mid-run. The paper's ablation shows that scoring only terminal states lets 34 of 36 deliberately broken implementations (mutants) pass, and preview-scenario-only scoring lets all 36 pass while accepting 85.7% of solutions the full evaluator rejects.
Is GameLogicBench affected by data contamination?
Repo-tier tasks derive from public repositories, so the authors audited open-network access: in four of five audited configurations, agents retrieved upstream source and their Repo scores rose (largest: Kimi-K3 with Claude Code, 3 to 12 of 23 tasks). The headline benchmark numbers therefore use egress-sealed containers, though memorization of upstream code from pretraining was not separately audited.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text omits parts of Sections 4.2–4.6 (including Table 4 and fragments of Table 5) and Appendix D tables; judgments on those sections rely on the visible fragments and captions.
- Figures 1–9 are only partially recoverable from embedded caption text; paired Figure 7 scores and Figure 5 percentages are read from captions and may miss detail.
- Appendix G.2 (human-annotator standards) is truncated mid-document, so the full human-review standard could not be assessed.
- The stated GitHub release (NJU-LINK/GameLogicBench) could not be inspected from the provided text; the high reproducibility rating assumes the release matches the paper's description.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.