GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents
Boyu Qiao, Zixin Tang, Xiaoshuai Hao, Wenbo Li
cs.CV · cs.AI
Quick verdict
Claims first black-box game-replication benchmark; isolation is well enforced, but LLM-judge scoring, no significance tests, and contamination risks remain.
Summary
GameReplica is a benchmark asking whether VLM-driven coding agents can recover a web game's latent specification purely through visual interaction and rebuild it as a runnable replica. Each of 125 tasks (25 games at five difficulty levels L1–L5, spanning five mechanism families) confines the agent to screenshot–action HTTP endpoints inside a containerized, network-isolated environment; deliverables are a self-contained HTML replica and an induced rule checklist. A dual-judge LLM protocol scores rule consistency, implementation consistency, and visual fidelity against curated reference rubrics. With the Claude Code runtime fixed, Claude Opus 4.8 leads at 71.6% overall (visual 93.1, implementation 67.4, rule 54.2), followed by GPT-5.5 (42.9), Grok v4.5 (27.9), GLM-5.2 (17.8), Kimi K3 (12.7), and Deepseek V4-Pro (4.0). Every model scores far higher on visual fidelity than on rules or implementation; weaker models drop sharply from L1 to L5 while the top model loses only 3.5 points, and swapping runtimes matters far less than swapping models. Most failures originate in the exploration phase (e.g., byte-by-byte PNG decoding). Limitations: all three tracks rely on LLM judges without human validation or significance testing; reported ± is cross-instance spread, not replication variance; the game list, rubrics, and code release are absent from the text.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GameReplica is the first closed-loop benchmark requiring agents to recover a game's latent specification from pixel-only interaction and deliver a verifiable executable replica (125 tasks: 25 games × L1–L5 × 5 mechanism families).
- Under a fixed Claude Code runtime, Claude Opus 4.8 reaches 71.6% overall while the other five frontier models span 4.0–42.9%, with identical rankings across all three evaluation tracks.
- All models replicate visual appearance far more readily than game rules (visual > implementation > rule for every model), and the gap widens with difficulty.
- Model capability dominates agent-runtime choice: swapping models changes implementation-consistency scores by more than 18× (67.4 vs 3.6) versus about 1.29× for swapping runtimes (67.4 vs 52.4).
- Weak models' failures overwhelmingly originate in the exploration phase (e.g., Manual PNG decode: 36/60 for Deepseek V4-Pro, 23/52 for Grok v4.5), not in code generation.
Strengths
Enforceable black-box boundary
Section 3.3 realizes interface closedness, observation determinism, and output isolation via a two-container Docker Compose topology: the Playwright-hosted game server sits on an internal bridge network with no outbound route, and the agent container's traffic is allowlisted to the inference gateway only — environmental enforcement, not prompt-honored compliance as in most prior vision-to-code benchmarks.
Traceable, validity-checked difficulty construction
Section 3.2 defines structural (Nact, Ntrans, Ngoal, Nunit, Nobj) and solving (Nstep, mean branching factor) complexity vectors and gates each L1–L5 family on monotonicity, mechanism authenticity, and rule observability; Appendix Tables 9–19 report full per-level trajectories for all 11 producer configurations, and Appendix D publishes the entire agent system prompt including [confirmed]/[inferred]/[unknown] confidence tags.
Consistent rankings across independent tracks
Diagnostic depth beyond scalar scores
Clean two-axis disentangling of model vs runtime
Weaknesses
Entirely LLM-judged scoring with no calibration
Ambiguous and internally inconsistent statistics
No data-contamination control
Benchmark content and aggregation under-specified
Survivorship-biased process-cost metrics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GameReplica (arXiv 2609.22308, under review) claims to be the first closed-loop benchmark for end-to-end black-box game replication by VLM-driven coding agents. The task: given a running web game accessible only through screenshot–action HTTP endpoints, induce its visual elements, action constraints, state transitions, and termination conditions, then deliver (a) a self-contained single-file HTML replica and (b) an induced rule checklist. The corpus is 125 tasks — 25 games at five difficulty levels (L1–L5) across five mechanism families. Central claims: (1) the best agent, Claude Opus 4.8 under Claude Code, reaches 71.6% overall while the other five models span 4.0–42.9% (Table 1); (2) every model shows visual fidelity ≫ implementation > rule consistency (Opus: 93.1/67.4/54.2); (3) difficulty amplifies the gap — Grok v4.5 falls 35.9→15.5 from L1 to L5 while Opus only moves 72.1→68.6 (Table 2); (4) model choice dominates runtime choice, with an >18× implementation-score swing across models versus ~1.29× across runtimes (Tables 1, 3); (5) failures originate predominantly in the exploration phase (Figure 7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is GameReplica and what does it measure?
GameReplica is a benchmark that measures whether AI coding agents can figure out how a web game works purely by looking at screenshots and interacting with it (clicks, keys, scrolls, resets), then write a working replica as a self-contained HTML page plus a written list of the rules they inferred. It contains 125 tasks: 25 web games at 5 difficulty levels (L1–L5) across 5 mechanism families, with the agent blocked from reading the original source code.
Which models were tested and how did they perform?
Six frontier models were tested with the Claude Code agent runtime: Claude Opus 4.8 scored best at 71.6% overall, followed by GPT-5.5 (42.9), Grok v4.5 (27.9), GLM-5.2 (17.8), Kimi K3 (12.7), and Deepseek V4-Pro (4.0). Every model found visual appearance much easier to replicate than actual game rules, and weaker models' scores collapsed as difficulty rose from L1 to L5, while Claude Opus 4.8 dropped only 3.5 points.
How are the game replicas evaluated?
Each replica's source code, the agent's induced rule document, and replica screenshots are compared against reference checklists by LLM judges using a two-round assessment-and-review protocol (gemini-3.6-flash assesses, gpt-5.6-sol reviews). There are no executable unit tests; all three scores — rule consistency, implementation consistency, and visual fidelity — are judge-based and combined into one overall score.
What are the main limitations of the paper?
The evaluation relies entirely on LLM judges with no human validation or agreement statistics; the reported ± measures variability across the 125 tasks rather than across repeated runs, with no significance testing; there is no control for models having memorized classic game mechanics during pretraining; and the paper text does not enumerate the 25 games or commit to releasing code, rubrics, or judge prompts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–7 appear only as extracted plot labels and captions; visual details of endpoint breakdowns and cost-quality plots could not be inspected directly.
- The 25 games and five mechanism families are never enumerated in the extracted text; they may exist in an appendix table or on the referenced project page that was not captured.
- Apparent numeric inconsistencies between Table 1 and Tables 2/14 (e.g., 4.0 vs 3.9 for Deepseek V4-Pro) may partly be text-extraction artifacts rather than errors in the paper itself.
- No project page URL or explicit code/data release statement is visible in the extraction; availability may exist but is unverifiable from the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.