GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay
Yiran Wang, Xingyilang Yin, Junfu Pu, Guangzhi Wang, Kaifeng Li, Mingyu Ouyang, Huiqiang Sun, Lingen Li, et al.
cs.CV · cs.AI
Quick verdict
Large AAA gameplay dataset+benchmark (5k hrs/21 games/47 models) with solid scale, but VLM-generated ground truth and no online AAA eval need scrutiny.
Summary
GameHorizon Suite couples an automated annotation pipeline, a large gameplay dataset, and a benchmark for measuring AI play across temporal horizons. GameHorizon-Annotator uses action-aware segmentation plus a VLM (Gemini 3.5 Flash) to build a three-level instruction pyramid — short-horizon operations (1–5 s), medium-horizon goals (1–2 min), long-horizon strategies (5–8 min) — from videos with directly recorded keyboard/mouse actions. GameHorizon-Data spans 5,000 hours from 100 expert players across 21 titles, with 411.03 million action events and 6.18 million instructions. GameHorizon-Bench offers a reproducible offline track of 5,000 four-option MCQs (three primary tasks, ten diagnostic variants) and a stepwise online track of 20 Minecraft tasks with 62 verifiable subtasks and reset-on-failure. Evaluating 47 models via more than one million invocations yields a 64.7% mean offline accuracy (25% chance), a difficulty hierarchy (T1 57.3%, T2 65.1%, T3 71.6%), near-chance zero-shot transfer of dedicated game agents to AAA titles (22–28%), and a positive rank association between offline scores and online success for 12 sampled models. Main limitations: instructions and MCQ options stem from the same VLM pipeline without reported human validation; the online track runs only in Minecraft, not AAA games; rankings lack confidence intervals or significance tests; and licensing for releasing AAA footage is unaddressed.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GameHorizon-Data is the first large-scale (5,000 h, 21 games) AAA gameplay dataset aligning videos, directly recorded human actions, and dense multi-horizon instructions (6,184,036 total; Table 2).
- The automated bottom-up GameHorizon-Annotator scalably produces a three-level instruction pyramid (1-5 s operations, 1-2 min goals, 5-8 min strategies) that would be prohibitively costly to label manually.
- The 5,000-question offline MCQ track is a valid, reproducible proxy for actual gameplay ability, evidenced by a positive association between offline tiers and online success for 12 sampled models (Table 9).
- Multi-horizon instructions improve future-action planning by about 7 percentage points on average (34.8% to 41.8%, Table 7), whereas short-horizon instructions mainly aid current-action perception (+14.1 pp).
- Dedicated game agents fail to transfer zero-shot to AAA games (22.0-28.1% on T1 vs 25% chance; Table 5), despite one of them matching general models in-domain on Minecraft VPT data.
Strengths
Direct human action recording at unprecedented scale
Synchronized 2K/60fps video with timestamped keyboard/mouse logs (20 Hz mouse, ~2.84 keyboard events/s; 411.03M events) from 100 trained players across 21 titles (Sec. 3.1, 3.3), avoiding the IDM pseudo-label noise acknowledged for VPT and D2E in Table 2.
Multi-horizon instruction pyramid with per-frame alignment
6,184,036 instructions (5.95M L1, 189k L2, 47k L3) with average spans 2.63 s / 82.6 s / 330.4 s; every frame aligned to all three levels (Table 1). Variant T*1 (Table 7) empirically supports the design: L1 instructions add +14.1 pp to current-action perception while L2+L3 add +7.0 pp to future-action planning.
Reproducible offline track decoupled from harnesses and environments
Stepwise online protocol with failure localization
Unusual breadth of evaluated model families
Weaknesses
Entirely VLM-generated ground truth with no human validation
Ground-truth circularity in MCQ construction
Online track contradicts the AAA motivation
No inferential statistics anywhere
Unspecified scoring threshold for game agents
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GameHorizon Suite is a dataset-plus-benchmark paper (Tencent ARC Lab and university collaborators) for measuring multimodal AI gameplay across temporal horizons. It ships three components (Sec. 3.1): **GameHorizon-Annotator**, an automated pipeline labeling gameplay with a three-level instruction pyramid (L1 operations 1-5 s, L2 goals 1-2 min, L3 strategies 5-8 min); **GameHorizon-Data**, 5,000 hours from 100 expert players across 21 titles with directly recorded keyboard/mouse actions (411.03M events) and 6,184,036 instructions; and **GameHorizon-Bench**, an offline track of 5,000 four-option MCQs plus a stepwise online track of 20 Minecraft tasks with 62 verifiable subtasks. The central claims: (i) first large-scale AAA dataset aligning video, human actions, and dense multi-horizon instructions (Table 2); (ii) the offline track validly proxies online gameplay ability (Sec. 4.4); (iii) a 47-model, >1M-invocation evaluation yields a meaningful capability hierarchy.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the GameHorizon Suite?
A three-part resource for evaluating AI gameplay released by Tencent ARC Lab and collaborators: (1) an automated pipeline that labels gameplay video with instructions at three time horizons (seconds-long operations, minutes-long goals, multi-minute strategies); (2) a dataset of 5,000 hours of gameplay from 21 games recorded by 100 expert players with synchronized keyboard/mouse action logs; and (3) a benchmark combining 5,000 offline multiple-choice questions with 20 online Minecraft tasks broken into 62 verifiable subtasks.
Which models were evaluated and what were the main results?
47 models across five families (general VLMs, unified multimodal models, GUI agents, coding agents, dedicated game agents) with over one million invocations. Mean offline accuracy was 64.7% against a 25% random baseline; GPT-6-Astra led at 80.2%. Dedicated game agents scored 22-28% on AAA titles (near chance), and on the online track the best model (GPT-6-Astra) passed 45% of long-horizon tasks while ten of twelve tested models passed at most 10%.
Is the online evaluation done in AAA games?
No. Only the offline question-answering track covers the 21 AAA games. The online track is implemented in Minecraft because AAA games do not expose underlying game states needed for automatic verification; the authors therefore test the offline-to-online link only in Minecraft, which is one of the review's main criticisms.
Is the dataset or code available?
The paper states the dataset, annotator, and benchmark will be released via the project page (gamehorizon-suite.github.io) and GitHub (TencentARC/GameHorizon). Availability could not be verified from the paper text alone, and the paper does not discuss licensing of the commercial game footage, which is a practical risk for the promised release.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (annotation prompts in Figs. 6-8, benchmark diagrams, online rollout examples in Figs. 4-5/9) are images and could not be analyzed from the extracted text; exact prompt wording is unverifiable.
- The provided metadata lists 8 authors while the paper text lists 15 (including Cheng Cheng, Wangbo Yu, Xiaodong Cun, Chi-Man Pun, Zhiguo Cao, Ying Shan); the review reflects the full text.
- Minor internal inconsistency: the introduction cites a 7.2 pp planning gain from multi-horizon instructions while Table 7's overall row computes +7.0 pp (34.8 to 41.8).
- References dated 2025-2026 and release links could not be verified (no network access).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.