Compiled Agency: Frontier General-Purpose Coding Agents Build Winning Game Players from Bare Interaction - from Flappy Bird to StarCraft II and Civilization
cs.AI · cs.CL
Quick verdict
Claims agents compile winning game players from one bare session; SC2 evidence strong but session counts tiny and Civilization exams incomplete at freeze.
Summary
The paper introduces Gauntlet, a develop-freeze-evaluate protocol asking whether general-purpose coding agents, given only a game description, a raw observation/action interface, and an empty policy file, can autonomously engineer a standalone game-playing program in one session. The controller is frozen and scored on held-out instances with zero model calls. On an unpublished bullet-hell roguelike, deployed systems span 0-86% held-out success; every observed Fable 5 session exceeds the best Opus 4.8 session (exact permutation p=0.0286, 4 vs 4 sessions). At full scale, a 958-line frozen Terran controller wins 14/16 held-out StarCraft II Elite scenarios and defeats two cheating AI tiers, and three isolated Fable sessions independently implement convergent Freeciv conquest programs, one winning 4/12 held-out full games by total conquest against the novice AI - presented as a first for language agents. Audits of 75 policies indicate control structure and doctrine, not code volume, predict success. Limitations: decisive cells have only 2-4 sessions, systems are compared as deployed (harness/effort co-vary), public titles admit pretraining contamination, two of three Civilization held-out exams were still incomplete at freeze, and no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A frontier general-purpose coding agent, used exactly as shipped with no game-specific harness and only a description, raw interface, and empty policy file, can compile a standalone winning game-playing program in one autonomous session with zero model calls during play (compiled agency).
- On a private unpublished roguelike, deployed systems span 0-86% held-out success and show a sharp generational threshold: every observed Fable 5 session outperforms the best observed Opus 4.8 session (exact permutation p=0.0286).
- A 958-line frozen Terran controller authored in a single session defeats every fair StarCraft II built-in AI (14/16 held-out Elite scenarios) plus Cheater:Vision and Cheater:Resources, a competence band previously reached only by dedicated RL systems or LM systems with per-turn model calls.
- Single-session Freeciv programs achieve engine-verified full-game conquest wins on held-out seeds against the novice AI at modest rates (best completed exam 4/12), the first end-to-end standalone Civilization wins for language agents.
- What gets compiled is control structure and doctrine rather than code volume, and because fresh sessions produce widely varying artifacts, the distribution of frozen policies - not any champion - is the correct system-level measurement.
Strengths
Genuinely novel measurement construct
Gauntlet evaluates an artifact-producing process rather than step-by-step model play: architecture is an output (Fig. 1), artifacts are frozen with zero model calls at play time, and the paper cleanly separates controller competence (held-out instances) from system reliability (independent fresh sessions) as two distinct variation sources (Sec. 2.1).
Session-level statistics on a contamination-resistant task
The private, never-released roguelike removes the direct contamination route; the generational claim is made at the capability-producing unit (all four Fable sessions at 51-80/80 clear the best Opus session at 47/80) with an exact permutation over all 70 labelings (p=0.0286), under matched task version, prompt hash, harness, and exam.
Verifiable, inspectable evidence
Unusual candor and self-audit
Scale span with external anchors
Weaknesses
Very small samples at the decisive cells
Deployed-system confound in the capability ladder
Civilization evidence incomplete and narrow at freeze
Pretraining contamination cannot be excluded for public titles
No stated code/data release; operator-mediated full-scale runs
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'compiled agency' in this paper?
The paper's term for a coding agent converting one autonomous development session into a persistent, standalone executable game-playing program. The agent is given only a game description, a raw observation/action interface, and an empty policy file; everything else - state tracking, simulation, strategy, control logic - is code the agent writes itself. The finished program runs with zero AI-model calls during play, and the phrase 'compiled' refers to development experience being compiled into code, with the model removed from the action loop.
Did an AI actually beat StarCraft II's built-in AI?
According to the paper, yes: a 958-line program written in a single session by a coding agent won 14 of 16 held-out matches at the Elite difficulty against the built-in AI, and when replayed down the difficulty ladder it beat every fair tier plus two cheating tiers (extra vision and extra resources), losing only to the strongest cheating tier (Cheater:Insane). Important caveats: only some sessions succeeded (fresh sessions of the same system scored anywhere from 0/16 to 14/16), and the play-time program makes no AI-model calls - it is frozen code.
Can LLM-based agents win full games of Civilization?
This paper reports the first standalone language-agent wins of complete Freeciv (open-source Civilization) games: three independent coding-agent sessions produced programs that achieve engine-verified total-conquest victories. However, the wins are against the novice AI only, on small 1v1 maps, at modest held-out rates (the one completed exam won 4 of 12 games), and many losses were 'alive and score-ahead at the turn deadline' rather than elimination. The authors frame it as a foothold, not mastery.
Is the Gauntlet code or data available?
No code or data release is stated in the reviewed text; the paper says artifacts are archived and describes a refreshable frozen leaderboard manifest (dated 2026-07-17), but no repository URL or availability statement appears in the extracted content. Readers seeking replication should check the arXiv page and authors' sites for supplementary releases.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract is truncated mid-sentence in the extraction and figures are available only as garbled embedded text, so figure-level values were cross-read from captions and body text rather than inspected visually.
- No code/data availability statement is visible in the extracted text; one may exist in an appendix or supplement that was not included.
- All quantitative results are taken as reported by the authors; no independent verification of runs, seeds, or audits was possible from the text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.