Combining LLMs and Genetic Search for ARC-AGI-2
cs.NE · cs.AI
Quick verdict
Claims GA evolves LLM-seeded programs to lift ARC-AGI-2 solves from 2/60 to 6/60; promising but tiny scale, non-random subset, incomplete control.
Summary
The paper combines a small local LLM with a genetic algorithm to solve ARC-AGI-2 grid puzzles. A quantized Qwen3.5-4B model generates candidate programs in a 41-operation DSL (27 core, 14 LLM-gated) through a four-prompt pipeline (plan, operations, parameters, chromosomes); these seeds populate two 4,000-chromosome islands that a GPU-executed 'Machine' then evolves with mutation, crossover and 2% elitism, without any further LLM calls, capped at 200 generations or 600 seconds per task. On the first 60 tasks (ordered by Id) of the 120-task public evaluation set, the LLM pool alone solves 2 (3.3%); after evolution, 6 tasks (10.0%) have a program whose exact training fit also reproduces the test output, with the four GA-derived solutions emerging at generations 22, 35, 50 and 80. Unseeded GA runs solved nothing, but the authors concede this control changed during development and is not clean. Main limitations: the non-random, partially development-tuned task subset, single stochastic runs, an incomplete random-seed ablation, and a DSL/perception layer that cannot express all ARC rules. The work is a transparent, honestly caveated preliminary study; its contribution is the demonstration that LLM seeding places genetic search in a productive region of program space, not a competitive ARC-AGI-2 score.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A quantized 4B LLM, prompted once per task, generates DSL seed programs useful enough that 2/60 ARC-AGI-2 tasks are solved outright before any evolution.
- Genetic search over those seeds converts near-miss programs into additional correct test solutions, raising solves from 2/60 to 6/60 (Table 3).
- Without LLM seeding, the GA finds no exact training fits at all in the tested (admittedly non-clean) control configuration (§4.3).
- A fixed-width gene DSL (20 genes x 6 fields, 41 operations) makes every mutated program executable, allowing ~8,000-program generations and ~1.6 million program evaluations per task on GPU.
Strengths
Clean decomposition of credit between components
Table 3 isolates LLM-seed-only (2/60, 3.3%) from LLM+GA (6/60, 10.0%), and Table 4 itemizes all nine exact training fits with seed-program counts (18-156), source (LLM/GA) and generation (22-162), explicitly separating correct from failed test outputs.
Mutation-safe execution substrate
The fixed-width gene format (Eq. 1: 20 genes of [op,a-e]) with per-operation argument handling guarantees every mutated or crossed program executes (§3.2), enabling 8,000 candidates per generation on GPU via documented tensors (Table 1: State [P,N,21,30,30], Registers [P,N,176]).
Unusual configuration transparency
Honest self-critique
Strict solve criterion
Weaknesses
Small, biased evaluation slice
Admittedly unclean control
No variance quantification
Attribution confound via gated operations
No compute-matched LLM-only baseline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: can genetic search improve LLM-generated programs for ARC-AGI-2? The claim chain is (1) a quantized Qwen3.5-4B produces seed programs in a 41-operation DSL via a four-prompt pipeline (§3.4); (2) two 4,000-chromosome islands, evolved by a GA with no further LLM calls, convert near-miss seeds into correct test solutions; (3) unseeded GA lands nowhere. Headline numbers: 2/60 (3.3%) LLM-only vs 6/60 (10.0%) after evolution, on the first 60 tasks (by Id) of the 120-task public evaluation set. The authors explicitly frame this as preliminary (§5.1: the six solves 'do not show that this design is optimal'), which is the right framing.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What score does this system achieve on ARC-AGI-2?
On the first 60 of 120 public evaluation tasks (ordered by task Id), the system solves 6 tasks (10.0%): 2 solved directly by the LLM's generated programs and 4 more after genetic evolution. This is far below frontier ARC-AGI-2 systems; the authors present it as a proof of concept, not a competitive result.
Which LLM is used and how large is it?
A quantized Qwen3.5-4B (Q4_K_M GGUF) run locally with llama.cpp, thinking mode disabled. It is called only before evolution, via four prompts (plan, operations, parameters, chromosomes), to generate seed programs in a 41-operation domain-specific language.
Does the genetic algorithm work without the LLM seeds?
In the authors' tests, unseeded GA runs found no exact training fits on any task. However, they admit the control setup changed during development, used only the 27 core operations, and covered illustrative runs rather than all 60 tasks - so the comparison is suggestive but not a clean controlled experiment.
Is code or data available for this paper?
The paper points to an external blog page (rockin.ai) hosting LLM conversations and GA run resources, plus full hyperparameter tables and the DSL operation list in appendices. No code repository is named in the text itself, and the perception module is not published.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are referenced but their image content (architecture diagram, fitness trajectories) cannot be inspected from extracted text; curve-shape statements rely on the paper's verbal descriptions.
- Additional resources (LLM conversations, GA runs) live on an external blog that was not accessible for this review; artifact completeness is unverified.
- This is a non-peer-reviewed arXiv preprint (v1, 23 Sep 2026); the review covers that version only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.