PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks
Claas Beger, Ryan Yi, Melanie Mitchell
cs.AI
Quick verdict
PotARCin extends ARC to five rule-use dimensions, exposing 25-52pp drops and self-contradiction; a rigorous benchmark study with acknowledged confounds.
Summary
The authors introduce PotARCin, a benchmark that evaluates abstract skill acquisition on the Abstraction and Reasoning Corpus (ARC) across five dimensions: Definition (writing an executable rule program), Classification, Constrained Generation, Editing, and Inversion, adapting the Potemkin-understanding framework of Mancoridis et al. to ARC. Using public generator programs (ARC-GEN, RE-ARC) and validated verifier programs covering 400 ARC-AGI-1 training tasks, plus five corruption procedures for invalid pairs, they evaluate five frontier models (GPT-5.4, Gemini 3.1 Pro, Claude Opus 4.6, Kimi K2.5, MiniMax M2.5). Full-task accuracy, meaning passing all five dimensions, falls 25-52 percentage points below standard output-grid accuracy (84-89% for the proprietary models), and rankings reorder: Kimi K2.5, weakest on output grids at 63.4%, nearly doubles MiniMax's full-task accuracy (38.6% vs 20.6%). On a new held-out 50-task set, P-ARC, full-task accuracy drops to 1-8%. Diagnostics show that single-sample evaluation overestimates robustness (15 of 20 GPT-5.4-passed tasks fail under resampling) and that models frequently contradict a rule they themselves formalized correctly (21.1% agreement on ARC-AGI-1 when the other answer is wrong). Limitations include likely pretraining exposure to the ARC-AGI-1 training data, verification-burden asymmetry between dimensions, corruption labels defined relative to a designated rule, only two runs with substantial pass-set churn, and unequal compute configurations such as Claude's 120K thinking budget.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On ARC-AGI-1 training tasks, full-task accuracy across all five PotARCin dimensions is 25-52 percentage points below standard output-grid accuracy for five frontier models.
- Multi-dimensional evaluation reorders model rankings: Kimi K2.5 has the lowest output-grid accuracy (63.4%) yet nearly doubles MiniMax M2.5's full-task accuracy (38.6% vs 20.6%).
- Conditioned on a correct executable Definition, models agree with their own program in only 21.1% (ARC-AGI-1) / 17.1% (P-ARC) of cases where their answer in another dimension is incorrect, evidence of Potemkin-like rule contradiction.
- Single-sample full-task accuracy overestimates robust rule competence: 15 of 20 tasks GPT-5.4 initially passed all five dimensions fail at least one of ten resampled configurations.
- On the held-out hand-crafted P-ARC set, full-task accuracy collapses to 1-8% across all five models.
Strengths
Verified, fully automated scoring infrastructure
Verifier programs (Kaggle code-golf placements plus RE-ARC) are validated against train/test examples, the ARC-GEN stable set (~250 pairs/task) and 50 generator samples, covering 394/400 tasks plus 6 hand-implemented ones; Appendix D shows 84-85% of Definition failures are wrong-rule errors, not code-load failures, so the dimension measures rule inference rather than coding ability.
Statistical care unusual for benchmark papers
Pooled Wilson 95% CIs, an independence baseline for the joint metric (+7.4 to +18.5 pp observed above it), explicit trivial-baseline disclosure for Classification (always-invalid 19.1%/16.0%), and run-to-run churn reported via Jaccard (0.49-0.79, Table 7) rather than hidden.
Diagnostics separate arithmetic from genuine inconsistency
Mimetic-similarity requirement is empirically justified
Held-out P-ARC set with human corruptions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PotARCin (arXiv 2609.27288, Santa Fe Institute) extends the Abstraction and Reasoning Corpus beyond the single output-grid task. Each ARC task's inferred transformation rule is probed along five dimensions adapted from the Potemkin-understanding framework of Mancoridis et al. (2025): Definition (write an executable rule program), Classification (judge five candidate pairs, all-or-nothing at task level), Constrained Generation (produce a novel valid pair), Editing (repair a corrupted pair under a normalized edit-distance constraint of 0.70), and Inversion (infer an input from an output). Central claims: (1) full-task accuracy, passing all five dimensions, is 25-52 percentage points below standard output accuracy on the ARC-AGI-1 training set across five frontier models (Table 3); (2) multi-dimensional evaluation reorders models that standard accuracy ranks alike; (3) models contradict their own correctly formalized rules (Table 4); (4) single-sample evaluation overestimates robustness (Section 3.2). A held-out 50-task set, P-ARC, is contributed.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PotARCin and how does it differ from standard ARC evaluation?
PotARCin is a benchmark that tests whether AI models have actually learned an ARC task's transformation rule, not just whether they can produce one correct output grid. It probes five uses of the inferred rule: writing an executable rule program (Definition), judging valid/invalid example pairs (Classification), generating new valid examples (Constrained Generation), repairing corrupted pairs (Editing), and inferring an input from an output (Inversion). Standard ARC evaluation checks only the final output grid.
How much worse do frontier models do on PotARCin than on standard ARC?
On the 400 ARC-AGI-1 training tasks, models scoring 63-89% on the standard output task pass all five dimensions only 21-58% of the time, a 25-52 percentage-point drop. On the new held-out set P-ARC (50 hand-crafted tasks), full-task accuracy falls to 1-8% for every model tested.
What does Potemkin understanding mean in this paper?
It refers to cases where a model appears to understand a rule on a benchmark but fails closely related probes of the same rule. PotARCin finds direct evidence: when a model's own Definition program is correct, only 21.1% (ARC-AGI-1) or 17.1% (P-ARC) of its incorrect answers in other dimensions agree with that program, meaning its mistakes are not consistent applications of its own stated rule.
Is the P-ARC dataset publicly available?
The paper states P-ARC will be released upon publication under a permissive open license. Each of the 50 tasks ships in standard ARC JSON format together with its generator and verifier programs, 50 stable generated examples, and the three human-error grids used as corruptions. Release of the evaluation harness code itself is not explicitly stated in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 (consistency boundaries, benchmark overview, sampling-budget curves) are visual and could not be verified from extracted text; their underlying numbers do appear in Tables 3 and 6.
- The evaluation harness release is not explicitly stated (only the P-ARC dataset is promised under a permissive license), which tempers the reproducibility assessment.
- OCR of grid-heavy appendix figures is lossy; per-dimension values were taken from Tables 3, 4 and 6, which extracted cleanly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.