ISA-Bench: A Benchmark for Computational Reasoning Across Instruction Set Architectures
Aditya Pola, Arkaprava Majumdar, Vineeth N. Balasubramanian
cs.AI
Quick verdict
Careful new benchmark of 92 tasks across 5 unfamiliar ISAs; the novel REG analysis is promising but rests on an unvalidated GPT-5.5 judge.
Summary
ISA-Bench is a benchmark of 92 tasks drawn from five programming games with deliberately unfamiliar instruction sets: SIC-1 (single SUBLEQ instruction, 31 tasks), Human Resource Machine (36), TIS-100 parallel node grid (7), MHRD NAND-only circuit design (8), and BOX-256 pixel-drawing assembly (10). Each game ships a parser, virtual machine, and test verifier, enabling automated scoring and structured feedback over up to five attempts. The authors evaluate 14 open-weight models (8B-32B) spanning thinking, code-specialized, and general-purpose classes. The best model, Gemma-4 31B, solves 56/92 tasks (60.9%) with retries; nine models solve under 25%; no model solves any BOX-256 task. Thinking models average a 31.9% solve rate versus 13.4% for code-specialized and 9.8% for general-purpose models, though comparisons are uncontrolled for scale or training. Parse errors dominate first attempts (88.7% on SIC-1, 81.4% on BOX-256). A reasoning-execution gap (REG) analysis, scored by GPT-5.5 under a fixed rubric, finds the largest planning-versus-implementation disconnect on BOX-256 (0.96) and SIC-1 (0.94) and a near-zero gap on MHRD (0.06). Key limitations: the REG judge is never validated against human labels, solved tasks are auto-assigned 3/3 scores that mechanically shrink gaps, system prompts include worked patterns matching benchmark tasks (e.g., XOR in MHRD), contamination from popular game solutions cannot be excluded, and variability is estimated for only three models.
Review complete
Assessment at a glance
6
risk flags identified
● high 3
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Thinking/reasoning models achieve higher average solve rates (31.9%) than code-specialized (13.4%) and general-purpose (9.8%) open-weight models on unfamiliar-ISA synthesis, though the comparison controls for neither scale nor training.
- Parse errors are the largest first-attempt outcome category in every game, exceeding 60% on SIC-1 (88.71%), HRM (72.22%), and BOX-256 (81.43%), indicating unfamiliar syntax is a dominant failure source.
- Iterative execution feedback substantially but unevenly raises solve counts (aggregate HRM 76 to 154; Phi-4-Reasoning 1 to 24 overall; BOX-256 unchanged at zero).
- A reasoning-execution gap exists — models often identify plausible computational strategies but fail to implement them — largest on BOX-256 (0.96) and SIC-1 (0.94) and near-zero on MHRD (0.06) under GPT-5.5-judged rubric scores.
Strengths
Deterministic, executable evaluation
Each game ships a parser, VM, and verifier (Sec. 3, Fig. 2), so correctness is execution-checked rather than heuristic; Appendix F adds graded rewards (HRM instruction/step efficiency, MHRD NAND-count efficiency) usable for RL fine-tuning.
Genuine coverage gap addressed
Five environments span distinct computational primitives (single-instruction SUBLEQ, accumulator planning, blocking message-passing over 12 nodes, NAND-only circuit synthesis, self-visible pixel memory); Table 1 positions this against HumanEval/MBPP/SWE-bench, which all test mainstream languages.
Hedged, internally consistent reporting
Above-average reproducibility provisions
Actionable retry findings
Weaknesses
REG is circular for solved tasks
Unvalidated LLM judge
Prompt-task leakage
Statistically fragile inference
Thin verification and contamination exposure
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
ISA-Bench — Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ISA-Bench?
ISA-Bench is a benchmark of 92 programming tasks built from five games (SIC-1, Human Resource Machine, TIS-100, MHRD, BOX-256) whose instruction sets are deliberately unfamiliar to language models. Each game comes with a parser, virtual machine, and test verifier so models can be scored automatically and receive structured feedback across up to five attempts.
Which models performed best on ISA-Bench?
Among the 14 open-weight models (8B-32B) evaluated, Gemma-4 31B was best, solving 56 of 92 tasks (60.9%) when retries with feedback are counted, but only 23 on the first attempt. Nine of 14 models solved under 25%, and no model solved any of the 10 BOX-256 pixel-drawing tasks. Thinking models averaged 31.9% versus 13.4% for code-specialized and 9.8% for general-purpose models.
What is the reasoning-execution gap (REG) in this paper?
REG is the difference between a plan-correctness score and an implementation-correctness score (each 0-3) that GPT-5.5 assigned to model attempts under a fixed rubric. A high REG means the model identified a plausible computational strategy but could not express it as working code in the target instruction set. Measured gaps were largest for BOX-256 (0.96) and SIC-1 (0.94) and near zero for MHRD (0.06).
Is ISA-Bench's code and data publicly available?
Yes. The paper links an anonymous open-source repository (anonymous.4open.scive mirror of the ISA-Bench repo), publishes the full system prompts and the judge prompt in its appendices, and evaluates publicly downloadable model weights via Ollama, so the evaluation pipeline is externally rerunnable.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 (environment overview, pipeline, per-game bar charts, outcome distributions, BOX-256 target patterns) are images; only their textual descriptions were reviewable.
- Appendix tables were extracted as flattened text; per-cell values were cross-checked against the prose where possible, but Table 5 column alignment carries minor transcription risk.
- The anonymous repository was not visited (offline review); artifact quality and what exactly is released are unverified.
- Possible extraction artifact: the second author's name appears as 'Arkarprava' in the paper header vs 'Arkaprava' in the metadata.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.