EnigmaForge: The Question Is Hidden in the Story
cs.AI
Quick verdict
Claims a verified, renewable benchmark for hidden-question solving in LLMs; strong generator design and controls, but thin citations and no human baseline.
Summary
EnigmaForge is a procedurally generated benchmark in which language models receive a bundle of narrative documents with no stated question; a small logic puzzle is hidden in the record, and the model must notice it, work out what it is, solve it, and take the action the record's rules require. Every instance is minted from a seed and verified at generation time: a DPLL solver proves the intended solution unique, an independent brute-force oracle cross-checks every instance, and ablation certificates show each clue is load-bearing. The evaluation runs 25 frontier models and 4 deterministic baselines over 600 instances (120 families, 17,400 scored records) under three matched conditions — question stated, formal world stated, or nothing stated. The headline measure, intuition (task success with no stated question), reshuffles the leaderboard: a 22× spread where fact recovery spans only 1.6×, with the second-best fact-recoverer ranking fourteenth. Most models lose 9–37 points when the question is withheld; Grok-4.6 is indifferent, and GPT-6 Sol performs significantly better without it. Content filters blocked a substantial share of attempts for several models, reported as missing mass rather than failure. Limitations include the absence of a human baseline, thin related work, and untested generator-overfitting risk.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- EnigmaForge mints narrative-embedded logic puzzles from seeds whose unique solutions are proved by a SAT/DPLL engine with brute-force cross-checks and per-clue ablation certificates, making the corpus renewable and structurally contamination-resistant (§3).
- Intuition — implicit-condition task success — spreads the frontier 22× (3.6–80.1) while fact F1 spans only 1.6×, and the two orderings disagree sharply (the second-best fact-recoverer ranks fourteenth) (§5.1–5.2).
- Withholding the stated question costs most models 9–37 points, is irrelevant to Grok-4.6 (0.000, CI [−0.062, +0.058]), and significantly improves GPT-6 Sol (−0.142, CI [−0.204, −0.075]) (§5.4).
- Content-filter blocks are condition-correlated and substantial (585/600 for one model), so benchmarks scoring refusals as failure run a hidden filter-survival axis through their leaderboards (§5.5).
- World reconstruction is no longer the bottleneck: earned decisions (correct action from a fully correct world) span 28–88 with only one model above 80, locating current failure in decision conversion (§5.3, §6).
Strengths
Proof-carrying generation
Uniqueness is proved by requiring UNSAT for the solution's negation (§3.2), a brute-force oracle must agree on every instance, and per-clue ablation certificates (§3.3) force every clue to be load-bearing — ground truth is verified at birth rather than trusted.
Matched conditions yield a genuinely novel measurable
Every family appears in explicit, formal, and implicit conditions; paired within-family contrasts turn 'does being told the question help?' into a per-model quantity with CIs (§4, §5.4), including a significantly negative case (GPT-6 Sol, −0.142, CI [−0.204, −0.075]).
Self-voiding controls and honest floors
Filter interference reported as missing mass
Reproducibility and provenance
Weaknesses
Thin related work undermines the novelty claim
Logical contradiction in the worked example
No external anchor for difficulty or performance
Internal inconsistencies and underspecified scoring
Untested adversarial contamination and single-config runs
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EnigmaForge (arXiv 2609.30144, cs.AI) proposes a procedurally generated benchmark for *problem discovery*: language models receive a bundle of narrative documents with **no stated question** and must notice the embedded logic puzzle, infer what it is, solve it, and take the action the record's rules require (§1). The paper claims three design contributions — proof-carrying generation (§3), matched explicit/formal/implicit conditions (§4), and *intuition* (implicit-condition task success) as a headline measure that stated-question benchmarks cannot register (§5). Central empirical claims: intuition spreads the frontier 22× (3.6–80.1) while fact F1 spans only 1.6×; the second-best fact-recoverer (Claude Opus 5.5, F1 0.919) ranks fourteenth on intuition; withholding the question costs most models 9–37 points, leaves Grok-4.6 unmoved (0.000, CI [−0.062, +0.058]) and significantly helps GPT-6 Sol (−0.142, CI [−0.204, −0.075]); and content-filter interference is condition-correlated and must be reported as missing mass (§5.5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the EnigmaForge benchmark?
EnigmaForge is a procedurally generated LLM benchmark where the model receives a set of fictional documents (letters, receipts, logbook margins) with no question attached. A small logic puzzle with a provably unique solution is hidden in the record; the model must notice it, work out what it is, solve it, and take the action the record's own decision rule requires.
What does 'intuition' mean in the paper?
Intuition is task success when the model is given only the story, with no stated question (the implicit condition). It is reported alongside fact recovery (world reconstruction) and guided success when the question is stated, and it separates models far more sharply than fact recovery does.
What were the main findings of EnigmaForge?
Across 25 frontier models and 600 instances, unguided task success spread 22× (3.6 to 80.1) while fact recovery spanned only 1.6×, reordering the leaderboard. Most models lost 9–37 points without the question; Grok-4.6 was indifferent and GPT-6 Sol scored significantly better without it. Several models were heavily blocked by their own content filters.
Is the EnigmaForge code and data public?
Yes. The paper points to a public repository (github.com/robottwo/enigmaforge) with the generator, seeds, settings, per-module SHA-256 provenance, the 17,400-record results artifact, and an interactive leaderboard; any instance regenerates from its seed with one command.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 appear only as captions and model-name lists in the extraction; data points could not be analyzed, so ordering and difficulty-trend claims rest on prose and tables.
- Table 1's column headers extracted garbled ('Task Disc. ret. Earn. dec. Reas. disc. Cover.'), so per-column values for individual models may be misaligned.
- Only two references appear; the reference list may be truncated in extraction, making the thin-citation criticism provisional.
- The abstract/§4 five-vs-six difficulty-level discrepancy may be either an authoring inconsistency or an extraction artifact; treated here as authoring.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.