Playing log(N)-Questions over Wikipedia Abstracts: Communication Efficiency Between Paired Frontier Models
cs.CL
Quick verdict
Frontier-LLM self-communication win rate follows p^log2 N (p=0.928) with Opus 5 last; credible paired design and honest reporting, thin per-cell statistics.
Summary
This technical report evaluates six frontier LLMs on the two-agent log(N)-Questions game: a questioner seeing N Wikipedia lead paragraphs (N=4 to 1024) must identify a hidden target document in exactly log2(N) yes/no questions posed to an answerer who sees only the target. Both roles run on the same model, measuring zero-shot self-communication under information asymmetry. Across 408 games (total cost $363), Kimi K3 (56/68), Gemini 3.8 Flash (55/68), Grok 4.6 (51/68), GPT-5.6 Sol (49/68) and GLM-5.3 (45/68) cluster together while Claude Opus 5 trails badly (28/68) — an ordering that inverts general-intelligence leaderboards. Pooled win rate of the leading five follows win=p^log2(N) with per-round reliability p=0.928 (r=-0.973). Three-judge adjudication (2,931 judgements; 97.8-98.7% agreement, self-judgements excluded) splits losses into answer errors (47-55%) and discrimination failures (42-49%), with prediction errors negligible (2-4%). All 34 unanimous answer errors of the weakest model were 'No' replies on properties stated in the document's first sentence. Information per question correlates with win rate (r=+0.88); only the two models that bisect on document titles extract a full bit per question. Limitations: one document set per size with eight games per cell, prompts tuned in a pilot run solely on Opus 5, non-deterministic arms at provider-default temperatures, and no confidence intervals on the headline parameter.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Pooled over the five leading models, win rate follows win = p^log2(N) with a single per-round reliability parameter p = 0.928 across nine set sizes (r = -0.973).
- Losses decompose roughly equally into answer errors (47-55%) and discrimination failures (42-49%), with prediction errors negligible (2-4%), under three-judge adjudication with self-judgements excluded.
- Claude Opus 5 finishes far behind (28/68 vs 45-56), inverting general-intelligence leaderboards; 32 of its 34 unanimous answer errors are 'No' replies on properties stated in the document's first sentence.
- Information per question, estimated from answer balance, correlates with win rate at r = +0.88, and only the two models that partition on document titles extract a full bit per question.
- Two China-hosted providers refuse ordinary Wikipedia paragraphs (located by bisection), and cost is uncorrelated with performance (r = -0.05).
Strengths
Paired, frozen protocol makes cross-model comparison clean
All six arms run on identical documents in identical order with identical targets (§4.2), with nested sets and bit-reversed prefix-extensible targets so any run extension reuses completed games. Every cross-model difference is therefore paired, not confounded by task sampling.
The central compounding claim is tested and honestly caveated, not just asserted
§6.1 fits win=p^log2(N) (p=0.928, r=-0.973 over nine sizes) and reports a stationarity check on per-round failure rates (chi-square=10.5, df=9, p=0.31), while explicitly flagging that the fitted curve exceeds 1 at N=4 and that the fit is descriptive rather than derived.
Error decomposition is more rigorous than typical LLM-as-judge usage
Exemplary transparency about engineering failures and full cost
Weaknesses
Per-cell statistical power is thin and the N-curve is confounded
Correlation-heavy inference on 6-9 points with no multiplicity control
Prompts were tuned exclusively on the last-place model
Uncontrolled non-determinism and effort mismatch
Abstract overstates what the independence test shows
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper ports the two-agent log(N)-Questions game of Potash and Suleman (2019) to six frontier LLMs, each playing both questioner and answerer over nested sets of N=4 to 1,024 Wikipedia lead paragraphs. Its claims: (1) pooled win rate of the five leading models follows win=p^log2(N) with per-round reliability p=0.928 across nine sizes (r=-0.973, §6.1); (2) losses split roughly equally into answer errors (47-55%) and discrimination failures (42-49%), with prediction errors at 2-4% (§6.4); (3) Claude Opus 5 finishes last (28/68 vs 45-56), inverting general leaderboards, with 32 of 34 unanimous answer errors being 'No' replies on first-sentence facts (§6.4, §7); (4) information per question correlates with win rate (r=+0.88), and only the two title-bisecting models extract a full bit (§6.2, §6.3); (5) China-hosted APIs refuse ordinary encyclopedia content, and cost is uncorrelated with performance (§5.5, §11). The authors repeatedly and correctly frame this as a task-specific evaluation, not a capability ranking.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the log(N)-Questions game for LLMs?
It is a two-agent game in which a questioner sees N documents and must identify one secretly chosen target using exactly log2(N) yes/no questions. An answerer sees only the target document and each question, replying with one word. When both roles are played by the same model, the game measures how well that model communicates with itself across an information asymmetry, since failures cannot be blamed on a capability gap between different agents.
Which model performed worst in this study and why is that surprising?
Claude Opus 5 won only 28 of 68 games (41%), far behind Kimi K3 (56), Gemini 3.8 Flash (55), Grok 4.6 (51), GPT-5.6 Sol (49) and GLM-5.3 (45). The surprise is that published general-intelligence leaderboards rank Opus 5 at or near the top. The authors trace its losses mainly to a strong 'No' bias in the answerer role: 32 of 34 unanimous answer errors were 'No' replies about properties stated in the document's first sentence.
How much did this 408-game LLM evaluation cost?
The full study cost $363.15 across six model arms of 68 games each, spanning document sets from 4 to 1,024 Wikipedia paragraphs. Costs per win ranged from $0.36 (GLM-5.3) and $0.39 (Gemini 3.8 Flash) to $2.46 (GPT-5.6 Sol), and cost was essentially uncorrelated with win rate (r=-0.05). An additional ~$47 was spent on a Gemini arm discarded due to an engineering bug, and ~$21 on pilots and debugging.
Why did some Chinese AI providers refuse parts of the Wikipedia corpus?
Moonshot (Kimi K3) refused three of the 1,024 paragraphs — on Taiwan's navy, a purged Chinese intellectual, and a detained Hong Kong activist — and Z.ai (GLM-5.3) refused the third of these. The refusals arrived as opaque errors naming the whole prompt, so the author located the triggering paragraphs by bisection, iteratively halving the document block. Blocked documents were substituted with unused pool documents for the affected arms only, and the substitutions are recorded in the released result files.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata abstract is truncated mid-sentence ('a strategy ab'), but the full abstract and body text through the references appear intact in the extracted text.
- Figures are available only as textual descriptions and embedded table data; curve shapes and plot panels could not be inspected directly.
- External leaderboard scores cited in §8 (e.g., Artificial Analysis values) and 2026-era model documentation could not be verified independently within this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.