The Convention Gap: Towards Measuring Implicit Communication in Cooperative AI Evaluation
Makoto Fukushima, Hua-Dong Xiong, Ehsan Moradi Pari
cs.AI · cs.HC
Quick verdict
Proposes an exactly computable 'convention gap' for implicit communication in Hanabi; strongly validated via OBL dose-response; cross-dataset confound remains.
Summary
The paper introduces the 'convention gap': the difference between a posterior probability of life loss computed exactly from literal hint information in Hanabi and the observed loss rate, proposed as a metric of implicit convention use in cooperative play. Using a custom replay engine, the authors process ~101,000 play actions across three public datasets: human-human (hanab.live, 9,017 plays), AI-AI (HOAD, 62,890 plays), and human-AI (HanabiData; 15,472 human plays analyzed). The gap forms a gradient: +26.2 pp human-human, +16.4 pp human-AI, -0.7 pp AI-AI, concentrated at plays of cards with zero or one hint (+46 pp human pairs at zero hints), collapsing at two or more hints. Within human-AI play, gaps vary by partner (Full +24.1, Intentional +20.2, Outer +6.2 pp) and replicate within participants. A known-answer validation on Off-Belief Learning agents yields +1.6 pp at the convention-free level rising monotonically to +21.7 pp, matching the designed hierarchy. Main limitations: the cross-setting gradient is confounded with dataset provenance; only three rule-based AIs from a single 2017 study supply the human-AI evidence; and the 'convention' interpretation partly subsumes other non-literal channels such as positional and timing inference. Code, data, and a reproduction notebook are released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The convention gap — exactly computable in Hanabi from literal hint information — separates cooperation settings: +26.2 pp human-human, +16.4 pp human-AI, −0.7 pp AI-AI (Table 2).
- The human gap concentrates where literal information is absent: +46 pp at zero-hint plays and +36 pp at one-hint plays in human pairs, collapsing to ≈0 at two or more hints (Table 3).
- Within human-AI play the gap varies by partner (Full +24.1 > Intentional +20.2 > Outer +6.2 pp) and moves inversely with human failure rates; the Full-vs-Outer ordering replicates within participants (App. L).
- Applied to Off-Belief Learning agents whose convention content is controlled by construction, the metric recovers the designed hierarchy (+1.6 pp at OBL1 rising monotonically to +21.7 pp), validating construct validity (§4.5).
- Game score depends on corpus roster composition while the convention gap separates actor types at the agent level, so benchmarks should report both (§4.4).
Strengths
Exact, auditable baseline rather than a learned one
The literal-information posterior (§3.2, App. D/E) is exact enumeration with hard hint constraints and falling-factorial joint conditioning. Verification is layered: global invariants (no posterior-0 play failed among 70,247; no posterior-1 succeeded among 1,641), brute-force agreement on 936 plays, and a hand-checkable worked example (5/8 = 0.625).
Known-answer validation with a pre-registered dose-response prediction
§4.5/App. X: OBL hierarchy gap reads +1.57 pp [+1.33, +1.81] at the convention-free level, rising monotonically to +21.68 [+21.34, +22.02]; OBL1 cross-play over 10 seeds holds at +1.22; the zero point is shown to be partner-supplied (decay to +0.1 vs OBL5, Kitagawa decomposition attributing 72% to within-stratum change). This is textbook construct validation.
Unusually thorough robustness work
Candid reporting of negative and unresolved results
Reproducibility infrastructure
Weaknesses
Headline cross-setting gradient is confounded with dataset provenance
Narrow human-AI agent coverage
Construct scope: 'convention gap' conflates multiple non-literal channels
Per-partner gap ordering is arithmetically tied to loss rate
OBL validation has single-seed limitations
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper proposes the **convention gap**: the difference between the mean posterior probability of life loss — computed exactly from literal hint information in two-player Hanabi — and the observed life-loss rate, as a metric of implicit convention use (§3.1). Its five claims: (1) the gap separates cooperation settings (+26.2 pp human-human, +16.4 pp human-AI human plays, −0.7 pp AI-AI; Table 2); (2) it localizes where conventions operate (+46/+36 pp at zero/one hint in human pairs, collapsing to ≈0 at 2+ hints; Table 3); (3) it varies by AI partner in human-AI play (Full +24.1, Intentional +20.2, Outer +6.2 pp; Table 4), inversely tracking human failure; (4) it carries information game score does not (§4.4); (5) it passes a known-answer test on the Off-Belief Learning hierarchy (+1.57→+21.68 pp monotone; Table 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'convention gap' in this paper?
It is a metric for cooperative play, defined in the game Hanabi as the difference between the failure probability predicted from only the literal content of players' hints (an exactly computable probability) and the actually observed failure rate. A large positive gap means players succeed more often than literal information alone can explain — evidence that they read meaning beyond the literal message using shared implicit conventions.
What are the paper's main findings?
Across about 101,000 Hanabi play actions, the gap was +26.2 percentage points for human-human pairs, +16.4 pp for human-AI pairs, and −0.7 pp for AI-AI pairs. It was largest when a played card had received no hints (+46 pp in human pairs) and vanished once two or more hints touched the card. Validation on Off-Belief Learning agents, whose convention content is controlled by design, produced a monotonic rise from +1.6 to +21.7 pp, matching the intended hierarchy.
Is the code and data available?
Yes. The paper states that code, processed datasets (hanab.live usernames replaced by hashes), and an executed reproduction notebook are released at github.com/dockmfgit/hanabi-convention-gap, archived on Zenodo (doi:10.5281/zenodo.21975884). Everything runs CPU-only; raw hanab.live games can be refetched by game id via the public API.
Does a positive convention gap prove that humans use conventions?
It proves that human play succeeds more often than an exact literal-information baseline predicts. The authors defend the convention interpretation with several controls (AI calibration, hint-count localization, OBL dose-response), but the residual also pools other non-literal channels such as positional and timing inference, so the interpretation is partly conceptual rather than a strict identification.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt (head and tail kept, marked middle gaps); some appendix details (e.g., Appendices B, I, J, U, V and parts of §3.3 and the Discussion) were skipped and not directly verified, though their headline numbers are quoted in visible sections.
- Figures were not extracted as images; all figure-related judgments rely solely on the captions present in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.