For Your Eyes Only: Evaluating Coordination Between Isolated Language Model Instances
Alexander Shirnin, Aleksey Kudelya
cs.CL · cs.AI
Quick verdict
FYEO game shows isolated LLM instances coordinate zero-shot; double-pass design controls bias, but no significance tests and a weak covertness bar.
Summary
The paper introduces For Your Eyes Only (FYEO), a cooperative signalling game probing whether isolated instances of the same LLM can coordinate through natural language alone, without shared memory or task-specific training. A Sender receives two words (one designated target) and writes a description of each; an independent Receiver must identify the target. Seven models from four families were tested on 300 word pairs drawn from SimLex-999 and the Brysbaert concreteness norms, split into high-similarity, abstract, and concrete subsets. The Double-Pass Success Rate plays each pair twice with swapped targets (chance = 25%), controlling positional and lexical bias. In the unconstrained Freedom setting, five of seven models exceed 90% (Gemini 3.1 Pro scores 100%); under a Constrained setting where an LLM Inspector filters explicit markers, most models degrade sharply, while Gemini 3.1 Pro retains 96.33% (91.67% filtered). Exploratory 30-pair analyses show deliberate sabotage is possible and cross-family coordination is weaker. Key limitations: no statistical significance tests or confidence intervals anywhere; single-run temperature-0 decoding via closed APIs; Inspector reliability validated on only 100 manually reviewed samples; the 30-pair subset was selected on prior same-model success, guaranteeing the within-model diagonal in cross-model tests; and no code or data release is stated.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
5 strengths · 5 weaknesses · 0 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Zero-shot coordination between isolated instances of the same model exists broadly: five of seven evaluated models exceed 90% DPSR in the unconstrained Freedom setting.
- Instructing the Sender to avoid detectable signals degrades most models sharply (8.67-65% Constrained), but Gemini 3.1 Pro retains near-perfect coordination (96.33% Constrained, 91.67% after Inspector filtering).
- Capable models can redirect coordination toward deliberate misdirection: in the Sabotage condition Gemini 3.1 Pro reaches 100% Constrained/Filtered DPSR on a 30-pair subset, though other models are inconsistent.
- Coordination is consistently weaker across model families than within them, with cross-model Constrained-Filtered DPSR ranging from 40.00% to 83.33% versus 100% within-model on the shared 30-pair subset.
Strengths
Bias-controlled metric design
DPSR (§3.3) swaps target/distractor across two passes with fixed presentation order, so positional and lexical-preference biases each fail one pass; chance is exactly 25%, making scores directly interpretable.
Degenerate strategies foreclosed by design
The N=2 choice (§3.1) means making one description stylistically salient carries no directional information, ruling out the degenerate signalling strategy common in reference games.
Fully transparent prompts and strategy taxonomy
Human-grounded, internally consistent benchmark
Candid limitations
Weaknesses
No inferential statistics anywhere
Inspector validity underpowered
Selection bias in exploratory analyses
Token-budget confound
Reproducibility limited
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FYEO (arXiv 2609.19504v1) is a framework-plus-benchmark paper asking whether two isolated instances of the same LLM can coordinate through free-form text with no shared memory or coordination-specific training; the motivation (§1) is safety-relevant implicit state transmission between model instances. The claims, restated from §1, §5 and §6: (1) zero-shot same-model coordination exists broadly (5/7 models above 90% DPSR in Freedom); (2) a concealment constraint collapses most models while Gemini 3.1 Pro retains 96.33% (91.67% filtered); (3) coordination can be redirected into deliberate misdirection (Sabotage, §6.1); (4) coordination is weaker across model families than within (§6.2). Claims 1-2 are well supported descriptively; claims 3-4 rest on an exploratory 30-pair subset and are proof-of-concept, as the authors themselves concede.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can AI models secretly communicate with each other through text?
This paper shows isolated instances of the same LLM can coordinate through ordinary text: a Sender writes descriptions of two words and an independent Receiver identifies the hidden target above chance in five of seven models tested (unconstrained setting). Coordination drops sharply when the Sender must avoid explicit markers - except Gemini 3.1 Pro, which keeps 91.67% after filtering. The result comes from a structured game where both sides are told the rules, not from spontaneous behavior in real deployments.
What is the Double-Pass Success Rate (DPSR) in this paper?
It is the paper's main metric. Each word pair is tested twice with the target and distractor swapped between passes while the presentation order stays fixed. A trial counts as correct only if the Receiver picks the right word in both passes, so a model biased toward the first description or toward one specific word fails one pass and scores zero. Pure random guessing yields 50% x 50% = 25%, which is the stated chance baseline.
Which model was best at hidden LLM-to-LLM coordination?
Gemini 3.1 Pro. It scored a perfect 100% in the unconstrained Freedom setting across all 300 pairs, 96.33% under the concealment constraint, and 91.67% after an Inspector model filtered out outputs containing explicit markers. Most other models fell to between 8.67% and 65% under the constraint.
Does this paper prove LLMs already use steganography in the wild?
No. The authors state explicitly in the Limitations section that both Sender and Receiver are fully informed of the game and their objectives, and that results should not be interpreted as evidence that such coordination emerges in ordinary deployments like summarization, retrieval, code generation, or agentic systems. The work is a controlled capability probe, and the 'covertness' bar is defined by a lightweight GPT-4o-mini Inspector.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Full text including all appendices was extracted; figures are text-rendered examples, so visual layout and any non-textual content could not be checked.
- Review is based on arXiv v1 preprint (Sep 2026), not a peer-reviewed version; closed-API results are time-boxed to a window ending May 2026.
- No supplementary files, code repository, or benchmark release could be consulted beyond what the paper text states.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.