Clueing up LLMs with Tool-Augmented Deductive Reasoning
Rebecca Ansell, Autumn Toney-Wails
cs.AI
Quick verdict
A possibility-matrix tool gives near-perfect LLM deduction in Clue, but the tool largely solves the game itself and results rest on 6 games per condition.
Summary
The paper tests whether offloading belief-state tracking to an external tool improves multi-step deductive reasoning by LLM agents in a text-based, six-player Clue environment, extending the authors' prior work. Each game uses three GPT-4o-mini and three Gemini-2.5-Flash agents under three conditions: baseline (no tool), matrix (all agents), and mixed (one matrix plus two baseline players per family), six games each (18 total, 36 player-games per condition). The possibility-matrix tool maintains YES/NO/MAYBE values over all cards and holders, tracks envelope candidates, and emits an 'accusation ready' cell containing the solution once one candidate per category remains. Results: no baseline player ever fully solves (accuracy 0.24-0.30), while 94.5% of matrix players identify all three solution cards (Gemini perfect); incorrect deductions drop sharply for Gemini (2.39 to 0.11 per game); baseline players in mixed games improve to 0.89-0.92 without prompt changes, attributed to richer public suggestions; and agents still delay accusing 8-18 turns after the solution is certain, showing risk-averse autonomy. Key limitations, partly acknowledged by the authors: the tool effectively performs the deduction it claims to measure, the mechanism is not isolated against intermediate baselines, and only 6 games per condition are run with no statistical inference.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 4 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Injecting an externally maintained possibility matrix into agent prompts yields near-perfect accusation accuracy (94.5% full solves; Gemini 1.00) versus zero full solves for baseline agents.
- A minority of tool-augmented agents improves the performance of non-tool agents in mixed games (baseline accuracy 0.89-0.92 vs 0.24-0.30), attributed to more informative public suggestions.
- Structured belief tracking reduces unsupported inferences, most clearly for Gemini-2.5-Flash (incorrect deductions 2.39 to 0.11 per game).
- Agents remain autonomous decision-makers: they delay final accusations by mean 8-18 turns even after the tool signals the solution is certain, indicating risk-averse play unaffected by the tool.
- Win outcomes appear driven more by model behavior (GPT-4o-mini's willingness to accuse; 13/18 wins) than by tool use.
Strengths
Carefully instrumented environment enabling objective evaluation
All gameplay elements are recorded (game log, state, hands, cards seen, prompts, reasoning; Sec 3.4), allowing programmatic ground-truth labeling of every deduction against true card assignments rather than relying on model self-report.
Large, clean effect on the target task
Baseline players never fully solve across all 6 baseline games (mean accuracy 0.24-0.30, Table 1), whereas 34/36 matrix players reach 3/3 (94.5%, Sec 4.1) and Gemini's incorrect deductions fall from 2.39 to 0.11 per game (Table 1) - the headline contrast is unambiguous even at this sample size.
Novel mixed-agent design addressing spillover (RQ2)
Behavioral analysis beyond accuracy
Candid limitations section
Weaknesses
The tool largely solves the task it claims to measure
No statistical inference anywhere
Mechanism unisolated
Undiscussed internal contradiction in MIXED
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the possibility matrix tool in this paper?
It is an external component that maintains, for each LLM agent playing Clue, a table of YES/NO/MAYBE values for each of the 21 cards against each possible holder (six players plus the hidden envelope). It is updated automatically from public suggestions, passes, and private card reveals, injected into the agent's prompt every turn, and outputs an 'accusation ready' cell containing the solution once exactly one candidate remains per category (suspect, weapon, room).
How much did tool augmentation improve the LLM agents' Clue performance?
Dramatically on accuracy: in baseline games no agent ever fully identified the solution (mean accusation accuracy 0.24-0.30 out of 1), whereas 94.5% of tool-augmented players identified all three solution cards and Gemini-2.5-Flash was perfect (1.00). Even non-tool agents in mixed games improved to 0.89-0.92, apparently by benefiting from richer public suggestions.
Which LLMs were tested and how many games were played?
GPT-4o-mini and Gemini-2.5-Flash, with three agents of each family in every six-player game. The study ran 18 games total: six all-baseline, six all-tool (matrix), and six mixed (one tool user plus two baseline players per model family), each capped at 20 rounds.
Did the tool make the agents accuse immediately once they knew the answer?
No. Even after the tool signaled that the solution was certain, agents waited on average 8-18 further turns before making the winning accusation, continuing to gather information instead of acting. The authors read this as evidence that the structured state informs but does not control agent decisions, so the tool did not induce risky, early accusations.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 are rendered as garbled inline text in the extraction; figure-dependent claims were verified against Table 1 and in-text numbers only.
- The per-player accuracy grid (Figure 3) and timing plots (Figures 5-6) could not be fully cross-checked cell by cell; quoted percentages were verified arithmetically where possible (e.g., 34/36 = 94.5%).
- No appendix or prompt text is included in the extracted text, so prompt-level details that may drive behavior cannot be assessed.
- Minor extraction artifacts exist in the references (e.g., HTML entities in the Diplomacy entry); they do not affect the review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.