WordPolo: Evaluating Language Models Through Iterative Semantic Feedback
cs.CL · cs.AI
Quick verdict
Contexto-style benchmark that scores reasoning process via novel progression metrics; insightful, but LRM conclusions rest on small n=50 subsets.
Summary
The paper introduces WordPolo, a Contexto-derived benchmark in which a player discovers a secret word through iterative guesses scored by rank in GloVe-6B cosine-similarity space (~317K vocabulary). Because every guess must be interpreted against all previous feedback, the task entangles reasoning process with outcome. The authors evaluate four LLMs (GPT-4.1, Claude 3.5 Haiku, Llama 4 Maverick, Qwen 3 without reasoning), three LRMs (o4-mini, DeepSeek-R1, Qwen 3 with reasoning), three human annotators, a weighted-centroid heuristic, and a random baseline on 1,500 puzzles (979 from a public Contexto solution archive, 521 from the MsFit crossword corpus); LRM and human runs use a 50-puzzle subset. Beyond solve rates (4%-62%), they propose Proximity, Mean Progression, and Weighted Progression metrics that credit partial progress in failures. Key findings: o4-mini leads (62% on n=50) while other LRMs show no advantage; Qwen 3 performs better with reasoning disabled (36% vs 12%); Llama 3.1 solve rates scale with size (0/8/36% at 8B/70B/405B); an error taxonomy (overthinking, underthinking, output errors) derives from ~400 annotated guesses; ablations cover history ordering, truncation, prompts, and embedding spaces. Main limitations: the small n=50 LRM/human subset, memorization risk from archived puzzles, and unvalidated error annotations.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Solve-rate-only benchmarks conceal reasoning-quality differences that WordPolo's progression and proximity metrics expose.
- LRMs other than o4-mini show no systematic advantage over standard LLMs on WordPolo, and reasoning tokens actively hurt Qwen 3 (12% vs 36% solve rate without them).
- LRM failures arise from overthinking, underthinking, and output/faithfulness errors that compound over a 100-guess game.
- Successful players share adaptive early exploration and late-game refinement, with top LLMs mirroring human progression patterns.
- Findings are robust to the embedding space: model rankings persist across GloVe 2014, GloVe Dolma, and fastText 2M despite low cross-space rank correlations.
Strengths
Novel, automatic process-based metrics
Proximity Score, Mean Progression, and Weighted Progression (§4.3) credit partial progress in failed games, which pure solve rates cannot. Appendix E (Table 6) convincingly justifies the log-scaled Proximity design: linear scaling compresses GPT-4.1 vs Claude 3.5 to a 0.003 gap despite a ~116-rank difference in average best guesses.
Unusually thorough ablations
History ordering vs rank ordering and top-10 truncation (App. D, Tables 4-5), altered prompt exemplars (App. C.1, Table 3), Markdown stripping (App. H, Table 9), forced explanations (App. I, Table 10), and three embedding spaces with Spearman correlations (App. G, Table 8) — most benchmark papers ship none of these.
Honest statistical reporting by genre standards
Decision-relevant, counterintuitive findings
Weaknesses
Central LRM claims rest on a 50-puzzle subset
Memorization/contamination risk is asserted away, not tested
Mean Progression barely discriminates reasoning from noise
Error taxonomy is thin and unvalidated
Single-run protocol and prompt sensitivity underexplored
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
WordPolo (arXiv:2609.19006) recasts the Contexto word game as a benchmark for LLM and LRM reasoning. A player guesses a secret word; each guess receives a rank from 1 (the target) to ~317,756, computed by cosine similarity in GloVe 6B space with Faiss nearest-neighbor search. The central claim is that iterative, feedback-dependent tasks make reasoning observable and unavoidable, and that solve rate alone hides meaningful variation. Secondary claims (§1): (1) accuracy conceals reasoning-quality differences; (2) LRMs are hindered by overthinking, underthinking, and unfaithfulness; (3) successful players share adaptive exploration and late-game refinement. The paper positions itself against puzzle benchmarks (GameArena, GuessArena, TurtleBench) and post-hoc process metrics (ROSCOE, ReCEval), distinguishing itself by automatic per-guess scoring.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is WordPolo?
WordPolo is a benchmark adapted from the word game Contexto. A player tries to discover a secret word by guessing; every guess receives a rank from 1 (the target) up to ~317,000, computed from cosine similarity between words in GloVe embedding space. Players get up to 100 guesses and must use the ranks of previous guesses to navigate toward the answer.
Which models perform best on WordPolo?
In the paper, o4-mini leads with a 62% solve rate on the 50-puzzle subset, ahead of GPT-4.1 (42%) and a novice human annotator (48%); an expert human solved 100%. Claude 3.5 Haiku (6%), DeepSeek-R1 (16%), and Qwen 3 with reasoning enabled (12%) underperform, and the heuristic baseline reaches 50%.
Why do reasoning models underperform on WordPolo?
The authors' manual annotation of about 400 guesses attributes LRM failures to overthinking (running out of response limits while deliberating), underthinking (fixating on one semantic area and ignoring feedback), and output errors (formatting violations or guesses that differ from what was reasoned). Notably, Qwen 3 scored better with reasoning tokens disabled (36% vs 12% solve rate).
Could models have memorized WordPolo answers?
Possibly: 979 of the 1,500 puzzles come from a public archive of daily Contexto answers spanning 2022-2025, so the words and the game itself may appear in model training data. The authors argue their progression metrics would expose memorization (a model that suddenly guesses correctly without steady progress), but they do not report a direct comparison between archived and newly sourced puzzles.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (heatmaps, word clouds, progression curves) are present only as captions and embedded text fragments; their visual content could not be verified.
- The demo/repository link was not accessed; actual code and dataset availability beyond the paper's claim is unconfirmed.
- Some table values were extracted from dense running text (e.g., Table 1) and could contain minor transcription artifacts, though internal cross-references are consistent.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.