Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMs
Kohsei Matsutani, Gouki Minegishi, Core Francisco Park, Takeshi Kojima, Yusuke Iwasawa, Yusuke Iwasawa, Yutaka Matsuo, Yutaka Matsuo
cs.AI
Quick verdict
Shows few-shot demos from different worlds boost in-context graph prediction and causally engage world representations; careful, though effects are modest.
Summary
This paper studies why large language models fail to use world representations they can construct in context. Building on Lepori et al. (2026), who showed that after observing a random walk on a latent graph, LLMs encode the graph topology internally yet fail a two-step-right node prediction task zero-shot, the authors add few-shot demonstrations, each generated from a different world sharing the same graph topology but a different vocabulary assignment. Across six base models from four families (Qwen3-4B/8B/14B, Llama-3, Gemma-4, Ministral-3) and five graph topologies with 300 test samples per condition, accuracy rises monotonically with 0-32 shots, far above the 4.0-6.7% random baselines. Probing shows linear decodability of adjacency (AUROC) and normalized Dirichlet Energy remain flat as shots increase, indicating unchanged representation construction; instead, the probe-defined world representation subspace relocates, shifting in directions up to roughly 80-90 degrees from the zero-shot subspace, and subspace distance between consecutive shot counts shrinks (the 'Shot Vector'). Iterative amnesic-style interventions on this subspace degrade correct-answer probability more in 16-shot than 0-shot settings relative to matched random orthogonal interventions, with bootstrap CIs over 300 paired cases, supporting increased causal use. Gains from different-world demonstrations also appear on ARC-AGI-1&2, Mind2Web cross-website, and Othello. Limitations: world representations cover static structure only, intervention effects are modest, absolute accuracies stay well below 50% on most topologies, and the practical-task section offers no internal-representation analysis.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Few-shot demonstrations generated from different worlds (same topology, different node-vocabulary assignments) monotonically improve two-step-right graph prediction across 6 models from 4 families, far above 4.0-6.7% random baselines.
- Representation construction is unchanged by shots: linear probe AUROC and normalized Dirichlet Energy stay flat from 0 to 32 shots, so improvement comes from how representations are read out, not from better construction.
- Few-shot demonstrations relocate the world representation into subspaces nearly orthogonal (up to ~80-90 degrees) to the zero-shot subspace, with the shift direction (Shot Vector) distinct from and near-orthogonal to Task Vectors.
- Iterative interventions on the world representation subspace impair correct-answer probability more in 16-shot than 0-shot settings relative to matched random orthogonal interventions, indicating increased causal use.
- Different-world few-shot demonstrations also improve performance on ARC-AGI-1&2, Mind2Web cross-website, and Othello, suggesting the effect generalizes beyond synthetic graphs.
Strengths
Cleanly controlled behavioral experiment
Section 3 fixes the query across all shot counts (0-32) and sources demonstrations from different worlds (same topology, different fruit assignments), explicitly ruling out copying across demos; 300 test samples per topology per model, five topologies, with random-guess baselines of 4.0-6.7% stated per grid.
Representation construction is dissociated from predictive use
Section 4.1 shows probe AUROC and normalized Dirichlet Energy stay flat across shots (e.g. layer 20 Qwen3-4B, AUROC ~0.70-0.75 vs ~0.5 random-feature baseline) while Figure 5(b) shows probe-subspace overlap with 0-shot dropping to ~0.16-0.32 at layer 36, cleanly separating construction from readout.
Interventions use a matched control
Robustness and generality checks
Weaknesses
Absolute performance remains weak
Causal effect sizes are modest and the control behaves non-trivially
Practical-task claims lack mechanistic support
Key alternative baseline underexplored
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper asks why LLMs fail to *use* world representations they can *construct* in context. Lepori et al. (2026) showed that after observing a random walk on a latent graph, models encode the topology (recoverable by probes) yet fail zero-shot at predicting the node two steps right. Matsutani et al. extend this with few-shot demonstrations, each generated from a different world: same topology, different fruit-to-node assignment. Central claims: (1) different-world demonstrations monotonically improve accuracy from 0 to 32 shots on 6 base models across 4 families (Qwen3-4B/8B/14B, Llama-3, Gemma-4, Ministral-3); (2) this gain is not better representation construction (probe AUROC and normalized Dirichlet Energy are flat, Figure 4) but relocation of the probe-defined world representation subspace into directions up to ~80-90 degrees from the zero-shot subspace (Figure 5b-c), with the shift (Shot Vector) distinct from Task Vectors (D.3); (3) iterative amnesic interventions on this subspace selectively degrade predictions more at 16-shot than 0-shot versus norm-matched random orthogonal interventions (Figures 7, 25); (4) the pattern transfers to ARC-AGI-1&2, Mind2Web cross-website, and Othello (Figure 8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper on few-shot demonstrations and world representations?
The paper finds that adding few-shot demonstrations, each drawn from a different 'world' (same graph topology but different word-to-node assignments), lets LLMs succeed at a graph prediction task they fail zero-shot. Mechanistically, the internal graph representation does not improve; instead it relocates to a nearly orthogonal subspace from which the model can actually read it out for prediction, and interventions on that subspace selectively hurt performance.
Which models and tasks were used in the study?
Six open-weight base models from four families: Qwen3-4B/8B/14B-Base, Llama-3, Gemma-4, and Ministral-3. The core task is two-step-right node prediction on five graph topologies (lattice, triangle, torus grids) from random-walk observations. Generalization tasks are ARC-AGI-1&2 (pixel accuracy@2), the Mind2Web cross-website web-agent benchmark, and Othello legal-move prediction.
Does this mean few-shot examples fix LLMs' inability to use learned world models?
Only partially. Accuracy rises substantially from zero-shot but mostly stays below 50% even at 32 demonstrations, and the causal-intervention evidence shows increased but still partial use of the world representation. The authors themselves note the gains are not solely due to world-representation use. The contribution is a mechanistic account of why few-shot demonstrations help, not a solution.
Is code available for this paper?
Yes, the paper states code is available in the public repository kohseim/icwm, and all models used are open-weight, with experiments on synthetic graphs and public datasets (ARC-AGI, Mind2Web, Othello), which makes reproduction feasible at moderate GPU scale.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Section 4.2 body text is truncated in the extraction; the subspace-relocation narrative is reconstructed from Figure 5, Figure 20-24 captions, and the Discussion.
- Several figures (3, 16-19, 22-25) appear as raw axis labels and number grids, so visual trends and error-bar presence could not be verified directly.
- Appendix E.1 cross-family results (Llama-3, Gemma-4, Ministral-3 probing/interventions) are referenced but only partially included, limiting verification of family generality.
- Figure 7 (main intervention figure) was not present in the extracted text; assessment relies on Section 5.1 prose and the detailed Figure 25.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.