LLMs Learn Better In-Context from Rules than from Examples
Xiang Fu, Seungmin Cho, Yukyung Lee, Najoung Kim
cs.CL
Quick verdict
Claims rules beat examples for in-context learning; a controlled 5-task, 3-family study largely supports this, with idealized-rule caveats.
Summary
The paper systematically compares two modes of in-context learning — rules (verbal task descriptions) versus examples (input-output demonstrations) — on five programmatically generated tasks spanning games (Set Game, Tapatan), arithmetic (Operator Function), and linguistic inference (Noun Class Agreement, Lexical Category Inference), each at three difficulty levels. Base and instruction-tuned checkpoints from Gemma 3, Qwen 2.5/3, and OLMo 3/3.1 (7B–32B) answer identical test items under rules-only, examples-only, and combined conditions; GPT-5.4 serves as a reference excluded from statistics. Linear mixed models show rules beat examples overall (β=+0.171, p<.001), but the advantage is task-dependent: largest for Operator Function (+0.401) and Set Game (+0.319), absent for Tapatan and Lexical Category Inference, and base models even significantly favor examples on Tapatan (−0.122). Adding examples to rules yields no significant gain (+0.012). Instruction tuning improves rule-based learning (+0.213) without degrading example-based learning (−0.024). Example-based learning shows the predicted conjunctive-over-disjunctive advantage (+7.1 to +10.7 points) that rules eliminate. Scaling demonstrations mostly plateaus or degrades. Main limitations: rule prompts were iteratively refined for clarity while examples were not optimized, possible pretraining exposure to the game variants, and scaling experiments confined to one model family.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Open-weight LLMs (7B-32B) learn novel complex tasks more reliably from rules than from minimum-coverage examples, overall beta=+0.171 (p<.001, Figure 3).
- Adding examples on top of rules yields no statistically significant gain (beta=+0.012), and scaling demonstration counts mostly plateaus or degrades performance (Appendix C).
- Instruction tuning amplifies rule-based learning (+0.213) while leaving example-based learning statistically intact (-0.024); example-based learning enjoys no privileged status in base models.
- The rule-over-example advantage scales with the task's need for algebraic abstraction (Operator Function +0.401, Set Game +0.319) and shrinks toward zero for distributional or parametric-knowledge tasks (Tapatan +0.020, Lexical Category Inference +0.014/+0.066).
- Example-based learning shows a conjunctive-over-disjunctive advantage (+7.1 base, +10.7 instruct points) that rule-based learning eliminates, consistent with similarity-based induction.
Strengths
Paired, task-controlled design
The same latent task and identical test items are used across rules, examples, and combined conditions (Section 2); demonstration sets are resampled per test item with randomized ordering (seed 123), avoiding artifacts of one fixed prompt, and examples/combined conditions are matched for paired comparison (Appendix B).
Token-budget transparency preempts the length confound
Table 5 reports mean prompt tokens per condition; rules win at a fraction of the cost (Operator Function hard: 158 rule tokens vs. 7,022 example tokens), so the advantage cannot be attributed to prompt length.
Consistent statistical rigor
Honest reporting of disconfirming results
High reproducibility
Weaknesses
Asymmetric prompt engineering favors the rules condition
GPT-5.4 diverges from the central taxonomy in Table 6
No format-versus-learning error decomposition
Scaling and combination conclusions rest on narrow bases
Multiple comparisons unaddressed; variance modeling is coarse
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether LLMs acquire a novel task in context more effectively from verbal rules (instructions) or from input-output demonstrations (few-shot), holding the latent task and test items fixed. Five programmatically generated tasks — Set Game, Tapatan, Operator Function, Noun Class Agreement, and Lexical Category Inference (with AND/OR variants) — each at three difficulty levels, are presented under rules-only, examples-only, and combined prompts. Fourteen open-weight checkpoints from Gemma 3, OLMo 3/3.1, and Qwen 2.5/3 (7B-32B; base and instruction-tuned) are evaluated, with GPT-5.4 as a reference excluded from statistics. Central claims: (1) rules beat examples overall; (2) adding examples to rules gives no significant gain; (3) instruction tuning improves rule-based but not example-based learning; (4) the rule advantage tracks task demands — algebraic abstraction favors rules, distributional sensitivity and parametric knowledge shrink it.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLMs learn better from instructions or from few-shot examples?
This study of five synthetic tasks and 14 open-weight models (7B-32B) found that verbal rules beat minimum-coverage examples overall (+0.171 accuracy on a 0-1 scale, p<.001), especially on algebraic tasks like composing new arithmetic operators and the Set Game; the advantage shrinks or disappears on distributional tasks such as judging board-game outcomes or semantic category membership.
Does adding few-shot examples on top of instructions improve LLM performance?
In this paper, no: appending examples after rules changed accuracy by a non-significant +0.012, and scaling examples into the hundreds mostly plateaued or degraded accuracy on the OLMo models tested, with some drops traced to models looping repetitively until the output cap.
Does instruction tuning affect in-context learning from examples?
It improved rule-based learning substantially (+0.213 accuracy) while leaving example-based learning statistically unchanged (-0.024), meaning instruction-tuned models became better rule-followers without losing the capacity to learn from demonstrations — and base models showed no special advantage for example-based learning.
When are few-shot examples better than instructions for LLMs?
When tasks rely on distributional similarity or world knowledge rather than explicit computation: conjunctive semantic categories favored examples (+7.1 to +10.7 accuracy points over disjunctive ones), and base (non-instruction-tuned) models significantly preferred examples on the Tapatan board-state judgment task (-0.122).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extraction is complete through the appendices and Table 8; no truncation of body text was observed. However, figure-internal data (e.g., the boxplot distributions in Figure 2 and the scaling curves in Figure 13) are only partially recoverable from embedded annotations, so per-condition accuracies cited in this review come from the text-embedded tables (Tables 5-7) and coefficient annotations (Figures 3-7, 14-16).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.