Memory vs. Context? Influential Factors of Factual Recall in Language Models
Guilhem Fouilhé, Nicholas Asher, Philippe Muller
cs.CL · cs.AI
Quick verdict
Solid replication and stress-test of Yu et al. (2023) over 31 models: size/frequency trends mostly hold, but phrasing alone swings memorization up to 95 pp.
Summary
This paper reproduces and stress-tests Yu et al. (2023)'s study of how language models arbitrate between memorized facts and contradictory in-context statements. The authors rerun the world-capitals substitution-conflict experiment on 31 open-weight models (Pythia 160M-12B, GPT-2 small-XL, Qwen3 and Qwen3-Base 0.6B-32B, and Ministral-3 in Base/Instruct/Reasoning stages) using 47,299 prompts, and extend to five further ParaConflict relations. Systematic ablations vary filler type, length, and position, question templates, negated conflicts, and conflict repetition. The original trends mostly hold: memorization probability rises with model size (pooled Spearman rho=+0.53) and with subject-entity frequency in 17 of 31 models, though the frequency effect vanishes on Qwen3-14B and 32B; post-training shifts the balance inconsistently across families (Qwen3 lowers it, Ministral-3 Instruct raises memorization by up to +41.4 points on World Capital). Prompting dominates: coherent supporting context raises in-context preference on every model (median delta +0.19), question-template changes alone swing memorization by up to 95 points (Qwen3-14B), while unrelated prose filler mimics coherent context only on Pythia. A logit-lens analysis shows answer preference decodes mainly in late layers. Limitations: a wordfreq Zipf proxy replaces true training-corpus frequencies, instruct/reasoning models run without chat templates under 10-token greedy decoding, and no confidence intervals or multiple-comparison control are reported. Code is public.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Larger models rely more on memorized answers under substitution conflict (pooled Spearman ρ=+0.53, p<0.001, Table 1), but the effect is non-linear and strongly family-dependent.
- Subject-entity frequency effects replicate in 17 of 31 models but disappear on Qwen3-14B and 32B (Section 4.2, Table 6).
- Post-training shifts the memory-context trade-off inconsistently across families (Qwen3 pooled -12.7 pp vs. Ministral-3 Instruct +20.9 pp, Table 2).
- Question-template changes alone swing P(memorized) by up to 95.0 percentage points at a fixed conflict statement (Qwen3-14B, Appendix K).
- Coherent supporting context raises P(in-context) on every one of the 31 models (median Δ+0.19, Table 5), whereas semantically unrelated prose does so consistently only on Pythia models above 1B (Appendix C).
Strengths
Scale and coverage of the replication
31 checkpoints across four families and training stages with exact HF identifiers, dtypes, and vocab sizes (Table 9); 47,299 world-capital prompts (218 countries) plus five further ParaConflict relations (n=193-532), going well beyond Yu et al.'s Pythia/GPT-2 scope.
Validated dual answer classification
Greedy 10-token generative substring matcher cross-checked against a teacher-forced log-probability argmax over both candidates (Appendix H: Pearson r≈0.98 for P(mem)); the lower P(ctx) agreement (r≈0.73) is honestly diagnosed as third-answer generations rather than hidden.
Clean no-conflict baseline disambiguates mechanism
Ablation breadth with large per-cell samples
Honest reporting of boundary results
Weaknesses
Frequency proxy validity undermines the headline negative result
Abstract-body inconsistency in the headline effect size
Post-training conclusions drawn from an off-distribution regime
No uncertainty quantification across 31 parallel tests
Mechanistic analysis is thin and largely confirmatory
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Memory vs. Context? Influential Factors of Factual Recall in Language Models
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
It reproduces Yu et al. (2023): when a prompt asserts a fact contradicting what a language model memorized, larger models and more frequent entities tilt the model toward its memorized answer. But the effect is fragile - merely rephrasing the question can swing reliance on memorized knowledge by up to 95 percentage points (measured on Qwen3-14B), and post-training changes the balance inconsistently across model families.
Which language models were evaluated in this study?
31 open-weight checkpoints: the full Pythia suite (160M-12B), GPT-2 (small through XL), Qwen3 dense post-trained models (0.6B-32B) plus matching Qwen3-Base models (0.6B-14B), and Mistral's Ministral-3 at 3B/8B/14B in Base, Instruct, and Reasoning stages. All were evaluated on identical raw text prompts without chat templates.
Does false in-context information override what an LLM memorized?
It depends on the prompt. A coherent passage supporting the false claim shifts all 31 tested models toward the in-context answer (median probability gain +0.19). Semantically unrelated filler text reliably does so only for Pythia models above 1B; several Qwen3 and Ministral models move the opposite way. Simply repeating the false statement twice pushes every Pythia model size to 85-97% in-context answers.
Is the code and data for this paper available?
Yes. The code is at github.com/gfouilhe/IC-factual-public, prompts derive from the public ParaConflict dataset, and all 31 model checkpoints are open-weight on Hugging Face with exact identifiers listed in Table 9 of the paper, so the experiments can be rerun.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-12 could not be inspected; figure-based claims (e.g., Figure 2's 29%→93% shift for Pythia-2.8B under prose filler) rely on in-text descriptions and surrounding tables only.
- The abstract states an 80 pp maximum phrasing effect while Section 4.3/Appendix K report 95.0 pp; the true intended maximum is unresolved from the extracted text.
- Table 9 checkpoints appear as ranges in the extraction (e.g., 'pythia-{160m...12b}'); the complete 31-model list was taken on trust from the count and Appendix N prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.