From Retrieval to Weights: Parametric Individualization of Small Language Models with Individual Text Corpora
Christoph Wigbels, Ali Abusaleh, Markus T. Jansen, Alexander Mehler, Markus J. Hofmann
cs.CL · cs.IR
Quick verdict
DoRA adapters write personal search corpora into a 0.6B SLM (dz=1.27 text fit) yet fail person-specific answer alignment; rigorous, honest nulls on answers.
Summary
The paper tests whether an individual's Google search history (their individual text corpus, ITC) can be consolidated into the weights of a small language model, as a cognitive-simulation analogue of episodic-to-semantic memory consolidation. From 515 German participants who answered 36 multiple-choice knowledge items, a stratified subsample of 150 (138 for answer-level analyses) was drawn; per-participant DoRA adapters (r=16) were trained by continued pretraining on each ITC into frozen Qwen3-0.6B, and evaluated in a 2×2 design (adapter × ITC-RAG retrieval) using a bias-corrected PMI readout plus a length-normalized robustness view. At the text level, adapters fit their own participant's held-out text 0.107 nats/token better than other participants' texts (dz=1.27), an individuality effect that grows with corpus size in rank order. At the answer level, log-loss match improves in both views, but match accuracy does not improve under the primary PMI view, and the correctness-split decomposition attributes the gain to generic knowledge rather than to the participant's specific answers or errors. Retrieval adds nothing on top (null H2 interaction), and no effect is carried by ITC size (H3). Key limitations: the base model's correctness (~.33) sits below the participants' lower quartile (.556); only one small base model is tested; corpora are German-only with machine-translated English; and the private search-history data are planned only for NDA-gated GESIS release, with no code release stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Per-participant DoRA adapters write the individual text corpus into a frozen Qwen3-0.6B: each adapter fits its own participant's held-out text 0.107 nats/token better than other participants' texts (dz=1.27).
- At the answer level, adapters improve log-loss match in both scoring views but not match accuracy under the bias-corrected PMI readout; the gain is generic knowledge, not person-specific alignment with the participant's answers or errors.
- ITC-RAG retrieval adds nothing on top of the adapter (null adapter×retrieval interaction in the primary PMI view), so consolidation and retrieval do not complement each other here.
- None of the effects is carried by ITC size: the individuality effect grows with corpus size only in rank order (ρ=.29) and survives size partialing (size-adjusted dz=1.29).
Strengths
Bias-aware readout design validated empirically
Section 3.5 scores all cells over option contents with domain-conditional PMI (neutral 'Antwort:' null prompt) as primary view and length-normalized likelihood as robustness view; a permutation-balanced control bank confirms content-driven readout (control accuracy 0.50 base, 0.70–0.72 adapters vs 0.25 chance) while ID-based readouts sit at chance for adapters — directly addressing the known forced-choice bias literature (Zheng et al. 2024; Holtzman et al. 2021).
Person-specific text-level control absent from most personalization work
The injection check (g_own − mean g_foreign over 20 seeded partners, Appendix B protocol) isolates individual signal from generic domain gain: 0.380 nats/token injection (dz=3.74) vs 0.107 nats/token individuality (dz=1.27, Wilcoxon p<.001), and survives ITC-size partialing (size-adjusted dz=1.29 vs 1.27).
Built-in contamination control
Honest decomposition of its own headline effect
Serious data-protection treatment
Weaknesses
Title overstates the demonstrated construct
Base-model floor far below participants
Single base model and single adapter configuration
ITC quality confounds unmeasured
Reproducibility is structurally blocked
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reframes 'simulating what an individual knows' as **parametric individualization**: each participant's individual text corpus (ITC, built from their exported Google search history) is written into a frozen Qwen3-0.6B via a per-participant DoRA adapter (continued pretraining), and this consolidation route is compared against ITC-RAG retrieval in a 2×2 within-participant design. Three hypotheses are tested: H1 (adapters improve match with participants' answers), H2 (adapter×retrieval interaction: substitution vs complement), H3 (effects persist when ITC size is controlled). The data support a split verdict that the abstract states honestly: individual information demonstrably enters the weights (text level), but the adapters do not reproduce the participant's answer behavior (answer level).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can a language model learn what an individual person knows from their search history?
Partially, per this paper: training a small DoRA adapter per person on their search-history text measurably writes individual information into a 0.6B model's weights — the adapter predicts that person's unseen text better than other people's (dz=1.27). But the model does not better reproduce the person's answers to general knowledge questions once surface biases are corrected; the measured answer gains come from generic knowledge, not from mimicking that individual. True individual memory simulation is therefore not yet achieved.
What is an individual text corpus (ITC) in this research?
A person exports their Google search history via Google Takeout; the researchers crawl the web pages behind the visited URLs, clean out boilerplate and non-German text (English pages are machine-translated into German), remove duplicates, and keep participants with at least 2,500 unique word types. The result is a corpus of running text the person actually engaged with online, used as a proxy for what they know.
Why does the paper use PMI scoring instead of plain answer probabilities?
Adapters trained by raw continued pretraining do not follow instructions, so answers cannot be read from A/B/C/D tokens. Scoring the likelihood of each option's text introduces preferences for generic strings, so the paper subtracts each option's likelihood under a neutral 'Answer:' prompt (pointwise mutual information), and validates with a balanced control item bank that the readout is content-driven rather than surface-driven.
Is the data or code from this study available?
Not yet. The search-history-derived corpora are potentially identifiable; the authors plan a protected release through the GESIS archive under a non-disclosure agreement and a person-identifiability shared task for registered teams. No code release is stated in the paper, and 24 of the 36 knowledge items are deliberately unpublished as a contamination control.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's embedded text and the axis annotations of Figures 2–3 are garbled by PDF extraction, but every numeric value cited in this review was verified against the body text and Table 1; the extraction runs complete through the references and both appendices with no truncated sentences.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.