Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models
cs.CL · cs.CR · cs.LG
Quick verdict
Claims loss-based membership signals in LLMs are faint below ~1,000 verified copies and confounded above; rigorous, but rests on tiny banks and six novels.
Summary
The paper asks whether unusually low loss on a sentence indicates it was in a model's training data, replacing guessed membership labels with verified duplication counts. Using infini-gram indexes over OLMo-2's OLMo-mix-1124 and Pythia's Pile corpora, the author obtains exact per-sentence copy counts and builds three banks from six public-domain novels plus twelve famous lines; non-members are one-word synonym swaps verified to occur zero times. Findings: loss-based AUC on OLMo-2 1B is flat (about 0.57-0.66) across 1-999 copy bands and reaches 0.83 only for famous lines (median about 1,192 copies). A differential design reading the same sentence through both model families finds within-book rank correlation of -0.084 (7B, permutation p=0.016) and -0.065 (13B, p=0.067) over 747 sentences, under 1% of variance, while the large between-book correlation (-0.94 at 7B) is attributed to stylistic taste via an underpowered size-trend test. One-word-edit gaps (about 0.4 nats) are flat across ordinary counts, measuring word choice rather than memory. Swapping same-source edits for composed-prose controls moves AUC from 0.83 to 0.94, quantifying register confounding. Limitations include the twelve-sentence famous bank, six novels only, 8-bit precision at 13B, exact-match counts as floors, and no injected-sequence condition. Banks, counts, and code are released under MIT license.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Loss-based membership signal is flat within noise from 1 to 1,000 copies (band AUCs 0.66/0.59/0.57 on OLMo-2 1B) and only becomes strong at famous-line duplication levels (median about 1,192 copies, AUC 0.83).
- In the same-sentence differential design, exposure leaves only a faint trace where the two corpora disagree about dose: within-book rank correlation -0.084 at 7B (permutation p=0.016) and -0.065 at 13B (p=0.067) on 747 sentences, under 1% of variance.
- One-word-edit non-members inherit an author word-choice gap (about 0.4 nats/token) that is independent of copy count below roughly a thousand copies, so paired tests against such controls detect fluency, not memory.
- Swapping same-source edits for composed-prose controls moves the loss detector from 0.83 to 0.94 AUC on identical members, quantifying how control construction manufactures apparent membership signal.
- Combined, these form a pincer: where exposure is detectable (above ~1,000 copies) the public corpora agree because the text is famous, and where the corpora disagree the trace is too faint to build a test on.
Strengths
Verified dose replaces guessed labels
Sections 2-3 use infini-gram exact-match counts over two independently assembled public corpora (OLMo-mix-1124, the Pile), turning membership from a binary guess into a measured per-sentence copy count, with per-batch presence checks and explicit floor caveats.
Differential design cancels sentence-level confounds
Section 5 reads the identical sentence through both model families; fluency, quality, length, and topic cancel by construction. Book-demeaning plus 5,000-shuffle permutation tests address the between-book taste confound, and the raw pooled split (Pythia 12B r=-0.079, p=0.03) demonstrates the confound is real before centring removes it.
Control construction is measured, not just criticized
Exceptional reporting hygiene
Full release enables cheap replication
Weaknesses
Headline threshold rests on the smallest bank
Taste-vs-memory left unresolved
Narrow text domain
Marginal central effect without multiplicity control
Floor counts may compress the dose axis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper attacks the foundational measurement problem of membership inference on LLMs: nearly all evaluations use guessed member/non-member labels, so detectors can win on date, source, or style rather than membership (building on Duan et al. 2024, Meeus et al. 2024, Das et al. 2024). Because OLMo-2 and Pythia publish their full pretraining corpora (OLMo-mix-1124; the Pile) and infini-gram returns exact occurrence counts for any token sequence, the author turns membership into a measured dose and asks three questions: when does loss carry an exposure trace; can that trace be separated from sentence quality; and how much of reported detector success is manufactured by control construction. Central claims: (1) on OLMo-2 1B, loss-based AUC is flat across copy bands (0.66 for 1-9, 0.59 for 10-99, 0.57 for 100-999; full pass-3 bank 0.60, 95% CI 0.56-0.64); (2) famous lines (median about 1,192 copies) reach 0.83 (CI 0.65-0.98); (3) the differential same-sentence design yields within-book rank correlation -0.084 at 7B (permutation p=0.016) and -0.065 at 13B (p=0.067) on 747 sentences, under one percent of variance; (4) one-word-edit gaps average about 0.4 nats/token and are flat across 1-999 copies, growing with size only above about a thousand copies (0.550/0.787/0.932 at 1B/7B/13B); (5) swapping same-source edits for composed-prose controls moves AUC from 0.83 to 0.94. The 'pincer' synthesizes these: where exposure is detectable the two corpora agree (famous text is famous everywhere), and where they disagree the trace is faint.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper conclude about membership inference attacks on LLMs?
That at ordinary duplication levels (below roughly a thousand copies), loss-based membership evidence is extremely faint (within-book rank correlation about -0.08, under 1% of variance), and above that level the only strongly detectable sentences are famous ones, where exposure cannot be separated from whatever else makes text famous. A strong detector result at ordinary duplication likely reflects something other than membership, such as register or word-choice differences in how the test set was built.
How does the paper know which sentences were in the training data?
It uses two model families, OLMo-2 and Pythia, whose full pretraining corpora are public (OLMo-mix-1124 and the Pile). A public suffix-array index called infini-gram returns the exact number of times any sentence appears in each corpus, replacing guessed membership labels with measured duplication counts, and non-member sentences are verified to appear zero times.
Why can one-word-changed 'non-members' inflate detector scores?
Because the model prefers the original author's word choice: the loss gap between a sentence and its one-word synonym edit (about 0.4 nats per token) is the same whether the original appeared once or a hundred times. So paired tests built on such controls detect fluency and fit, not memorization - the gap only starts growing with copies and model size above roughly a thousand copies.
Is code and data available for this paper?
Yes. The authors release the sentence banks with copy counts from both corpora, the scoring and analysis code, and the figure scripts under an MIT license on GitHub (github.com/IamArmanNikkhah/detectable-only-where-confounded). Labelling a new sentence costs one free query to the infini-gram index.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata abstract is cut mid-sentence, but the complete abstract appears at the start of the provided full text; the body is complete through the references and Appendix A - no truncation detected.
- The paper contains no data tables; all numbers appear inline or in figure captions extracted as text, so figure-internal values could not be visually cross-checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.