A Circuit for Plural Reference: How LLMs Represent and Retrieve Singular and Plural Entities
Anh Danh, Rick Nouwen, Massimo Poesio
cs.CL
Quick verdict
Finds a 3-role attention-head circuit for plural pronouns in small LLMs; causal methods solid but template-narrow, attention-only, and code only promised.
Summary
This mechanistic-interpretability study asks how autoregressive LLMs represent and retrieve plural entities when predicting anaphoric pronouns. The authors build two synthetic pronoun-prediction datasets — Dpl, where a plural pronoun is preferred, and Dsg, where replacing the second conjunct with an inanimate entity makes a plural pronoun infelicitous — and first show behaviorally that Qwen3 and GPT-2 models' pronoun choice is significantly modulated by the ontological similarity of the conjuncts and by the conjunction (and vs. with), replicating psycholinguistic findings, while gender is only marginal (p=.081). Using activation and path patching on Qwen3-1.7B (replicated on Qwen3-0.6B and GPT2-medium), they isolate a three-role attention-head circuit: final-layer 'Pronoun Selection' heads L23H12/L23H13 attend to candidate antecedents and determine the pronoun; 'Plurality Formation' head L17H9, fed by K/V heads at layers 15–21, attends to the full construction only when it is a valid plural; and 'Pronoun Interpretation' head L13H6 acts as a coreference map, attending pronouns to their antecedents (~25% probability effect). Attention differences correlate with probability differences (r=.78/.77, p<.001). Key limitations: only two-element conjoined noun phrases in one template plus one variant, MLPs unexamined despite large effects, no variance reported for patching effects, and code only promised.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In Qwen3-1.7B the MHSA module at layer 23 — specifically heads L23H12 and L23H13 — is the only component that directly affects the probability of the plural pronoun (Pronoun Selection Heads).
- Heads such as L17H9, fed by K/V heads at layers 15, 17, 19 and 21, detect valid plural constructions and route plurality information to the Pronoun Selection heads (Plurality Formation Heads).
- Head L13H6 encodes coreference links, attending each pronoun to its antecedents, and accounts for around 25% of the probability difference (Pronoun Interpretation Head).
- The circuit generalizes within the Qwen3 family, to GPT2-medium, and to a second sentence template.
- LLM pronoun preference mirrors human psycholinguistic findings: ontological similarity and the conjunction and/with significantly modulate plural reference, while gender does not (p=.081).
Strengths
Causally grounded, theory-driven interventions
Corrupted prompts (§4) manipulate exactly the two factors the mixed model identified as significant — the second conjunct's ontological status and the conjunction — rather than arbitrary corruptions; path patching then isolates direct effects per Wang et al. (2022), and the K/Q/V decomposition (Figure 5) traces the K/V pathway into the Selection heads' value vector.
Behavioral stage replicates psycholinguistics with proper statistics
Table 2's linear mixed model (verbs as random effects, 27 groups) recovers the ontological-similarity gradient N+N > N+R > N+O (β=−0.199 and −0.855, both p<.001) and the and>with effect (β=−0.669, p<.001), matching Koh & Clifton (2002) and Moxey et al. (2004, 2012) — a genuine model-human alignment result.
Cross-model and cross-template validation
Converging attention evidence beyond patching
Weaknesses
Attention-only 'circuit' with a known missing piece
No uncertainty quantification for any patching result
Internally inconsistent sample sizes
Narrow linguistic coverage, single-token design
Some interpretations outrun the interventions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.03687, Utrecht University / Queen Mary University of London) studies how autoregressive transformers represent plural entities when predicting an anaphoric pronoun, using Qwen3-1.7B (main analysis), Qwen3-0.6B, and GPT2-medium. It makes three central claims: (1) a compact attention-head circuit with three roles underlies plural pronoun prediction — Pronoun Selection heads (L23H12, L23H13 in Qwen3-1.7B) at the final token, a Plurality Formation head (L17H9) fed by K/V heads at layers 15–21, and a Pronoun Interpretation head (L13H6); (2) the circuit generalizes within the Qwen3 family and to GPT2-medium, and across two sentence templates; (3) models' plural-pronoun preferences replicate two psycholinguistic constraints — ontological similarity of conjuncts (Koh & Clifton 2002) and conjunction type, 'and' vs. 'with' (Moxey et al. 2004, 2012).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'plural reference circuit' discovered in LLMs?
It is a small set of attention heads that the paper shows jointly handle plural pronoun prediction in models like Qwen3-1.7B and GPT2-medium: 'Pronoun Selection' heads in the final layers (L23H12/L23H13 in Qwen3-1.7B) look at the candidate antecedents and decide which pronoun to emit; a 'Plurality Formation' head (L17H9) checks whether a conjoined phrase forms a valid plural entity and passes that signal forward; and a 'Pronoun Interpretation' head (L13H6) links each pronoun back to earlier mentions of its referent.
Do language models interpret 'they' in 'John and Mary' the way humans do?
Largely yes on two factors: the paper finds models prefer a plural pronoun when the two conjuncts are ontologically similar (e.g., 'Mary and Emma' versus 'Mary and her bike') and when they are joined by 'and' rather than 'with' — the same preferences documented in human psycholinguistic experiments. Gender match between the conjuncts did not significantly change pronoun choice (p=.081).
Which models were analyzed in this study?
Qwen3-1.7B (the main analysis), Qwen3-0.6B, and GPT2-medium. The same circuit pattern was found in all three, with identical head coordinates across the two Qwen3 sizes (both have 28 layers and 16 heads) and analogously positioned, renumbered heads in GPT2-medium (L18H2/L15H8 for selection, L14H14 for plurality formation).
Is the code and data for this paper available?
The authors state that code and data 'will be made available' at github.com/dangthithaoanh/plural-reference-circuit. The datasets are synthetic and fully specified by prompt templates in the paper (Tables 3–4), so they can be regenerated independently of the release.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Numeric values embedded in figures (heatmap magnitudes in Figures 2, 3a, 5 and the Figure 8 axes) are not recoverable from the PDF text extraction; patching magnitudes could only be assessed from in-text statements such as the '~25%' effect of L13H6.
- Figure axis labels are partially jumbled in the extraction (e.g., Figure 2's y-axis), so figure-level detail was judged from captions and body text rather than plotted values.
- Author name appears as 'Anh Dang' in the paper's title block ('Dang Thi Thao Anh' in references) while the supplied metadata lists 'Anh Danh' — a metadata/text discrepancy, not an extraction loss; the body text runs complete from abstract through Appendix E with no truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.