Patterns of Priming in Production: Lexical, Semantic and Structural Alignment in Language Model Generation
Giulia Pucci, Ruizhe Li, Arabella Sinclair
cs.CL · cs.AI
Quick verdict
Seven open LMs show structural priming in production, amplified by semantic coherence; rigorous design, but the coherence condition conflates lexical overlap.
Summary
The paper asks whether structural priming—observed in human production and in LM comprehension—also shapes LM generation. Using a controlled sentence-completion paradigm on English dative constructions, the authors build PrimeLMDative+ (over one million augmented PO/DO/non-dative sentences from PrimeLM, BLiMP and ROC), prime seven open-weight base models (Phi-2, Gemma-2-2B/9B, EuroLLM-1.7B/9B, Qwen3-1.7B/8B) with single sampled completions, and label outputs PO/DO/non-dative with a fine-tuned BERT classifier validated on 2,700 hand-annotated completions (accuracy 0.86–0.90; Cohen's kappa 0.71–0.92). Against no-context, non-dative, and the harder dative non-match baselines, primed structures rise significantly (e.g., Qwen3-8B PO from 10.4% no-context to 30.8% primed), confirming priming in production (RQ1). Absolute gains are larger for PO, relative gains for DO—an inverse-frequency pattern. Semantically coherent primes (created by reusing the prime's last animate noun in the target) boost matching further, especially for DO (9.9–18.1 percentage points, all p<.001) (RQ2). Primed completions also show higher semantic similarity and lexical overlap with primes across five metrics (RQ3), suggesting coupled structural and lexico-semantic alignment. Limitations: the coherence condition conflates semantic relatedness with lexical overlap; hand-annotated replication (Appendix I) yields much weaker significance, especially for DO; only English datives, no same-stimuli human baseline; single stochastic sample per prompt.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Open-weight LMs exhibit structural priming in production: matching-structure completion rates rise significantly above no-context, non-dative, and opposite-structure dative baselines for all seven models (Tab. 13).
- Absolute priming gains are larger for PO while relative gains are larger for DO, consistent with inverse-frequency effects attenuated by PO's higher base rate (Fig. 3, Tab. 10).
- Semantic coherence between prime and target stem amplifies structural priming, especially for DO (+9.9 to +18.1 points, all p<.001; Tab. 14).
- Structurally primed completions show greater lexico-semantic alignment with the prime across five metrics, indicating coupled multi-level alignment rather than purely syntactic repetition (Tab. 3, Tab. 9).
Strengths
Multi-baseline design with a genuinely hard control
Priming is tested against three increasingly strict baselines — no context, non-dative prime, and dative non-match (opposite structure) — using the unrelated condition to isolate structure from semantics (§5.1). All model×structure×baseline z-tests survive BH-FDR (Tab. 13, e.g. Qwen3-8B PO 10.4% no-context vs 30.8% primed, ∆D=11.2).
Validated automatic annotation at scale
The BERT classifier is benchmarked against a dependency-feature logistic regression (Tab. 2: 0.86–0.90 accuracy on generated text vs 0.52–0.59) and against 2,700 hand-annotated completions (Cohen's kappa 0.71–0.92); a custom sentence splitter reaches 98% vs spaCy's 67% (Tab. 6).
Absolute/relative dissociation reconciles production with comprehension findings
Cross-family robustness with effect sizes
Transparent reporting of divergent evidence
Weaknesses
Coherence condition is confounded with lexical overlap
Headline claims rest on classifier labels whose gold-label replication is weak
DO estimates are the most error-prone yet carry the strongest claims
Decoding is underspecified and variance unquantified
Shallow-copying control is indirect and scope is narrow
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether structural priming — robust in human production (Bock, 1986) and in LM comprehension (Sinclair et al., 2022; Jumelet et al., 2024) — also shapes what LMs actually generate. Three research questions structure the work: (RQ1) presence of priming against three baselines; (RQ2) amplification by prime–target semantic coherence; (RQ3) whether primed completions also align lexico-semantically. All experiments concern English dative constructions (PO vs DO) across seven open base models. The abstract's framing — priming 'operates across multiple levels of linguistic representation' — is mostly matched by associational evidence, but two claim-to-evidence gaps are detailed below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is structural priming in language models?
Structural priming is the tendency to reuse a syntactic structure recently seen in context. This paper shows that after reading a dative prime sentence (e.g., 'gave a toy to a man' or 'gave the man a toy'), open language models become significantly more likely to complete a new sentence with the same dative structure, mirroring a well-documented human phenomenon.
Do LLMs show human-like syntactic priming when generating text, not just when scoring sentences?
Yes, according to this study. Across seven base models (Phi-2, Gemma-2, EuroLLM, Qwen3), the rate of matching-structure completions rose significantly above no-context, non-dative, and opposite-structure baselines in a sentence-completion task, and semantic coherence between prime and target amplified the effect, especially for double-object datives.
What is the inverse frequency effect in structural priming?
It is the finding that priming is relatively stronger for the less preferred structure. Here, rarer double-object datives showed the larger relative increases over baselines, while more common prepositional datives showed larger absolute gains — suggesting inverse-frequency effects operate in LM production but are partly masked by the higher base rate of PO outputs.
Is the code and data for this priming study available?
Yes. The paper states that code and data are available at https://github.com/the-context-lab/primedproduction, covering the PrimeLMDative+ stimulus set, the generation pipeline, the BERT structure classifier, and the annotation materials.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction renders primed/non-primed markers (P vs P-bar) and some subscript symbols ambiguously; interpretations rely on surrounding captions and table logic rather than the symbols themselves.
- Some table cells show minor spacing artifacts from PDF extraction (e.g., '0 0.26' in Tab. 3), but all cited values were recoverable and verifiable in context.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.