ATTRICITE: Training an Open 4B Model for Citation Recovery toward Faithful Attribution
Yee Man Choi, Xuehang Guo, Songcheng Cai, Yimu Wang, Yi R. Fung, Qingyun Wang
cs.DL · cs.AI
Quick verdict
Trains an open 4B GRPO agent for citation recovery, lifting Qwen3-4B 49.4→59.8% target-match; rigorous evaluation, but a 31B open model still leads.
Summary
The paper introduces ATTRICITE, an open 4B-parameter model fine-tuned from Qwen3-4B with GRPO for tool-using citation recovery: given a passage whose citation is masked, the agent searches a live Semantic Scholar index within a five-action budget and must select the paper the original author cited. Training and evaluation use CITEALIGN, 7,607 single-citation passages from five CS venues (ACL, CVPR, ICLR, ICML, NeurIPS, 2024-2025), with a 709-instance benchmark subset whose 299-instance test split is temporally held out from 2025 publications and deduplicated against the eligible 2024 pool at source, target, and passage level. Across three runs at temperature 0.7, GRPO raises target-match accuracy from 49.39%±1.54% to 59.75%±0.19%; a paired Bayesian bootstrap gives a 10.37-point gain (95% CI 6.36-14.49). ATTRICITE outperforms gpt-oss-20b (49.72%) and trails GPT-5.4-mini (63.66%) by 3.90 points, while Gemma 4 31B IT leads at 72.02%; on a 143-instance biomedical transfer set accuracy rises from 42.89% to 54.08%. Most of the gain comes from retrieval recall (+9.48 points) rather than selection. Limitations: the benchmark retains only Semantic Scholar-retrievable targets, no matched supervised baseline isolates the RL contribution, checkpoint selection used a noisy 60-instance validation set, and author-chosen citations are only a proxy for faithful attribution.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GRPO fine-tuning improves Qwen3-4B's target-match accuracy on the 299-instance CITEALIGN test set from 49.39%±1.54% to 59.75%±0.19% (10.37 points, paired Bayesian 95% CI 6.36-14.49).
- Despite only 4B parameters, ATTRICITE outperforms gpt-oss-20b (49.72%) and comes within 3.90 points of GPT-5.4-mini (63.66%), while Gemma 4 31B IT leads overall at 72.02%±0.97%.
- Fine-tuning raises retrieval recall from 58.64% to 68.12% (+9.48) and conditional selection accuracy from 84.22% to 87.73% (+3.51), associating most of the gain with surfacing the target paper.
- The learned policy transfers to a 143-instance biomedical set, improving accuracy from 42.89%±1.07% to 54.08%±3.15% under the Medicine field filter.
- CITEALIGN's temporally separated splits (2024 train/val, 2025 test), with every test target absent from the eligible 2024 pool, enable evaluation on previously unseen cited works.
Strengths
Rigorous uncertainty quantification
Three runs per model at temperature 0.7 with SDs and two-sided 95% t-intervals; the headline gain uses a paired Bayesian bootstrap over 299 aligned items (95% CI 6.36-14.49 points, P(improvement)>0.999, P(gain>5)>0.996), and the exploratory study applies paired randomization tests with Holm correction across six hypotheses.
Temporally separated, multi-level deduplicated evaluation design
The 299-instance test split comes exclusively from 2025 publications and every target is absent from the complete eligible 2024 pool, with additional dedup on source-paper titles, target titles, and exact passages (Sections 3.1, A.1).
Unusually deep audit trail
Honest reporting of failure modes and negative results
Efficient, fully released pipeline
Weaknesses
Discoverability-filter circularity
No matched supervised baseline isolating the RL contribution
Comparator framing flatters the contribution
Proxy validity for faithful attribution is weak in places
Checkpoint selection on a tiny, noisy validation set
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ATTRICITE (arXiv 2609.14248) targets citation recovery: given a scientific passage whose citation is masked, an agent must recover the exact paper the original author cited, searching a live Semantic Scholar index within a five-action budget inherited from CiteGuard. The authors treat the published author's citation as an observable human attribution signal, making target recovery a proxy for the source-identification stage of faithful attribution; claim-source verification is explicitly deferred as the complementary next stage. Central claims: (1) GRPO fine-tuning lifts Qwen3-4B from 49.39%±1.54% to 59.75%±0.19% target-match accuracy on the 299-instance CITEALIGN test set (10.37 points; paired Bayesian 95% CI 6.36-14.49; P(improvement)>0.999); (2) the 4B model beats gpt-oss-20b (49.72%±1.84%) and sits 3.90 points under GPT-5.4-mini (63.66%±1.84%), while Gemma 4 31B IT leads at 72.02%±0.97%; (3) the gain is mostly retrieval (recall 58.64%→68.12%) rather than selection (84.22%→87.73%, Table 5); (4) the policy transfers to a 143-instance biomedical set (42.89%→54.08%, Table 4); (5) CITEALIGN (7,607 instances from ACL/CVPR/ICLR/ICML/NeurIPS 2024-2025; 709-instance benchmark subset) supports temporally separated evaluation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is citation recovery and what does ATTRICITE do?
Citation recovery means: given a scientific passage whose citation has been replaced by a [CITATION] marker, find the exact paper the original author cited by searching a live Semantic Scholar index. ATTRICITE is an open 4B-parameter model (fine-tuned from Qwen3-4B with GRPO reinforcement learning) that performs this task as a tool-using agent within a five-action budget, released with the CITEALIGN dataset of 7,607 citation-bearing passages from top CS conferences.
How accurate is ATTRICITE compared with other models?
On the 299-instance CITEALIGN test set (2025 papers, temporally held out from training), ATTRICITE reaches 59.75% target-match accuracy, up from 49.39% for base Qwen3-4B. It beats the larger open gpt-oss-20b (49.72%), trails GPT-5.4-mini (63.66%) by 3.90 points, and is exceeded by Gemma 4 31B IT at 72.02%. On a 143-instance biomedical transfer set it improves over its base model from 42.89% to 54.08%.
Is the model, data, or code available?
Yes. The model checkpoint and collection pipeline are released on GitHub (KathCYM/AttriCite). Dataset metadata and annotations are CC BY 4.0 (passages can be reconstructed locally from the original PDFs via an MIT-licensed script), the code is MIT-licensed, and audit artifacts, prompts, and the statistical-analysis seed are included.
What are the main limitations of this work?
Four stand out: (1) the benchmark only includes papers that Semantic Scholar search can surface in its top-20 for some passage-derived query, so accuracy is measured on a filtered, somewhat easier population using the same backend as evaluation; (2) there is no supervised fine-tuning baseline, so the gain cannot be attributed to reinforcement learning specifically; (3) checkpoint selection relied on a noisy 60-instance validation set; and (4) recovering the author's actual citation is only a proxy for faithful attribution, since roughly a fifth of audited passages cite background rather than directly supporting sources.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-3 are only available as captions plus surrounding text; the per-checkpoint validation curve and error-bar visual detail were not analyzable from the extraction.
- Absolute numbers depend on a live Semantic Scholar index; even with released code, snapshots, and search dates, re-runs will drift.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.