From Citations to Contributions: LLM-Assisted Credit Scoring of Research Articles
Sana Ebrahimi, Suraj Shetiya, Abolfazl Asudeh
cs.DL · cs.AI · cs.CL · cs.IR
Quick verdict
Contribution trees split paper credit between original work and citations using LLM estimators; elegant framing, but human validation covers just 50 papers.
Summary
The paper reframes scientific credit assignment: rather than counting citations uniformly, it decomposes each paper's credit into an original-contribution component and a citation-derived component. The core construct is the contribution tree, a document hierarchy whose edge weights partition importance with mass conserved at every level; the authors prove efficiency, additivity, and hierarchical conservation (Theorem 1). Because local importance is unobservable, LLMs act as noisy comparative estimators, queried repeatedly per parent node and averaged after normalization. Citation blocks within paragraphs are scored and their mass allocated to references, and credit propagates through a weighted citation graph in linear time via topological ordering (Theorem 2, Algorithm 1), yielding corpus-level contribution and a normalized influence score that removes self-credit. Evaluation covers 50 papers annotated by human experts (section scores plus top-4 foundational citations), with Claude Sonnet 4.6 and GPT-OSS-120B as additional references. Against humans, qwen3:1.7b is the best LLM (Spearman 0.669), yet the length-weighted frequency baseline is strongest against the API references; at citation level the frequency baselines clearly fail, and qwen3:1.7b agrees best with Sonnet 4.6. A 17-paper case study shows influence rankings diverging from citation counts (Bahdanau NMT 0.475 at in-degree 3 versus Attention 0.122 at in-degree 6). Limitations: small benchmark, annotators included the papers' own authors, citation-level ground truth rests largely on another LLM (only 43% recall of human top-4), and only code—not annotations—is released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Contribution trees decompose a paper's credit into original and citation-derived components while conserving importance across document levels, with provable efficiency, additivity, and hierarchical conservation (Theorem 1).
- LLMs serve as noisy comparative estimators whose averaged, locally normalized scores beat citation-frequency and length-weighted-frequency baselines at citation level, with tiny qwen3:1.7b the best overall (Figs. 4-6, App. H.2).
- Section-level importance is largely recoverable from length alone (length-weighted baseline ρ=0.760 vs Sonnet-4.6), so content-sensitive estimation matters mainly for fine-grained citation attribution (§4.1).
- Contribution-weighted citation graphs propagate credit in O(|P|+|E|) and yield influence rankings diverging from raw citation counts — Bahdanau NMT reaches I_P=0.475 at in-degree 3 while Attention Is All You Need gets 0.122 at in-degree 6 (Table 2).
Strengths
Interpretable, conserving reformulation of citation credit
The contribution tree (§2) forces edge weights to sum to one at every level, so every node score is auditable and the root's unit credit decomposes exactly into original vs citation-derived mass; Theorem 1 (App. C) states efficiency, additivity and hierarchical conservation, a global consistency guarantee absent from prior sentence-level citation-function work.
Efficient corpus extension with correctness proof
Algorithm 1 (App. F) propagates contribution mass through the weighted citation graph in O(|P|+|E|) via topological ordering, avoiding path enumeration; Theorem 2 proves total corpus mass equals |P| under per-paper normalization, and the 17-paper case study (Table 2) illustrates the resulting influence-vs-count divergence.
Unusually thorough reliability analysis of the labels
Practically useful deployment finding
Candid treatment of the length confound
Weaknesses
Central citation-level claims are validated mainly against another LLM
Annotation provenance invites self-assessment bias
Formal model and implementation diverge
Weak, self-designed baselines; no external comparison
Shapley framing does no formal work
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets credit assignment in scientometrics: given a paper, split its total credit between (a) its own original contribution and (b) cited prior work, then propagate those splits across a citation corpus. Three central claims: (1) the **contribution tree** formalizes this with provable conservation properties (Theorem 1, App. C); (2) LLMs — even a 1.7B-parameter qwen3:1.7b — are adequate noisy comparative estimators, outperforming frequency/length heuristics at citation level (Figs. 4-6); (3) contribution-weighted citation graphs yield influence scores diverging from citation counts (Theorem 2, Algorithm 1, 17-paper case study in Table 2). The Shapley/cooperative-game material (App. B) is explicitly abandoned as inoperational and functions purely as motivation — the honest framing is observation-based structural attribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a contribution tree?
A hierarchical representation of a paper (sections, subsections, paragraphs) in which each parent's importance is split among its children so that the weights sum to one at every level. The root (whole paper) has score 1, importance is conserved top-down, and the tree lets the paper's total credit be decomposed into an original-contribution part and a citation-derived part.
How is this different from citation counts or PageRank-style ranking?
Each citation edge carries a content-derived weight — how much of the citing paper's credit the cited work actually earns, judged from the text — rather than a uniform count of 1, and that credit then propagates recursively through the citation graph. In the paper's 17-paper case study, a paper cited 3 times in-corpus received influence 0.475 while the most-cited paper (6 citations) received only 0.122, showing the rankings can differ sharply from raw counts.
Which LLM works best and how large was the evaluation?
qwen3:1.7b — a model under 2 billion parameters — was the strongest of eight local models tested, both for section-level agreement with human annotators (Spearman 0.669) and for citation-level agreement with a Claude Sonnet 4.6 reference. The evaluation used 50 papers annotated by human experts, plus Claude Sonnet 4.6 and GPT-OSS-120B as additional reference annotators, and a separate 17-paper citation-network case study.
Is code or data available for this paper?
The authors state that code is available at github.com/sanaebrahimi/Importance_Scoring, and all prompts and model configurations are documented in the appendices. The 50-paper human annotation dataset is not stated to be publicly released, which limits exact reproduction of the human-reference numbers.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3-9 are present only as captions and axis ranges in the extracted text; in-figure data points are not recoverable, so figure-level quantitative claims in this review rely on numbers quoted in the body text and captions.
- The alpha scaling constant in the length-weighted-frequency baseline (App. G.1, footnote 8) is described only as 'a scaling factor' with no value given — a paper-level omission, not an extraction artifact.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.