RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases
Yingqian Wu, Jingcong Liang, Siyuan Wang, Zhenfei Yin, Philip Torr, Junchi Yu, Zhongyu Wei
cs.AI · cs.CL
Quick verdict
LLM agents forecasting arXiv field shares all trail an EWMA baseline; solid temporal design, but model-built instruments and unreleased code temper claims.
Summary
The paper introduces RAP (Research Attention Prediction), a rolling benchmark asking whether LLM research agents can forecast how arXiv activity will distribute within a field. At five semi-annual origins (2024-01 to 2026-01), an agent searches a temporally restricted, field-specific corpus (BM25, at most 20 calls) and predicts the next six months' paper shares across eight frozen directions; 278 AI/ML fields yield 1,390 episodes from 356,357 papers, scored by Spearman rank agreement with dependency-block bootstrap intervals. Across seven models, no natural condition beats an exact-count EWMA baseline (0.803 on the All-Origin Test panel, 0.840 Forecast-Strict); the best agent, GPT-5.5, reaches 0.595. Diagnostics show that under cumulative-history access, forecasting redirects search: State-oriented policies draw 100% of retrieved papers from the recent six months versus 47-80% for Forecast, and frozen-evidence replay attributes most of the resulting State advantage to acquisition rather than readout. With exact historical counts, only GPT-5.5 plus reopened Search slightly surpasses EWMA. Fine-tuning Qwen3-4B on 90 realised trajectories lifts pooled Spearman by 0.105 on held-out fields at later origins. Limitations include construction tools shared with evaluated model families, a gold label that is a counting convention over noisy assignments (42.6% boundary ties), single-annotator validation, and data/code not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No evaluated LLM agent under natural evidence access beats an exact-count EWMA baseline on next-six-month direction shares (best: GPT-5.5 fixed-window 0.595 vs 0.803 All-Origin Test).
- Under cumulative-history access, State carry-forward outperforms direct Forecast for all four diagnostic models, and matched-call replay attributes a shared component of this reversal to evidence acquisition rather than readout.
- Forecast-oriented search retrieves a smaller share of recent evidence (47-80% of returned slots) than State-oriented search (100%), and applying recent+relevance options to Forecast's own queries improves their evidence-future alignment by +0.086 to +0.124.
- Even with exact historical counts, calibrated beyond-persistence updating remains limited: only GPT-5.5 with reopened Search surpasses EWMA, and no model reliably tracks six-month revisions.
- Full-trajectory outcome-aligned SFT improves Qwen3-4B's pooled Forecast Spearman by 0.105 on dependency-disjoint Test fields at later origins, demonstrating within-task learnability without a mechanism-level repair.
Strengths
Outcome-grounded rolling design with strict temporal hygiene
Five semi-annual origins, pre-2024-only codebook evidence, cut-off-restricted corpora, documented-cutoff Forecast-Strict masks, and exposure-stratified contrasts (App. E.4-E.5) that preserve every structural sign pattern; Dev/Test split uses frozen dependency blocks with SHA-256-identified artifacts (Table 27).
Novel acquisition-vs-readout decomposition
Matched State twins (A2/A4), byte-identical evidence replay (Table 25), and deterministic re-execution of 86,284 Search calls isolate where the Expanding-history failure lives; Table 21 shows that recent+relevance options alone improve Forecast-query evidence-future alignment by +0.086 to +0.124.
Baselines and reliability taken seriously
Unusual claim discipline
Honest negative headline
Weaknesses
Instrument-model circularity
Soft gold standard
Multiplicity exposure in diagnostics
Reproducibility currently contingent
Construct ceiling of the target
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
RAP reframes "can LLM agents track shifting research attention" as a dense prediction problem: at each of five semi-annual origins (2024-01 through 2026-01), an agent searches a temporally restricted, field-specific arXiv corpus and forecasts the next six months' paper shares over eight frozen directions. The construction spans 356,357 papers (cs.CL/LG/AI/CV, 2022-06 to 2026-06), 278 fields, and 1,390 episodes; the primary estimand is episode-level Spearman agreement between predicted and realised compositions, aggregated field-macro with dependency-block bootstrap intervals. The central claims are negative and diagnostic: (1) no natural agent condition beats an exact-count EWMA baseline (Test All-Origin EWMA 0.803 vs best agent GPT-5.5 fixed-window 0.595; Forecast-Strict 0.840 vs 0.589, Table 2); (2) under cumulative-history access, a matched State-estimation task carried forward beats Forecast for all four diagnostic models, and frozen-evidence replay localises a shared component of this reversal to evidence acquisition - Forecast-oriented policies retrieve 47-80% of slots from the recent six months versus 100% for State; (3) with oracle history only GPT-5.5 plus reopened Search surpasses EWMA (+0.012 [+0.005,+0.018], Table 23); (4) outcome-aligned SFT lifts Qwen3-4B by +0.105 pooled Spearman on dependency-disjoint later-origin Test fields.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the RAP benchmark?
RAP (Research Attention Prediction) is a benchmark that tests whether AI agents can predict how arXiv papers in a given AI/ML field will distribute across eight research directions over the next six months. At each of five half-year cut-offs between 2024 and 2026, an agent may search only literature published before the cut-off, then outputs eight percentage shares that are scored against the realised distribution using rank correlation.
Can LLMs beat simple baselines at forecasting research trends?
In RAP, no. None of the seven tested agents beat an exact-count EWMA (exponentially weighted moving average) baseline: EWMA scores 0.803 Spearman on the full Test panel while the strongest agent, GPT-5.5, reaches 0.595. Even when given exact historical counts, only GPT-5.5 with reopened search slightly exceeded the baseline.
Why do LLM agents forecast research attention worse than they estimate the current state?
The paper's key finding is an evidence-acquisition bias: when asked to predict the future, agents' searches return a much smaller share of recent papers (47-80% of retrieved slots) than when asked to describe the recent past (100%). Replay experiments feeding identical evidence to both tasks show the accuracy gap comes mostly from what was retrieved, not how it was used.
Is the RAP dataset and code publicly available?
Not at submission time. The paper documents a frozen reproducibility archive (with SHA-256 hashes for assignments, splits, prompts, and results) that the authors plan to release upon acceptance; the arXiv-derived corpus itself is public, but the assignments and tooling are not yet distributed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 appear in the extraction only as captions plus in-text annotations; figure-internal values cited here come from the captions and appendix tables (e.g., Tables 20, 24), not rendered graphics.
- A few tables extracted with slightly misaligned column headers (e.g., Table 7), but every value cited in this review is legible in the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.