CITECHOICE: A Causal Audit of How Document Presentation Redistributes Citation Credit in Agentic Search
cs.CL
Quick verdict
Hash-verified replay: structured rendering redistributes citation credit (+0.50, Holm p=.033); primary incidence null — rigorous, honest, scope-limited.
Summary
CITECHOICE audits how document presentation causally affects which retrieved sources receive citations in agentic search. From 129 multi-turn transcripts produced by a GPT-5.4 browsing agent over Exa results, the authors select 113 same-call document pairs that independently support the same pre-specified answer fact — chosen without observing ranks or outcomes, with blinded human review confirming 103. A hash-verified 2x2 replay (452 trials, 89 answer-target families) crosses pair order with jointly generated, fidelity-checked prose versus structured renderings of one target. The pre-specified primary outcome, citation incidence, is inconclusive (+4.5 pp, CI [-1.4, +10.4]); a secondary outcome, target citation count, rises +0.50 markers (CI [+0.20, +0.84], Holm p=.033) without expanding the answer's citation budget or reducing the competitor's credit. Observational rank gradients (42.3 pp between ranks 1 and 5) dwarf controlled reordering (+7.9 pp, Holm p=.350), which falls to 0.0 pp in 56 held-out pairs. Repeating 30 frozen families shows 15% of binary citation decisions flip, with decoding accounting for ~45% of single-generation effect variance. A word-preserving list ablation reverses sign on the repeatability subset, ruling out a simple list-marker mechanism. Limitations include a single retriever and model, rewrite-package treatments that mix wording changes, LLM-scored audits, and effects that appear transcript-specific (cross-wave correlation r=-.445).
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Structured rendering causally concentrates citation credit: target citation count rises +0.50 markers per answer (95% CI [+0.20, +0.84]; Holm p=.033) without increasing total citations, unique sources, or reducing competitor credit, while the pre-specified incidence outcome (+4.5 pp, p=.168) remains inconclusive.
- Observational rank gradients vastly overstate controlled position effects: the 42.3-point rank-1 vs rank-5 citation-rate gap contracts to +7.9 pp in the scaled replay (Holm p=.350) and 0.0 pp in 56 held-out pairs.
- Citation evaluation has a quantifiable noise floor: 15% of binary citation decisions flip under identical regeneration, and decoding accounts for ~45% of single-generation family-effect variance.
- Neither list markers nor array position yields a stable one-variable mechanism: the word-preserving ablation reverses sign on the repeatability subset and the rank-by-structure interaction is null (-4.5 pp, p=.418).
Strengths
Outcome-blind, hash-verified causal design
Selection code and reviewers never see ranks, baseline answers, or citations (Section 4.3); the 2x2 replay (452 trials) passes a fail-closed integrity gate with 0 failures, and all eight prompts ship verbatim with SHA-256 hashes (Appendix D).
Honest statistical reporting
The pre-declared primary (incidence, +4.5 pp, p=.168) is reported as inconclusive; the surviving count effect (+0.50, Holm p=.033) is explicitly a corrected secondary; the 8.5 pp minimum detectable effect on 28 discordant families is stated, and post-hoc analyses are labeled as such (Section 5, Table 2).
Quantified evaluation noise floor
Blinded human adjudication with sensitivity analysis
Reproducibility artifact
Weaknesses
Headline rests on a corrected secondary endpoint
Treatment confounds organization with wording
No original-text arm; symmetric compression
LLMs throughout the measurement loop
Thin external validity; weak cross-model replication
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CITECHOICE (arXiv 2609.15164) asks a question adjacent to, but distinct from, citation *correctness*: when two retrieved documents in the same search call both verifiably support the same pre-specified fact, which one does the answer engine credit? The authors call this *citation allocation* and audit it causally. Three claims are advanced: (1) structured rendering *concentrates* credit — the target's citation count rises +0.50 markers (95% CI [+0.20, +0.84]; Holm p=.033) while total citations, competitor credit, and the primary incidence outcome (+4.5 pp, CI [-1.4, +10.4], p=.168) do not move; (2) observational rank gradients (42.3 pp between first-exposure ranks 1 and 5) exceed controlled reordering (+7.9 pp in the scaled 2x2, Holm p=.350; exactly 0.0 pp over 56 held-out pairs); (3) citation evaluation carries a quantifiable noise floor (15% binary flips; ~45% of single-generation family-effect variance is decoding noise). The claims are consistently scoped — the paper repeatedly declines to infer admission effects, a pure formatting mechanism, or a general rank advantage — which is unusual for this subfield.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does making web content more structured get it cited more by AI answer engines?
In this study, structured rendering raised the treated document's citation count by +0.50 citations per answer (95% CI [+0.20, +0.84]; Holm-adjusted p=.033) in frozen replays of GPT-5.4 agentic search sessions. However, whether the document was cited at all rose only +4.5 percentage points and was not statistically conclusive. The evidence supports redistribution of credit among already-eligible sources, not reliable admission into answers, and a pure list-marker manipulation did not produce a stable effect.
How much does ranking position actually matter for citations in AI search answers?
Observationally, documents first shown at rank 1 were cited 85.1% of the time versus 42.8% at rank 5 — a 42.3-point gap. But when the same two documents were swapped inside a frozen transcript, the controlled effect was only +7.9 percentage points in the main experiment and 0.0 points in 56 held-out confirmation pairs. The paper argues that observational rank gradients greatly overstate what reordering alone causes, because rank is confounded with relevance and quality.
Is a single AI-generated answer reliable for measuring citations in evaluations?
No, according to this paper. When 30 frozen answer sessions were regenerated with identical inputs, 15% of binary 'was this source cited' decisions flipped, exact citation counts agreed only 61.7% of the time, and decoding randomness accounted for an estimated 45% of the variance in single-generation effect estimates. The authors recommend repeating generations per condition and reporting between-generation agreement in citation evaluations.
What is 'citation allocation'?
The paper's term for an answer engine's choice of which document receives visible citation credit when multiple retrieved sources from the same search call support the same required fact. Because either source could legitimately be cited, the choice is an allocation decision between verified competitors rather than a correctness decision, and it determines which publishers get visibility in generative search.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 4 were available only as embedded text descriptions; visual interval bars were not inspected.
- The metadata abstract is truncated mid-sentence, but the full body, references, and appendices appear present in the extraction.
- Artifact repo contents and the claimed 150-test suite were not independently verified; reproducibility statements rest on Appendix M.
- Some appendix tables arrived as flattened inline text; minor extraction ambiguity is possible for a few table cells.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.