The Copy Ceiling: An Input-Exposure Control for Ontology-Grounded Generation over Curated Corpora
cs.CL · cs.CY
Quick verdict
Copy-ceiling control shows graph-RAG uplift is mostly answer delivery; rigorous and unusually candid, but demonstrated on a single corpus and question set.
Summary
The paper argues that 'grounding uplift' reported for retrieval over curated corpora conflates delivering facts already present in the context with reasoning over them. It proposes exposure accounting — classifying each gold item by whether the injected context exposed it and whether the answer recovered it — summarized by the copy ceiling: the recall a verbatim copy of the context achieves. On 510 templated questions over a curated 8,146-class ontology, ten models score 0.26 unaided and ~0.92 grounded against a 0.964 ceiling, so gain over copy is uniformly negative (-0.067 to -0.022); only 3 of 11,360 gold-item observations are credited beyond exposure, and all three fail relational adjudication. A stratified audit of 423 units under a GPT-4.1 judge with a symmetric quote gate estimates matcher credit precision at 0.971 and corrects the exposed-item delivery rate from 0.931 to 0.905. On unexposed gold targets, injection suppresses parametric recall (0.121 bare to 0.004 grounded). Rephrasing questions outside the graph's vocabulary collapses exposure to 0.328 while the absence-triggered fallback fires on 2/506 questions. Production controls show the served path beats no context (+0.27 judged), but content-specificity is not established after Holm correction. Main limitations: single corpus and question generator, LLM-judge dependence without human annotation, underpowered negative controls, and an unreleased scaffold index.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reported grounding uplift over curated corpora is largely delivery of gold names already exposed in the injected context: across ten models, gain over copy is uniformly negative (−0.067 to −0.022) against a 0.964 macro ceiling.
- Beyond-exposure recovery is essentially absent — 3 of 11,360 item×model observations — and all three such credits fail relational adjudication.
- Injecting the scaffold suppresses recall of unexposed but parametrically available gold: 0.121 bare vs 0.004 grounded, with adjudication validating 71 of 92 bare credits and none of the 3 grounded credits.
- Rephrasing questions outside the graph's title vocabulary collapses exposure from 0.964 to 0.328 while the absence-keyed fallback fires on only 2 of 506 questions, so the shipped gate cannot detect the dominant failure mode.
- The served path beats no context (+0.27 pooled judged quality), but no content-specificity contrast survives Holm correction, so the scaffold's specific content is not established as the cause.
Strengths
Deterministic, judge-free control with explicit semantics
Section 5 formally defines the four-count exposure accounting and the copy ceiling (matcher on context vs answer, exact normalised title or ≥80% of long words), distinguishes the scalar from the counts ('the ceiling is the number to quote; the counts are the number to trust'), and §6 shows the negative gain survives an embedding-based re-score in all 30 model×threshold combinations.
The paper audits its own instrument
§10.2 runs a 423-unit stratified judge audit with a symmetric quote gate (281 citations checked; 11 repaired, 18 conceded unresolved), estimates credit precision at 0.971 [0.941, 0.986], corrects the delivery rate 0.931→0.905, and Table 2's lower panel counts matcher failure mechanisms over the full frame without a judge (208 subject-substring credit units; 10 units lost to spaced expansion of run-together titles).
Disciplined reporting of null and negative results
Strong release and reconciliation posture
Operationally useful secondary findings
Weaknesses
Single corpus whose gold derives from the injected graph
All relational-correctness estimates rest on one LLM judge
Negative controls are underpowered and attrition-biased
Recall-only headline with no precision counterpart
Production-side instruments are saturated or unattested
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a specific measurement failure: when evaluation gold derives from the same curated corpus a RAG-style system injects, headline grounding uplift (0.26 unaided → ~0.92 grounded here) conflates delivering facts the context already contains with reasoning over injected structure. The proposed instrument is exposure accounting (§5): each gold item is scored by a deterministic surface matcher on (a) the shown scaffold text and (b) the answer, yielding four counts; the copy ceiling — recall of a verbatim context copy — is the scalar, and signed gain over copy the net. The authors claim narrow ground deliberately, conceding component priority to RAGChecker's context utilisation [47], the reader–context diagnostic [4], and Sufficient Context [24]; the contribution is a cheap, judge-free reporting convention plus documented failure modes. Claims map cleanly to evidence: uniformly negative gain (Table 3, −0.067 to −0.022 across ten models), near-absent beyond-exposure recovery (3/11,360), suppression of parametric recall (§10.4), the paraphrase exposure collapse (§8: 0.964→0.328, fallback firing 2/506), and a production lift (+0.27, Table 5) whose content-specificity controls fail to establish attribution (§10.7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the copy ceiling in RAG evaluation?
It is the recall that a verbatim copy of the retrieved/injected context would achieve against the gold answers — i.e., the fraction of gold items whose names already appear in the shown text. It is a deterministic, judge-free baseline: if a system's grounded recall sits at or below this ceiling, the evaluation cannot credit the system with recovering anything the context did not already expose.
Does this paper show that RAG or graph grounding is useless?
No. It shows that on this curated corpus, the headline recall uplift over a bare model is mostly delivery of facts the context already contains: gain over a copy baseline is uniformly negative across ten models (−0.067 to −0.022). The served path still improves judged answer quality (+0.27) versus no context, and the paper explicitly argues a system can be useful at zero gain because controlled delivery of curated knowledge is the product.
What is the 'suppression' finding in this paper?
On gold targets the retrieved scaffold did not expose, models answered correctly from parametric memory 12.1% of the time when asked bare, but only 0.4% with the scaffold injected (71 of 92 bare credits validated relationally vs 0 of 3 grounded). Injecting partial context is followed by loss of recall the bare model demonstrably has — a cost to price, which the authors note can also be desired behaviour when the corpus is authoritative.
Is the paper's evaluation reproducible?
Mostly. The harness, ten-model sweep rows, exposure decomposition, judge transcripts, and bootstrap scripts are released under version control with a manifest, and the matcher is deterministic. Gaps: the corpus-derived scaffold index (7.9 MB) is not released, so scaffolds cannot be regenerated, and the local Qwen3.8-27B backend is an unattested quantised derivative, so production-study magnitudes are not exactly reproducible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several middle sections were elided in extraction (parts of §9, §10.1, §10.6, §13–§14); statements about them (e.g., the third-family re-judge reproducing the Study 2 contrast in §10.6) are taken from in-text cross-references only.
- Figures 1–3 could not be analyzed; figure-derived numbers are cited from tables and captions in the text.
- Appendix C specifies an unexecuted protocol (two-edge composition); no results exist for it and it was excluded from assessment.
- The metadata abstract and the paper's own abstract differ slightly in ordering; the paper text was used as the source of record.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.