What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark
cs.CL · cs.AI · cs.LG
Quick verdict
Audits its LLM instability benchmark for window-missing evidence: flagged items wobble 0.255 vs 0.087; rigorous, honest, pre-registered repair fails.
Summary
The paper introduces Probity, a benchmark of 60 tasks and 470 items built from real venture-financing filings, and measures answer instability: twelve model configurations (eleven distinct models) answer every item 20 times at temperature 0.7. Its core contribution is a provenance audit of the authors' own corpus, verifying that each label's validating quote occurs in the source filing and inside the windowed excerpt shown to the model. The audit flags 36 items, separating evidence genuinely absent from the window (17), answers requiring computation over numbers the window supplies (17), and untraceable labels (2). Flagged items wobble at 0.255 against 0.087 on clean items; excluding them cuts apparent cross-model agreement (mean transfer 0.238 to 0.187, hits ratio 11.0 to 8.9) by about a fifth. A pre-registered repair — re-cutting windows so they contain the evidence for the 10 repairable items — failed its registered threshold (0.208 against 0.161; change 0.058 with an interval containing zero), so the association is reported as correlational only. An observability analysis shows just 1.9% of model-item pairs sit where instability can display at all. Main limitations: a single annotator with no inter-annotator agreement, an underpowered repair (10 items), routing-layer non-determinism unseparated in seven of twelve configurations, and unresolved licensing for redistributed filing texts. The corpus, all 112,800 raw responses, and a standalone verifier are released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Excerpt-built document benchmarks can contain items whose evidence lies outside the model's window; the provenance audit flags 36 of Probity's 470 items (31 window-only, 3 both, 2 source-only).
- Flagged items are far less stable than clean ones (wobble 0.255 [0.216, 0.298] vs 0.087 [0.080, 0.095]), and excluding them cuts apparent cross-model agreement by about a fifth (mean transfer 0.238→0.187, hits ratio 11.0→8.9).
- A pre-registered re-windowing repair failed to bring wobble below the registered threshold of 0.161 (achieved 0.208; change 0.058 [-0.049, 0.165]), so the missing-evidence/instability association is reported as correlational only.
- Only 1.9% of the corpus's model-item measurements fall in the 0.3 ≤ p ≤ 0.7 band where instability can manifest, bounding what an accuracy-built corpus can say about stability.
- Instability is partly shared across frontier models (8.9× chance overlap, p < 0.0005), but the serving-path split suggests part of that sharing may be routing infrastructure rather than models.
Strengths
Pre-registered falsifiable prediction with an honestly reported failure
Section 5 fixes the repair threshold at 0.161 in writing before the runs; the result (0.208 [0.145, 0.289] vs original 0.267, change 0.058 [-0.049, 0.165]) fails it and the causal claim is withdrawn with 'no reframing, re-slicing or threshold-moving'. Table 4 lists every claim the benchmark cannot support, the authors' own prediction first.
Failure-type decomposition rather than a single flag
Section 3 / Figure 2a separates evidence genuinely absent (17 items, wobble 0.338) from computational items whose answer must be derived from in-window numbers (17 items, 0.186) and untraceable labels (2 items, 0.125, difference containing zero) — three populations with different repairability that one flag would have conflated.
Statistical hygiene above benchmark-paper norm
Radical disclosure of the audit's own defects
Observability ceiling bounds interpretation
Weaknesses
Single-annotator oracle with no independent check
The repair null rests on 10 items at the registered floor
Serving-path confound sits exactly where the anomaly is largest
Single temperature and no direct measurement of the audit's counterfactual
Unresolved redistribution licence over the corpus
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Probity is a 60-task, 470-item benchmark over real U.S. venture-financing filings (965 filings, 863 filers, Appendix A) designed to measure answer instability — 'wobble': an item is unstable when a configuration's 20 runs at temperature 0.7 fail to return one identical normalised answer. The paper's real subject, however, is a self-audit: a provenance check that each label's validating quote occurs in the source filing and inside the windowed excerpt the model sees. The claims are: (1) 36 of 470 items are defective by construction; (2) flagged items wobble at 0.255 vs 0.087 (Table 2), and excluding them cuts apparent cross-model transfer from 0.238/11.0× to 0.187/8.9×; (3) a pre-registered re-windowing repair failed its threshold, so evidence-absence is reported as correlational; (4) only 1.9% of measurements sit where instability can show (Section 4, Figure 3a).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Probity benchmark and what does it measure?
Probity is a benchmark of 60 tasks and 470 items built from real U.S. venture-financing filings (SEC EDGAR). It measures 'wobble': whether a language model returns the same answer when asked the identical question about the same document 20 times at temperature 0.7. Twelve model configurations spanning eleven distinct models were tested; items flagged by the provenance audit wobbled at 0.255 versus 0.087 on clean items.
Did re-cutting the text windows to include the missing evidence make the models more stable?
No. In a pre-registered experiment on the 10 repairable items, repaired wobble was 0.208 against the registered threshold of 0.161; the change of 0.058 had a 95% interval of [-0.049, 0.165], containing zero. The paper therefore reports the link between missing evidence and instability as correlational only, and models kept disagreeing even when the answer was placed inside the window.
Is the Probity corpus and code publicly available?
Yes. The paper releases the corpus, the oracle with validating quotes, all 112,800 raw responses, the scoring engine, a standalone verifier that reproduces every reported number while importing nothing from the benchmark package, and a blind re-annotation pack, at github.com/eikiyo/FoFinNLP. Note the redistribution licence for the full SEC filing texts is stated as unresolved.
Why can't this benchmark say much about LLM instability in general?
Because the corpus was built for accuracy, not stability: 86.0% of model-item pairs are answered correctly on all 20 runs and only 1.9% fall in the 0.3 ≤ p ≤ 0.7 band where answers can flip. Among tasks answered correctly at least 90% of the time, the maximum reachable wobble is 0.115, below the 0.30 threshold that would define a correct-but-unstable task.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels are present in the extraction only as axis labels and captions; the rendered graphics themselves could not be visually assessed.
- PDF extraction interleaves Table 1 with related-work body text mid-sentence; all twelve rows and their values remain recoverable, so no content loss is claimed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.