Measuring Brand and Source Discovery under Repeated LLM Queries: A Finite-Sample Audit
cs.IR · cs.CL
Quick verdict
一个透明的有限样本审计，显示 LLM 品牌发现指标依赖于提取和定义；虽诚实但仅由 AI 验证，缺乏人类参考。
Summary
本文将基于样本的稀疏化（rarefaction）应用于 LLM 输出的重复查询审计：50 个英文购买问题，六种 API 配置，每个单元格 15 次调用（4,500 个响应），用以衡量较小数量的调用能恢复多大比例的已观测提及品牌和返回来源集合。在历史提取词典下，15 次调用中 10 次调用的已观测组织集合恢复中位数为 92.6%–95.2%；在重新裁定所有 45,683 个存储的候选字符串后，该比例降至 89.5%–94.7%。两个盲测的 Gemini 3.1 Pro 注释角色评估了 600 个完整答案，在规范名称方面的角色间微 F1 为 0.908，在跨度重叠方面的微 F1 为 0.975；这是基于 AI 的证据，没有人类参考研究。一项单独的匹配 2 月/9 月五次调用名册分析（每波 3,750 条记录）给出了 80.0%–92.5% 和 90.0%–100.0% 的单次调用恢复中位数，具体取决于问题子集。将 Sonar 来源主机限制为引用标记所指的条目，使中位数 A(1)/A(15) 从 88.0% 降低至 59.5%。论文异常坦诚：它审计了自己早期的语料库（错误 dated 的数据、错误的 brand-free 声明、分母错误），并明确否认了停止规则、因果检索效应和总体层面的声明。主要局限：整个验证链是 AI 生成的，跨引擎比较因不平等的输出上限和不受控的检索而受到混淆，面板仅包含 50 个确定性的英文问题，且作者存在商业利益冲突。
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Median ten-call recovery of the observed 15-call organization set is 92.6%–95.2% with the historical dictionary and 89.5%–94.7% after full re-adjudication of 45,683 candidates, so reported recovery percentages are extraction-dependent.
- The blinded two-role audit's inter-role micro F1 (0.908 canonical names, 0.975 span overlap) on 600 answers is AI-based evidence; no human reference study exists.
- Matched five-call roster recovery is instrument- and question-subset-dependent (February medians 80.0%–92.5%, September 90.0%–100.0%), so about-80-percent is not a portable per-system description under this instrument.
- Source-host accumulation changes with the source definition: median A(1)/A(15) is 88.0% for all API-returned Sonar hosts but 59.5% for hosts referenced by citation markers.
- None of the results establish exhaustive repertoires, causal retrieval effects or a universal stopping rule.
Strengths
Explicit finite-sample estimands
Section 3 frames rarefaction (Eq. 1) as interpolation within the collected set and uses the Q1/n terminal increment (Eq. 2) to separate sampling-level residual discovery from the chronological last call (singletons in 43–46/50 cells vs only 9–11/50 last-call additions; 32 vs 2 for Sonar). Chao2 is demoted to a diagnostic, with the Sonar paradox (median Sobs/Chao2 of 1.00 despite 64% of cells holding singletons and twelve cells with Q1=1, Q2=0) demonstrating why it cannot certify exhaustion.
Extraction sensitivity built into the design
Section 2.3 re-adjudicates all 45,683 stored candidates (6,867 kept, 38,815 rejected, one unresolved), quantifies the frequency filter's cost (10,351 excluded strings, a documented GoHighLevel recall failure, a parser bug admitting Wisp and ICF), and shows the headline A(10)/A(15) range shift from 92.6%–95.2% to 89.5%–94.7% (Table 2, Mistral moving 0.943→0.895).
Self-audit of prior work
Reproducibility provisions and source-definition sensitivity
Weaknesses
AI-only validation chain
Confounded engine comparisons
Post-hoc specification and ad hoc criteria
Narrow scope and commercial conflict of interest
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement audit, not a market or capability study. The author asks three questions (Section 1): how observed organization sets accumulate across repeated LLM calls; how source accumulation depends on the definition of a source observation; and which apparent completeness conclusions survive audits of extraction, denominators and estimator interpretation. The headline claims are deliberately conditional: median ten-of-fifteen-call recovery of the observed organization set is 92.6%–95.2% under the historical dictionary (Table 1) and 89.5%–94.7% after full re-adjudication of all 45,683 stored candidates (Table 2). The conclusion explicitly disclaims exhaustive repertoires, causal retrieval effects and a universal stopping rule — an accurate statement of what the evidence supports.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the 92.6%–95.2% recovery figure in this paper actually mean?
It is the median, across 300 question-and-engine cells, of the expected share of brands found in a full 15-call reference set that would already appear in 10 randomly chosen calls, computed with exact sample-based rarefaction. It measures recovery of the collected set — not of everything a model could ever mention — and drops to 89.5%–94.7% when the extraction dictionary is rebuilt from all 45,683 candidate strings.
Was any human validation used to check the brand extraction?
No. Extraction, re-adjudication and the 600-answer audit were all performed by Gemini model roles (a 3.7 Flash classifier and 3.1 Pro annotator roles), and the paper states plainly that no independent human annotation study was conducted. Agreeing AI roles may share the same errors.
Are the paper's data and code publicly available?
Partly. The raw corpora are on Zenodo (10.5281/zenodo.20788142 and 10.5281/zenodo.22693819) with byte-for-byte or manifest-hash verification, and historical code is on GitHub (Rankfor/rankfor-open, recommendation-saturation). However, the derived analyses of this specific revision had not yet been re-deposited at the time of writing.
Does this paper prove that ten repeated queries are enough for brand monitoring?
No. The authors explicitly reject a universal stopping rule. Ten-call recovery is high only relative to the finite 15-call reference set; singletons still occur in 43–46 of 50 cells for five configurations, and results depend on the extraction method, the question panel and how sources are defined.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 appear only as captions in the extracted text; the underlying curves cannot be verified from the text alone, but all cited numbers come from tables and prose that are fully present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.