Measuring GEO Visibility: Prompt Corpora Define the Answer Market
cs.IR · cs.CL
Quick verdict
一份严谨、坦诚的调查报告，认为 GEO 分数由 prompt 语料库和权重定义；没有新数据——主张基于示例和重新计算。
Summary
这份批判性调查报告认为，GEO（生成式引擎优化）可见性分数是由其测量工具定义的，而不仅仅是抽样：prompt 语料库、其权重和评分工具共同构成了一个“答案市场”，而这个市场未必代表实际的用户需求。在构建效度（Jacobs & Wallach）、总调查误差、IR 测试集研究和部分识别（Tamer）的基础上，作者形式化了涵盖情况标注、prompt 策略、执行条件和评分的协议元组（protocol tuple），其中包括一个 LLM 裁判，其指令甚至可以在答案保持不变的情况下改变分数。文中提出了四点贡献：五种 prompt 功能的分类法；通过权重类别上的线性规划计算出的可采纳分数集；一个“留一出”对比，区分了条件性归因与包含竞争干扰的全引擎干预；以及一个重新计算已发表聚合数据的示例。利用 Grossman 等人的 AI Overviews 激活率（总体为 65.6%；ORCAS 为 51.5%，Amazon Retail 为 17.4%，ELI5 为 94.6%），两种加权方案得出了 39.7% 对 70.5% 的面板分数；一个虚构的双系统示例展示了权重阈值 0.60 处的排名反转。该论文异常透明：没有声称有新实验，列出了部分阅读的来源（表 9），并声明了 AI 辅助。其主要局限在于该框架的实际价值尚未在真实使用数据中进行测试，且核心数据来源于刻意挑选的极端层。
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Prompt corpora, their weights, and the scoring apparatus constitute an active measurement instrument that helps define what a GEO visibility score measures, rather than merely sampling a fixed quantity.
- Reweighting three published Grossman et al. activation subcorpora (51.5%, 17.4%, 94.6%) yields panel scores of 39.7% vs 70.5% with no change within any subcorpus, showing the answer market is a construct distinct from user demand.
- Reweighting can reverse a system ranking only conditionally — when performance profiles cross between strata, at threshold λ=0.60 in the fictional two-system example.
- A citation's presence does not establish a source's contribution; the paper's leave-one-source-out contrast under a fixed documentary context (Eq 6) is distinct from full-engine interventions with competitor interference (Eqs 7–9).
- When weights are unsettled, reporting admissible score sets separates partial identification (values compatible with data and population assumptions) from normative sensitivity (variation across weighting conventions) and from sampling uncertainty.
Strengths
Exceptional provenance discipline
Table 9 lists exactly how partially-read sources were used (Messick 1994: record+abstract only; Manski 2003: introduction pp. 1–2; Zobel 1998: introduction p. 307); the AI Assistance Statement declares Codex (OpenAI) writing support; Section 2 states the survey cutoff (31 Aug 2026) and that inclusion probabilities are unknown.
Self-application of the framework
Appendix A applies the paper's own corpus card (Tables 8–9) to its 44-reference selection, making its selection bias inspectable rather than hidden — the epistemic move most survey papers omit.
Genuinely useful formal distinctions
Transparent, checkable demonstrator
Correct causal framing of attribution
Weaknesses
Zero new empirical content
Headline numbers are constructed contrasts
Elementary formalism
Illustration layer rests on a preprint cluster
Foundational authority partly borrowed unread
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a critical literature survey and methodological framework paper (44 references, cutoff 31 August 2026) on interpreting GEO (generative engine optimization) visibility scores. Central claim: the prompt corpus, its weights, and the scoring apparatus constitute an *active measurement instrument* whose choices define — not merely sample — the quantity a GEO score reports, an "answer market" that need not match real user demand. Four contributions are offered: (1) a taxonomy of five prompt functions (Table 1); (2) admissible score sets over weight classes computed as linear programs (Sec 4.4); (3) a leave-one-source-out contrast (Eq 6) separated from full-engine interventions with competitor interference (Eqs 7–9, after Hudgens & Halloran 2008); (4) an arithmetic demonstrator on published aggregates (Sec 7.5). The paper explicitly claims no new experiments — the version note, abstract, Sec 9.2, and Conclusion all say so.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is an 'answer market' in GEO measurement?
The weighted set of prompts or situations submitted to a generative engine over which source appearances, citations, or brand mentions are counted and aggregated into a visibility score. The paper argues this constructed market is defined by the evaluator's choices and need not represent actual user demand.
Does this paper present new experiments?
No. The authors state explicitly that no new experimental data were collected. The quantitative content is a fictional two-system ranking example and recalculations of published AI Overviews activation rates from Grossman et al. (SIGIR 2026).
What do the 39.7% and 70.5% numbers mean?
Two weighting schemes applied to three published AI Overviews activation subcorpora — ORCAS 51.5%, Amazon Retail 17.4%, ELI5 94.6% — produce panel-level scores of 39.7% vs 70.5% while the rate within each subcorpus stays unchanged, demonstrating that panel weights alone can dominate the reported number.
Is the proposed GEO measurement framework validated?
Not empirically. The paper says its comparative usefulness 'remains to be assessed' and proposes validation steps: comparing corpus construction against observed usage populations, testing the constraints chosen for admissible weight classes, and measuring annotation and LLM-judge fidelity.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: marked gaps ([...sampling gap...]) cover parts of Sections 3.1, 5.2–6.1, the end of 7.5, 9.2–9.3, and part of the reference list; assessments of those regions rest on framing visible elsewhere.
- Table 7 (Sec 8.3, datasheet-extension mapping) is referenced but its contents are not present in the provided text (Tables 5, 6, 8, 9 are present); its mapping was not assessed.
- The specific weight vectors behind the 39.7%/70.5% headline scores fall inside a sampling gap ('weights (0.2,' is cut off); only the resulting scores and the three strata are cited here.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.