Retrieved but not ranked: surface-form bias in structural retrieval, from mathematics to agent trajectories
cs.LG · cs.AI · cs.IR
Quick verdict
Two-domain study: embedding retrieval tracks surface over structure; rigorous, honest stats, but small trajectory sample and only three LLM judges.
Summary
The paper tests, under one shared protocol, whether embedding retrieval follows surface wording or underlying structure when the two are deliberately pulled apart, in two unrelated domains. In competition mathematics (MathNet-Retrieve; 500 queries, 117,088-item corpus), strict Hit@1 at the heaviest disguise tier is 0.0% for both production embedders (Gemini-embedding-001, Qwen3-Embedding-8B) although Hit@10 reaches 55.4% and 21.0%; in 95.2-99.8% of misses the winning candidate is more lexically similar than the gold item. On ALFWorld agent trajectories (118 queries, 336 trajectories), retrieval sits at or below exact hypergeometric chance once gold must differ from the query in both object and receptacle. A deliberately naive lexical reranker hurts in mathematics but helps in trajectories, closing 26-36% of the recoverable gap with confidence intervals excluding zero, so its sign diagnoses whether a benchmark's surface variation is adversarial or incidental. Three LLM judges (Gemini 3.1 Flash-Lite, GLM-5.2, Claude Haiku 4.5) all recover gap in both domains, but magnitudes span more than eightfold and the outlier judge differs by domain. Reranking gains concentrate on well-known competitions (+19.8 points in one of six cells), indicating a memorization component. A paired downstream experiment (210 problems, one solver) finds oracle retrieval statistically indistinguishable from deliberately bad retrieval, attributed to truncation-limited headroom. Limitations include few judges, a single trajectory corpus, provenance confounds, and one solver.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Embedding retrieval follows literal surface content over structure whenever the two are separable: strict Hit@1 is 0.0% at the hard disguise tier in mathematics (both embedders) while Hit@10 reaches 55.4%/21.0%, and trajectory retrieval falls below exact hypergeometric chance once gold must differ in object and receptacle.
- A deliberately naive lexical reranker's sign diagnoses a benchmark's surface-variation regime: it hurts in adversarially disguised mathematics and helps in incidentally varied ALFWorld trajectories, closing 26-36% of the recoverable strict gap with CIs excluding zero.
- LLM reranking direction replicates across three independently trained judges in both domains (all 21 judge-by-configuration cells positive), but effect sizes, tier profiles, and the outlier judge are not portable across domains.
- Part of the mathematics reranking gain is memorization: gains concentrate on well-known competitions (+19.8 points, CI [+6.7, +33.2], in one of six judge-by-embedder cells, directionally consistent elsewhere).
- In a paired downstream experiment (210 problems, two graders at 96-99% agreement), oracle retrieval was statistically indistinguishable from adversarially bad retrieval (McNemar p=0.678) because the solver's 69.5% zero-shot accuracy is largely a truncation proxy with 97-100% accuracy on completed answers.
Strengths
Validation gates before measurement
Section 3.7 reproduces MathNet Table 4 easy-tier values within one point (12.2/89.8/97.6% vs published 11.36/90.68/96.93%) and the trajectory benchmark's MEDIUM MAP within 0.007 (0.753 vs 0.746), with a pre-committed halt rule on deviation - rare pipeline hygiene for an eval paper.
Exact chance baselines enable the below-chance claim
Trajectory gold is defined by exhaustively rule-derived task-type labels (eleven anchored regexes, collisions flagged, human-audited on stratified 60-item samples with zero disagreements, Section 3.2), giving exact per-query hypergeometric chance (Table 4) instead of judged relevance pools.
A cheap, mechanistically supported diagnostic
Unusually honest calibration
Practically important side findings
Weaknesses
Trajectory domain is small and single-family
Judge-dependence claim rests on n=3 judges
Downstream null is one truncation-limited solver
Unmeasured same-family and provenance confounds
Contamination evidence is directional
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether embedding retrieval follows surface wording or underlying structure when the two are deliberately decoupled, under one shared protocol across two unrelated domains: competition mathematics (MathNet-Retrieve, 500 queries against the full 117,088-item corpus) and ALFWorld agent trajectories (118 queries, 336-item corpus). Five headline claims: (1) strict Hit@1 collapses to 0.0% under the heaviest disguise tier in mathematics while Hit@10 stays high (55.4% Gemini-emb, 21.0% Qwen-emb, Table 1); (2) trajectory retrieval falls to or below exact hypergeometric chance once gold must exclude the query's literal object and receptacle tokens (Table 4); (3) a naive lexical reranker's sign diagnoses the benchmark's surface-variation regime - it hurts in mathematics (-9.1/-4.8% gap share at easy tier) and helps in trajectories (+25.9/+36.4/+32.1%, CIs excluding zero); (4) LLM reranking direction replicates across three judges (all 21 judge-by-configuration cells positive) but magnitudes are non-portable; (5) downstream, oracle retrieval is statistically indistinguishable from adversarial retrieval (McNemar p=0.678) because solver headroom is truncation-bound. A Section 10 'what we do not claim' list and contamination discounting keep the claims calibrated.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'retrieved but not ranked' mean in this paper?
The embedding model usually places the correct item somewhere in the top 10 (e.g., 55.4% Hit@10 for Gemini-embedding at the hard disguise tier) but almost never first (0.0% strict Hit@1), because lexically similar decoys planted by the benchmark outrank the structurally correct answer.
What is the lexical reranker sign-flip diagnostic?
A deliberately naive reranker (word overlap, edit distance, length ratio) re-sorts the top-10 candidates. If it hurts retrieval, the benchmark's surface variation is adversarial (as in the math benchmark); if it helps, the variation is incidental (as in ALFWorld trajectories). It is free to compute and runs before any expensive evaluation.
Do LLM rerankers fix structural retrieval?
Only partially. Three LLM judges recover 5-63% of the recoverable ranking gap in mathematics and 43-76% in agent trajectories, all in the positive direction, but magnitudes vary more than eightfold across judges, tier profiles disagree, and the outlier judge differs by domain, so no single effect size transfers.
Does better retrieval improve the solver's final accuracy?
Not in this study: in a paired 210-problem experiment, giving the solver the correct problem-plus-solution was statistically indistinguishable from giving it a deliberately wrong similar problem (McNemar p=0.678). The reason is that the solver's 69.5% zero-shot accuracy was largely a truncation artifact - it answered 97-100% correctly whenever output finished within budget, leaving retrieval almost no headroom.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 extracted as loose text rather than clean plots, but every headline number they encode also appears in Tables 1-5 and the running text, so no quantitative information was lost.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.