ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval
Aaryan Kapoor, Md Abdullah Al Hafiz Khan
cs.SE · cs.AI · cs.CL · cs.IR
Quick verdict
Embeddings can't reliably separate correct code from single-edit buggy clones at rank 1; rigorous closed-world Python benchmark, findings credible but narrow.
Summary
The paper introduces ExecRetrieval, a Python code-retrieval benchmark of 939 tasks, each with one execution-verified canonical implementation and up to four execution-verified buggy distractors generated by single mechanical edits (six mutation types), yielding a 4,694-snippet pool with a 7–10-assertion test oracle per task. Construction uses a two-phase LLM pipeline (Claude Sonnet 4.6 for the task registry, GPT-5.4 with high reasoning for implementations; 91% first-attempt validation; five validation gates). The authors evaluate 23 dense embedding configurations plus BM25 under provider-native invocation, reporting exec@k, execution_precision@k, and canonical-ID nDCG with paired exact McNemar tests and query-level bootstrap intervals (5,000 replicates). Key results: the best system (Gemini Embedding 2) reaches exec@10=1.00 but only exec@1=0.331; on the four leading systems, rank-1 misses are paired buggy variants 91.5–99.4% of the time, and the canonical scores below at least one paired distractor on 66.8–78.4% of queries. A pool-density ablation shows one near-clone alone drops exec@1 from 0.993 to 0.678, and deception rates stay flat across mutation types (39.3–48.0%) and edit sizes. Main limitations, acknowledged by the authors: Python-only, mechanical rather than human-written bugs, a closed-world pool making all magnitudes conditional on planted near-clones, function-level tasks, and no reranker evaluation.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- With execution-verified single-edit buggy clones planted in the pool, top dense embedders reach exec@10=1.00 but only exec@1=0.331 (Gemini Embedding 2, 95% CI [0.299, 0.362]).
- When rank-1 retrieval fails on the four leading systems, the retrieved snippet is a paired buggy variant of the query's own canonical 91.5–99.4% of the time (Table 3).
- A single near-clone in the pool suffices to drop the best system's exec@1 from 0.993 to 0.678, so pool density scales but does not create the gap (Table 4).
- Deception is broadly distributed across the six mechanical mutation types (39.3–48.0% of 86,135 triples) and insensitive to edit size or stacked mutations, indicating a property of embedding near-identical code rather than of any specific bug pattern.
Strengths
Controlled counterfactual design with airtight validation
Every rank-1 failure is attributable to a single execution-verified edit; five validation gates plus distractor-sanity and cross-canonical AST sweeps guarantee 939/939 canonicals pass and 3,755/3,755 distractors fail at least one test (Section 3.3).
Statistical hygiene above venue norm
Exact McNemar on discordant pairs, query-level bootstrap (5,000 replicates, fixed seed), and a released full pairwise-test matrix; Table 5 cleanly separates the exec@1 null between Gemini 2 and 001 (p=0.958) from the exec@3 difference (p≈2.7×10⁻⁵).
Density ablation preempts the central confound
Exemplary reproducibility artifacts
Honest conditional framing with robustness checks
Weaknesses
Closed-world scope limits external validity
Synthetic task distribution with unquantified oracle strength
No reranker baseline despite the framing
Single language, function-level granularity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ExecRetrieval (arXiv 2609.01865) is a diagnostic benchmark paper asking one narrow, well-posed question: when the search pool contains, for each natural-language query, one execution-verified canonical implementation plus up to four execution-verified buggy variants differing by a single targeted edit, can dense embedders still rank a passing implementation first? The corpus: 939 Python tasks across ten algorithmic domains, 4,694 snippets, 7–10-assertion test oracles per task. The headline claims are precisely conditioned and internally consistent: (1) exec@10 saturates (Gemini Embedding 2 = 1.000, Table 7) while exec@1 peaks at 0.331 (95% CI [0.299, 0.362]); (2) rank-1 misses are paired buggy variants 91.5–99.4% of the time on the four leading systems (Table 3) — failure is structural, not random lookup error; (3) the canonical scores below at least one paired distractor on 66.8% (Gemini 2) and 78.4% (Qwen3-8B) of queries; (4) one near-clone alone drops the best system's exec@1 from 0.993 to 0.678 (Table 4), so density scales but does not create the gap.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ExecRetrieval?
It is a Python code-retrieval benchmark of 939 tasks. Each natural-language query comes with one execution-verified correct implementation and up to four execution-verified buggy variants differing by a single mechanical edit, all planted in a 4,694-snippet search pool. The test checks whether embedding models rank the correct code above its near-identical buggy twins.
Why do embedding models fail at rank 1 on this benchmark?
Single-token mutations barely change cosine similarity. On the best system, the correct implementation scores below at least one buggy twin on 66.8% of queries, and when rank-1 is wrong it is a paired buggy variant 91.5–99.4% of the time — a structural failure to separate near-identical code, not a random lookup error. Top-10 retrieval still succeeds (exec@10 = 1.00 for the best system).
Is the benchmark realistic for production code search?
Deliberately, no. It is a closed-world diagnostic: every reported number is conditional on a buggy near-clone being present in the pool. This isolates functional discrimination from topical matching, but the frequency of that situation in production code search is unmeasured, so exec@k here is not comparable to open-domain code-search scores.
Can the results be reproduced?
Yes, unusually so. The release includes frozen embedding matrices for all 23 configurations, a per-pair execution-result cache, SHA-256 manifests, verbatim generation prompts, and a single command (eval/reproduce_leaderboard.sh) that recomputes the entire leaderboard offline without any provider API calls.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 were extracted as interleaved label/caption text; all figure-derived numbers cited in this review appear in that label text or in tables (3,4,5,7,8,9), and no end-of-text truncation was observed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.