CodeTD: Topology of Attention Detects Hallucinations in Code LLMs
Daria Voronkova, Ilya Trofimov, Anton Dmitriev, Eduard Tulchinskii, Evgeny Burnaev, Serguei Barannikov
cs.SE · cs.AI · cs.CL · cs.LG · math.AT
Quick verdict
CodeTD claims pre-execution detection of incorrect code via TDA on attention maps; broad solid experiments, but gains over spectral baselines are modest and mixed.
Summary
CodeTD (arXiv 2609.07779) detects functional incorrectness of LLM-generated code before execution using topological data analysis of attention maps. For every layer/head it builds a weighted graph over prompt+generation tokens and computes asymmetric Manifold Topology Divergence summaries (MTD0/MTD1, both directions) plus diagonal-attention averages, then trains an XGBoost classifier. Across 10 Code LLMs (1.5B–34B), four benchmarks, and five languages, it outperforms spectral attention baselines (LapEigvals, AttnLogDet, AttnEigvals) by a few ROC-AUC points; only BCB carries significance testing (Fisher p=0.017). Margins often lie within fold-level standard deviations, and diagonal attention alone nearly matches the full method on several models (StarCoder2-7B HumanEval: 83.4 vs 82.5), questioning the topological margin. Benchmark-transfer claims hold with sizable drops and occasional reversals. Strengths include the prompt-masking causal experiment, best-of-N ranking gains, label-efficiency analysis, and a direct TOHA comparison. Main limitations: hallucination is equated with any functional-test failure, and significance is tested on one benchmark only.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Attention-map topology — specifically asymmetric MTD0/MTD1 divergence between prompt and generation token sets — predicts whether generated code is functionally incorrect, without executing it (Sec. 6).
- CodeTD outperforms spectral attention baselines (LapEigvals, AttnLogDet, AttnEigvals), prompting baselines, and CodeJudge across 10 models and 4 benchmarks, with global significance on BCB (Fisher p=0.017, Appendix T).
- The classifier transfers across benchmarks and across Code LLMs (feature-extracting model can differ from the generating model, Tables 3/24).
- Individual (layer, head) features are consistent hallucination predictors across five programming languages (Sec. 7.2, Table 22).
- Using CodeTD to rank N samples improves pass@1 over random selection for all tested models (Table 4).
Strengths
Unusually broad empirical coverage
10 Code LLMs (1.5B–34B), 4 benchmarks (HumanEval, MBPP, BigCodeBench, MultiPL-E), 5 languages (Table 2/27), with group-stratified 5-fold CV that keeps all generations of a problem in the same fold (Sec. 7.1), preventing prompt-level leakage — a pitfall many similar papers miss.
The core hypothesis is validated by an intervention, not just correlation
Appendix V (Table 11): masking 50% of prompt tokens drops CodeLlama-7B pass@1 from 25.9% to 15.0% and DeepSeek-Coder-6.7B from 34.8% to 19.5%, while syntactic correctness barely moves (92.0→90.3, 95.9→90.9) — direct causal support for the prompt-connectivity hypothesis.
Practically relevant evaluations beyond raw AUC
Baselines and ablations are comparatively complete
Weaknesses
Margins over the strongest baseline are small and not always consistent
The ablation partly undercuts the headline contribution
Statistical testing covers one benchmark only
Hallucination = any failed functional test
Transfer claim and compute cost are softer than advertised
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CodeTD (arXiv 2609.07779) proposes pre-execution prediction of functional incorrectness in LLM-generated code via topological data analysis of attention maps. The pipeline: for every layer and head, build a fully-connected weighted graph over prompt+generation tokens with weights w=1−a (Sec. 6), compute Cross-Barcodes and MTD0/MTD1 in both (P,G) and (G,P) directions normalized by set cardinality, add mean diagonal attention per set, and train XGBoost. The authors define hallucination as any failure of benchmark functional tests (Sec. 3, with logic errors ≈78% of failures per Appendix O/Table 26), a deliberate broadening they state explicitly. Claims: (1) CodeTD beats recent baselines; (2) transferability across benchmarks and models; (3) some (layer, head) features are cross-language predictors. Code is released (GitHub link in Sec. 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CodeTD and how does it detect code hallucinations?
CodeTD is a method that predicts whether code generated by a code LLM is functionally incorrect without running it. It reads the model's attention maps, treats prompt and generated tokens as two groups in a weighted graph, and computes a topological quantity (Manifold Topology Divergence, MTD) that measures how disconnected the generation is from the prompt. These features feed an XGBoost classifier trained on execution-based labels. Inference needs no test execution.
How well does CodeTD perform?
Across four benchmarks and 10 models (1.5B–34B), ROC-AUC is typically 69–88%, a few points above the strongest spectral attention baseline (LapEigvals), though it is outperformed in isolated settings (e.g., StarCoder2-7B on BigCodeBench: 83.7 vs 85.3). For best-of-N selection it lifts pass@1 over random choice, e.g., StarCoder2-7B on HumanEval from 28.6 to 43.3.
Does CodeTD need the code to be executed or tests to be available?
Not at inference — that is its main selling point. Training the classifier does require execution-derived correct/incorrect labels for generated samples on the target benchmark, but after training the classifier transfers to other benchmarks and other code LLMs.
What does 'hallucination' mean in this paper?
Any generated program that fails the benchmark's functional tests — including syntax errors, runtime exceptions, and logic errors (logic errors are ~78% of cases). This is broader than the everyday sense of confabulation and is explicitly defined in Section 3.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are not interpretable from text extraction (image content), so visual evidence (e.g., Fig. 3 feature distributions, Fig. 4 sparsity curve) was assessed only via the surrounding text and tables.
- Mathematical notation in the method section is OCR-garbled in places (e.g., the attention formula); the MTD definitions in the body and Appendix Q were cross-checked and are consistent.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.