When LLM Decompilers Recompile More and Preserve Less
Chang Liu, Edward Raff, Kristopher Micinski
cs.CR · cs.AI
Quick verdict
Shows passing shipped tests can mask behavioral divergence in LLM decompilers; rigorous fuzzing-based oracle, evidence broad but selection-biased.
Summary
The paper argues that LLM-based decompilers are judged almost solely by recompilability and passing shipped input/output tests, metrics that can reward outputs which build cleanly but behave differently from the original on unseen inputs. The authors present Decompile-Diverge, a differential-testing oracle: for each function it synthesizes a driver, uses AFL++ to grow an input corpus from the reference, then re-executes decompiled candidates on identical inputs, comparing crashes, hangs, and a relocation-masked digest of the bounded observable post-state. They evaluate eight systems in nine configurations — refinement (LLM4Decompile, SK2Decompile, Idioms, DeGPT-Qwen) and end-to-end (AutoDecompiler, Nova, SLaDe, GLM-5.2) — over four established corpora plus 300 GitHub library functions and 287 CVE-grounded functions. Among 12,133 candidates passing every shipped test, 4.9% diverge on the generated corpus; on GitHub, LLM4Decompile lifts Ghidra's build rate from 75% to 90% while Matched falls from 74% to 62% (McNemar p=.002); on CVEs, 25 of LLM4Decompile's 183 builds (8.7%, 95% CI 6.0-12.5%) silently remove the reference crash. Source analysis ties divergence to introduced fields, types, and callees. Limitations: oracle-compatible function selection excludes pointer- and struct-heavy code, modest fuzz budgets bound coverage (74-99% line), and the invention-divergence link is correlational.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Recompilability and passing shipped tests do not establish behavioral agreement: 4.9% of 12,133 shipped-test passers diverge on the fuzzing-generated corpus, with pass-then-diverge rates up to 13% for a single configuration.
- On real code, build-rate gains trade off against behavioral agreement: LLM4Decompile lifts Ghidra's GitHub build rate from 75% to 90% while Matched falls from 74% to 62% (exact McNemar p=.002).
- Refinement LLMs can silently erase disclosed vulnerabilities: 25 of LLM4Decompile's 183 CVE-track builds (8.7%, 95% CI [6.0%, 12.5%]) exhibit Crash Absence.
- Divergence and Crash Absence are associated with introduced fields, types, callees, and guards replacing visible front-end placeholders (OR 4.6, Fisher p<10^-6 on GitHub), shifting failures from inherited front-end errors to newly introduced ones.
Strengths
Novel test-free behavioral oracle
Section 4.1 combines synthesized per-function drivers, AFL++ corpus growth from the reference, ASan crash/hang detection, and an -O0 comparison of a relocation-masked digest of the bounded observable post-state (return value, written bytes, globals, .data/.bss), with a four-rerun flakiness guard; Matched is honestly framed as an upper bound and Divergence as a lower bound.
Compelling motivating evidence
Figure 1's decode6 case: all three systems pass 10/10 shipped ExeBench tests, yet match only 100/100 (Hex-Rays), 70/100 (SK2Decompile), and 66/100 (LLM4Decompile) stress inputs; Section 3 reports shipped suites miss 3-45% of behavioral divergence.
Unusually broad evaluation surface
Security-relevant new failure mode
Statistical care in headline comparisons
Weaknesses
Selection limits external validity
Backend substitution muddies system comparability
Mechanistic claim is correlational
Near-floor systems yield little signal
Modest fuzzing budgets
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This benchmark-and-measurement paper from Syracuse University and CrowdStrike argues that LLM-based decompilers are judged by the wrong proxies: recompilability (does it build) and re-executability on shipped input/output tests. Three headline claims are supported. First, passing shipped tests does not establish behavioral agreement: across the four established corpora, 4.9% of the 12,133 candidates passing every shipped test still diverge on the fuzzing-generated corpus (77% of them via silent output changes with no crash), and the per-configuration pass-then-diverge rate (Div|P, Table 2) reaches 13% for Nova on HumanEval. Second, on 291 real GitHub library functions, build rate and behavioral agreement rank systems differently: LLM4Decompile lifts Ghidra's build rate from 75% to 90% while Matched falls from 74% to 62%, with exact McNemar p=.002 on the paired counts. Third, on 287 CVE-grounded functions, refinement can silently erase disclosed vulnerabilities: 25 of LLM4Decompile's 183 builds lose the reference crash (8.7% of the track, 95% CI [6.0%, 12.5%]). A fourth, mechanistic claim links divergence to introduced fields, types, callees, and constants replacing visible front-end placeholders (Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Decompile-Diverge?
A behavioral evaluation oracle for decompilers: it synthesizes a test driver for each function, grows a fuzzing input corpus from the reference implementation with AFL++, and re-executes decompiled candidates on the same inputs. It labels each candidate Matched, Diverged, or Crash Absence by comparing crashes, hangs, and a relocation-masked digest of the bounded observable post-state - no hand-written tests needed.
Why is recompilability a misleading metric for LLM decompilers?
Because a function can compile and pass every shipped input/output test yet behave differently on other legitimate inputs. In the paper, 4.9% of 12,133 candidates that passed all shipped tests still diverged on the fuzzing-generated corpus, and the motivating example shows systems passing 10/10 shipped tests while matching only 66-70 of 100 stress inputs.
What is Crash Absence?
A new CVE-track failure label in the paper: a decompiled version of a vulnerable function that no longer crashes on the vulnerability-triggering input, silently hiding the disclosed vulnerability. 25 of LLM4Decompile's 183 CVE-track builds (8.7%) lost the reference crash, while raw Ghidra showed no such cases.
Which decompilation systems were evaluated?
Eight systems in nine configurations: refinement systems LLM4Decompile, SK2Decompile, Idioms, and DeGPT (with a Qwen3.6 backend replacing gpt-3.5-turbo); end-to-end systems AutoDecompiler, Nova, SLaDe, and GLM-5.2; plus raw Ghidra and Hex-Rays front ends as baselines, over four established corpora, 300 GitHub library functions, and 287 CVE functions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extraction ends at the reference list's final entry (Zou et al. 2024, D-Helix) at the true end of the provided text; any references or appendices beyond that point were unavailable.
- Figure 1 and Figure 2 content reached the extraction only as inline prose; visual layout and arrow annotations were not available, though the figures' full textual content was present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.