Dude: A Dual-Detection Multi-Agent System for Paper-Code Discrepancy Detection
Weijie Liu, Running Zhao, Wenhao Yuan, Jinfeng Xu, Zhanfeng Xu, Xiaoxi Zhang, Edith Cheuk-Han Ngai
cs.AI · cs.LG
Quick verdict
Dual-detection multi-agent LLM for paper-code discrepancy detection; F1 gains up to +18.7 pts, but rest on one benchmark and an author-validated LLM judge.
Summary
This method paper introduces Dude, a multi-agent LLM system that detects discrepancies between research papers and their code repositories. The authors argue that single-agent LLM detectors are limited by context capacity and their paper-oriented-only search, missing 'paper omission' discrepancies visible only from code. A naive multi-agent extension recovers recall but destroys precision due to two failure modes the paper names 'over-interpretation' and 'over-reporting', both attributed to granularity asymmetry between papers and code. Dude addresses these with (i) a granularity-aligned negotiation loop in which paper and code negotiators iteratively refine claim descriptions, and (ii) a two-stage salience filter (anchor-guided, then relevance-based) that suppresses trivial code claims. On SciCoQA, Dude reaches 80.43% recall / 93.67% precision (F1 86.55%) with GPT-5.4 and 75.00% / 93.24% (F1 83.13%) with DeepSeek-V4, versus 71.52% and 64.43% F1 for the single-agent baseline; recall gains reach 22.8 points. Ablations attribute the gains to both components, and Dude is the cheapest multi-agent method in tokens (0.76M vs 1.04M for Vanilla-MA on GPT-5.4). Main limitations: evaluation rests on LLM-as-a-judge matching validated by the authors on only 50 cases, no significance testing or run variance, a single public benchmark, and a real-world evaluation whose reference set is built from the compared methods' own outputs.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Single-agent LLM paradigms achieve inferior recall on paper-code discrepancy detection because they perform one-sided, paper-oriented detection and cannot spot paper-omission discrepancies observable only from code.
- Naive multi-agent dual-detection raises recall but sharply lowers precision (Table 1: −17.34 precision points for Vanilla-MA on GPT-5.4) due to over-interpretation and over-reporting rooted in paper–code granularity asymmetry.
- Dude's granularity-aligned negotiation and two-stage salience filtering jointly restore precision while retaining recall gains, improving F1 by up to 18.7 points over single-agent baselines on SciCoQA.
- Dude is the most token-efficient multi-agent method evaluated (0.76M tokens, +7.04% over Single-LM on GPT-5.4), owing to JSON-only agent communication and search memoization.
- Gains generalize across LLM backbones (including GPT-5.5, Table 7) and to 20 recent top-tier papers, where Dude reports the highest precision (93.75%, Table 5).
Strengths
Clear mechanistic diagnosis with matching remedies
The paper names two failure modes of naive dual-detection — over-interpretation and over-reporting — maps each to a concrete component (granularity-aligned negotiation §3.2.3, two-stage salience filtering §3.3), and validates the mapping in ablations (Table 4: dropping filtering costs −19.41 precision points on GPT-5.4; dropping negotiation hurts both recall and precision).
Strong, consistent gains aligned with the core thesis
F1 86.55% (GPT-5.4) and 83.13% (DeepSeek-V4) vs 71.52%/64.43% for Single-LM (Table 2); Fig. 3 shows the largest average gain (+28.5 points) exactly on paper omission, the category single-agent systems theoretically cannot see — evidence that directly supports the dual-detection argument.
Unusually complete reproducibility package
Honest treatment of evaluation circularity and failures
Token efficiency within the multi-agent class
Weaknesses
No statistical rigor for a stochastic pipeline
LLM-as-a-Judge validated only by the authors, on 50 cases
Circular reference set in the 20-paper real-world evaluation
Single public benchmark; baseline tuning asymmetry
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Dude (HKU and Sun Yat-sen University; arXiv 2609.03416v2) is a method paper targeting automated auditing of paper-code consistency. It makes three claims: (1) single-agent LLM detectors underperform on recall because they are inherently paper-oriented and miss 'paper omission' discrepancies visible only from code (§1, citing 46% best recall on SciCoQA from Baumgärtner & Gurevych 2026); (2) naive multi-agent dual-detection trades recall for precision via two named failure modes — over-interpretation (paper-oriented) and over-reporting (code-oriented) — both attributed to granularity asymmetry (Table 1: Vanilla-MA gains +16.30 recall but loses −17.34 precision against a single-agent GPT-5.4); (3) a negotiation loop plus two-stage salience filtering recover precision while keeping the recall gains (F1 up to +18.7 points over single-agent). The 'first dual-detection multi-agent system' framing is softened only by acknowledging BioCon (Xu et al., 2026) as concurrent.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the Dude system do?
Dude is a multi-agent LLM system that audits a research paper against its code repository and reports inconsistencies in three categories: paper-code conflicts, code omissions (claims in the paper with no code), and paper omissions (code behaviors never described in the paper). It runs two detection directions in parallel — paper-to-code and code-to-paper — with a negotiation loop and filtering stages to suppress false positives.
How much better is Dude than a single LLM?
On the SciCoQA benchmark, Dude reaches 80.43% recall and 93.67% precision (F1 86.55%) with GPT-5.4, versus 58.70%/91.53% (F1 71.52%) for a single-agent baseline — about 15 F1 points higher, and up to 18.7 points with DeepSeek-V4. The largest gains are on 'paper omission' discrepancies, which single-agent systems miss because they only search from the paper side.
Is the Dude code available?
Yes. The paper points to a GitHub repository (github.com/VinnyLiu0817/Dude) and an anonymous mirror (anonymous.4open.science/r/Dude), and its appendices reproduce all agent prompts, baseline configurations, and the evaluation judge prompt.
Can Dude replace human reviewers?
The authors explicitly say no. Dude's outputs can be incomplete or wrong — for example, it can misread demonstration notebooks as authoritative code — and the paper positions it as supportive evidence for researchers and reviewers, with final judgments remaining under human oversight.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 3's per-category recall values were extracted as concatenated chart labels; the grouping used in this review was verified arithmetically against the paper's own claimed averages (e.g., the 28.5-point paper-omission gain).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.