DeceptionAnalyser: A Web-Based AI Tool for Performing Structured Deception Analysis with Argumentation Schemes and LLMs
Stefan Sarkadi, Xabier Garmendia, Jack Mumford, Trevor Bench-Capon
cs.HC · cs.AI
Quick verdict
A 10-scheme deception argumentation library and LLM scrutiny tool; transparent, but validated only for consistency on two short narratives, not accuracy.
Summary
The paper makes two contributions: a library of ten argumentation schemes for deception analysis—each with premises encoding observable indicators and falsification-oriented critical questions—and DeceptionAnalyser, a Chrome extension implementing a two-stage pipeline in which an LLM first assesses each scheme premise against a PDF (green/yellow/red verdicts anchored to verbatim excerpts) and then answers the critical questions with argument–counterargument pairs, with human feedback loops at every stage. The authors explicitly forgo accuracy benchmarking, arguing no scheme-level ground truth exists and the tool flags claims for scrutiny rather than issuing deception verdicts. Evaluation is a reliability study: the Fox and Crow fable (11 models × 10 runs) yields unanimous green on every element with agreement 1.00, while a synthetic Operation Mincemeat report (10 models × 10 runs) shows 89–98% green with divergence concentrated in the Major premise and two models (GPT-5.4, gpt-oss-20b), which the authors interpret as warranted caution about a source-unstated inference. All 600 premise assessments were grounded in verbatim excerpts. Strengths include candid failure-mode reporting (F1–F3), verbatim prompts in the appendix, and released code. Key limitations: only two short documents underpin the quantitative claims; consistency is measured but validity never tested; agreement statistics lack confidence intervals or formal measures; and no evaluation with real analysts despite the human-AI teaming motivation.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- This is the first dedicated library of argumentation schemes designed for deception analysis, organised by locus of manipulation (Section 3, Table 1).
- A two-stage pipeline — LLM premise extraction with verbatim-excerpt grounding, then critical-question evaluation — supports transparent, explainable, analyst-controlled deception scrutiny (Section 4).
- Scheme detection is highly stable across ten contemporary LLMs and ten repeated runs: unanimous on a clear-cut fable (agreement 1.00) and 89-98% green with interpretable, localised divergence on an ambiguous intelligence-style report (Section 5.6, Tables 2-3).
- All 600 premise assessments were grounded in verbatim document excerpts, enforced by string matching rather than model-reported coordinates (Sections 4.4, 5.6).
- Because the schemes flag claims for scrutiny rather than output verdicts and no scheme-level ground truth exists, accuracy benchmarking is out of scope and consistency is the appropriate evaluation target (Sections 1, 5, 6).
Strengths
Careful epistemic framing of the scheme library
Section 3's two clarifications keep observable indicators (premises) separate from the deception hypothesis, relocating intent judgements to the critical-question stage; Section 3.11 and Table 1 organise the ten schemes by locus of manipulation (Influence as genus; nine species), directly answering the overlap objection.
Candid failure-mode reporting
Section 5.5 explicitly surfaces F1 (source-ungrounded inference: the tool asserts the British planted documents, which the synthetic report never states), F2 (ungrounded synthesised conclusion, Evidence: NA), and F3 (role inversion in CQ8) instead of hiding them, and links F1 to the exact element where models diverge in Table 3.
Enforced evidential grounding
Above-average reproducibility provisions
Weaknesses
Validity never assessed; corpus of two documents
Statistical reporting is thin
Human-AI teaming motivation is never empirically exercised
Prompt and selection sensitivity unexplored
Scope and priority claims slightly outrun the artifact
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper claims two contributions: (I) the first dedicated library of argumentation schemes for deception analysis — ten presumptive, defeasible schemes (Section 3), each with major/minor premises encoding only observable indicators, a scrutiny-not-verdict conclusion, and 5-8 falsification-oriented critical questions; and (II) DeceptionAnalyser (Section 4), a Chrome extension over PDFs implementing a two-stage LLM pipeline: per-premise assessment (green/yellow/red anchored to verbatim excerpts), a synthesised conclusion, then per-critical-question argument-counterargument evaluation, with human feedback loops at both stages. The authors explicitly disclaim accuracy benchmarking (no scheme-level ground truth exists; the tool flags claims for scrutiny, Sections 1, 5, 6) and instead offer a reliability study across ten contemporary LLMs x ten repeated runs on two narratives.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DeceptionAnalyser and what does it do?
It is a Chrome browser extension that applies ten argumentation schemes for deception to PDF documents. A large language model first checks each scheme premise against the text (green/yellow/red, with verbatim quotes as evidence), then answers each scheme's critical questions with argument-counterargument pairs. It flags claims for human scrutiny; it does not output a deception verdict.
Does DeceptionAnalyser actually detect deception accurately?
The paper does not measure accuracy. The authors argue no scheme-level ground-truth corpus exists and the tool is designed as a scrutiny aid. They instead report consistency: the same analyses repeated across ten LLMs and ten runs, which was unanimous for a clear-cut fable and 89-98% stable for an ambiguous intelligence-style report.
Which LLMs were tested in the reliability study?
GPT-4o, GPT-5.4, gpt-oss-20b, Claude Sonnet 4.5, Claude Haiku 4.5, Gemini 3.5 Flash, Gemini 3.1 Pro, DeepSeek v4-pro, DeepSeek v4-flash, and Qwen3.6-27B, plus an additional Gemma model for the fable only — ten runs per model per document, with a 9-run (Gemini 3.1 Pro) and 1-run (Gemma) asymmetry on the fable noted.
Is the code, prompts, or example data available?
Yes. The paper reproduces all three prompt templates in its appendix and points to a public GitHub repository (onekin/DeceptionAnalyser) containing prompts and example reports/texts; the extension itself is distributed via the Chrome Web Store.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 (pipeline diagram and UI screenshots) could not be analysed; interface descriptions rely on captions and body text only.
- The fable's per-model breakdown is not tabulated (unlike Table 3 for Mincemeat); the 11th (Gemma) model and the 9-run Gemini 3.1 Pro asymmetry are described in prose only.
- External artifacts (GitHub repo, Chrome Web Store listing) were not inspected; availability claims are taken as stated in the paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.