SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detection
Hanling Wang, Chenlong Wei, Yingjuan Li, Di Wu, Yuchao Zhang, Xiaohui Zhu, Yao Zhu
cs.CL
Quick verdict
Retrieval + LLM symbolic plan-solve reasoning for zero-shot harmful memes; beats MinD/Evolver but lacks significance tests; hyperparameters may be test-tuned.
Summary
The paper presents SyRHM, a zero-shot framework for harmful meme detection that combines fine-grained associative retrieval with symbolic-language-enhanced multi-stage reasoning. A vision-language parser converts each meme into semantic elements plus a description; weighted CLIP embeddings (image, elements, description) retrieve the three most semantically related memes from an unlabeled reference set, whose textual representations serve as reasoning premises. A Translator (Qwen2.5-14B-Instruct) converts premises and query into FOL-style forms, a Planner produces stepwise reasoning plans, and a Solver (Qwen2.5-VL-7B) executes the plan with the original image. On FHM, MultiOff, HarM-c, and HarM-p, SyRHM reports 66.10 accuracy (32.83 MCC) on FHM and 69.80 on MultiOff, surpassing reasoning baselines MinD and Evolver on every reported metric and all baselines on most settings, with the largest gains on fine-grained tasks (harm-level Macro-F1 51.90 on HarM-c, +14.04 over Evolver; 42.88 on HarM-p, +10.29). Ablations attribute most gains to the Planner (-3.67 accuracy when removed) and Translator (-2.54), with the Solver contributing little (-0.43). Key weaknesses: no significance tests or variance reporting despite small test sets (MultiOff n=149); an unstated selection protocol for retrieval hyperparameters (k, modality weights) that may have used test accuracy; and no formal verification of the LLM-generated logic. Code is released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Element-level associative retrieval over parsed semantic elements and descriptions provides more relevant evidence than global image-text similarity retrieval.
- Multi-stage symbolic-language-enhanced reasoning (Translator-Planner-Solver) outperforms single-pass VLMs and prior reasoning frameworks MinD and Evolver on most zero-shot evaluation settings across FHM, MultiOff, HarM-c, and HarM-p.
- Most of the performance gain comes from structured abstraction and planning (Planner removal: -3.67 ACC; Translator: -2.54), not from the multimodal Solver step (-0.43).
- The framework produces interpretable, traceable judgments without using any labels for model updates or decision supervision.
Strengths
Auditable multi-stage pipeline
Each stage emits inspectable artifacts — parsed elements/descriptions (Fig 3), FOL-style translations (Fig 4), stepwise plans (Fig 11), and execution traces (Fig 12); qualitative cases (Figs 5, 13-14) show the reasoning trace reversing a baseline's wrong prediction on the 'i'm down' Down-syndrome pun.
Unusually complete ablation coverage
Retrieval size k (Fig 6, peak 70.90 at k=3), modality weights (Table 3, S6 71.75 best), stage removal (Fig 7), LLM scaling (Table 8), retrieval encoders (Table 9), cross-dataset retrieval (Table 10), and per-stage token/time costs (Table 11, ~61.5 s/sample).
Fair baseline protocol
Hedged, honest claims
Weaknesses
No statistical rigor
Hyperparameter selection protocol unstated
'FOL' claims lack formal grounding
Internal numeric inconsistencies
Modest absolute performance and narrow baseline scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SyRHM targets zero-shot harmful meme detection under a retrieval-based protocol: the benchmark train split serves only as an unlabeled reference set, and no labels touch model updates, prompt tuning, or decision supervision (Sec 3.1). The paper claims three contributions: element-level associative retrieval, multi-stage symbolic-language reasoning, and superiority over reasoning baselines (MinD, Evolver) on 'most evaluation settings' across FHM, MultiOff, HarM-c, and HarM-p — note the abstract names three dataset families while the contributions say 'four benchmarks'. The hedged phrasing is largely accurate: SyRHM beats MinD/Evolver on every reported metric, and the sole all-baseline loss (MultiOff BACC 65.58 vs LLaVA-v1.6-Vicuna-13B's 65.94) is conceded in Sec 4.3.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SyRHM?
SyRHM is a zero-shot framework for detecting harmful internet memes. It parses each meme into semantic elements and a description, retrieves the most semantically similar memes from an unlabeled reference pool, and then runs a three-stage symbolic reasoning pipeline — Translator, Planner, Solver — built on Qwen2.5 models to decide whether the meme is harmful, producing an inspectable reasoning trace.
How well does SyRHM perform?
The paper reports 66.10% accuracy (32.83 MCC) on the FHM benchmark and 69.80% accuracy on MultiOff, beating the prior reasoning-based methods MinD and Evolver on every reported metric. The largest gains are on the fine-grained HarM tasks, e.g., 51.90 Macro-F1 for harmfulness-level prediction on HarM-c.
Is SyRHM's code available?
Yes. The paper states code is released at github.com/Scabbards1500/SyRHM, all four datasets (FHM, MultiOff, HarM-c, HarM-p) are public, and prompts and hyperparameters are documented in the appendices.
Is SyRHM really zero-shot?
In the sense that no labels are used to train or tune the detector. However, its retrieval pool is the benchmark's own training split, and the paper does not state how retrieval hyperparameters (top-k, modality weights) were selected, so reviewers should scrutinize possible test-set coupling.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Mathematical symbols in the FOL-style formulas (e.g., implication operators in Figures 4, 11, 12) appear to have been dropped or garbled by PDF extraction; the formulas as extracted look malformed, and the originals may be correct.
- Figures 6 and 7 are raster charts; their values were available only via the numbers quoted in the surrounding text (e.g., 70.90 at k=3; stage-removal accuracies 68.36/67.23/70.47), not read from the figures themselves.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.