IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English
Ana-Maria Luisa Mocanu, Sebastian Mocanu, Ciprian-Octavian Truică, Elena-Simona Apostol
cs.IR · cs.CL
Quick verdict
Claims 1st at JOKER 2026 Task 1 (0.6347 MAP) with an LLM-judge retrieval pipeline; strong ablations, but test-set tuning and no significance tests.
Summary
This CLEF 2026 JOKER Task 1 (English) system paper presents IROH, a three-stage humor retrieval pipeline that ranked first on the leaderboard with 0.6347 MAP. Stage 1 fuses BM25 (with query expansion) and bge-base-en-v1.5 dense retrieval via reciprocal rank fusion (k1=4000); Stage 2 reranks with a finetuned GTE-Reranker-ModernBERT-base (k2=1000); Stage 3 applies a weighted ensemble of three QLoRA-finetuned LLM judges (Qwen2.5-7B at 0.60, Gemma-4-31B generic 0.30, typed 0.10) fused with Stage 2 at 0.30/0.70, trained on Gemma-4-generated query-aware rationales under generic and typed prompts. Ablations across three cross-encoders, four embedders, and eight judge configurations show the judge is the dominant signal: the generic-rationale Qwen2.5-7B judge alone reaches 0.6055 MAP, beating all Gemma-4-31B configurations, while rationale-expanded BM25 indexing adds almost nothing (0.2876 vs 0.2868 MAP). Structured hard negatives inflate local validation but degrade official MAP in nearly all configurations. Main limitations: no significance testing; ensemble weights and blends tuned through hundreds of official CodaBench submissions, creating test-set selection risk; the calibration hypothesis is asserted without calibration metrics; no zero-shot judge baseline; corpus sizes are never stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- IROH ranks first on the JOKER 2026 Task 1 English leaderboard with 0.6347 MAP, versus a reported best of 0.3501 in 2025 (corpus not strictly comparable across editions).
- The rationale-distilled LLM judge ensemble is the dominant ranking signal, while appending rationales to the BM25 index contributes negligibly (0.2876 vs 0.2868 MAP, judge disabled).
- Structured LLM-generated hard negatives inflate local validation scores but degrade official MAP in nearly all configurations, attributed to distribution shift.
- Across ablated components, lighter models (the Qwen2.5-7B generic judge at 0.6055 MAP; the bge-base embedder) match or beat their larger counterparts within the full pipeline.
- Recall is bounded by Stage 1: rel_ret is flat at 4092 across all ensemble weightings but rises from 3564 to 4092 when k1 widens from 3000 to 4000.
Strengths
Unusually transparent ablation record
Tables 2-7 span 3 cross-encoders, 4 embedders, and 8 judge configurations; null results are reported as such — the confidence penalty had no effect (§5.7), rationale-expanded indexing moved MAP only 0.2876 vs 0.2868 (§5.8), and the top of Table 7 is candidly called 'effectively tied'.
Actionable validation-official divergence finding
Structured hard negatives inflate local validation but degrade official MAP in nearly all configurations (e.g., Qwen typed 0.4933 → 0.2987, Gemma generic 0.5718 → 0.4267, Table 4), a caution directly relevant to shared-task participants tuning on local splits.
Clear SOTA margin with honest context
Strong reproducibility provisions for a system paper
Component-isolated vs pipeline-level selection done right
Weaknesses
No statistical significance testing
Test-set selection across hundreds of configurations
Missing zero-shot judge baseline
Calibration claims asserted, not measured
Config provenance and corpus opacity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a shared-task system paper for JOKER Task 1 English (CLEF 2026): retrieve all jokes, puns, and wordplays relevant to a natural-language query from a corpus mixing humorous and non-humorous texts. Team VANGUARD (POLITEHNICA Bucharest) claims first place on the official leaderboard with 0.6347 MAP — against a reported best of 0.12 in JOKER 2024 and 0.3501 in 2025 (§5.8), with the acknowledged caveat that the corpus expanded across editions. Three research claims are advanced: the rationale-distilled LLM judge is the primary quality driver; structured hard negatives inflate local validation while degrading official scores; and lighter, better-calibrated components match or beat larger ones.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is IROH and what score did it achieve?
IROH is a three-stage humor retrieval system — hybrid BM25 + dense retrieval, cross-encoder reranking, and a weighted ensemble of LoRA-finetuned LLM judges — that ranked first in JOKER 2026 Task 1 English (CLEF 2026) with 0.6347 mean average precision.
How much better is 0.6347 than previous JOKER winners?
The best official MAP in JOKER 2024 Task 1 was about 0.12 and the strongest 2025 English run was 0.3501. The authors caution that the corpus was expanded across editions, so the comparison indicates progress on the task rather than a strictly controlled benchmark.
Did the LLM-generated hard negatives help?
No. Structured hard negatives raised local validation scores but lowered official CodaBench MAP in nearly all configurations — for example the Qwen judge on typed data scored 0.4933 without augmentation versus 0.2987 with it. The authors attribute this to distribution shift between the local split and the official evaluation.
Is the code and data available?
The authors point to a public GitHub repository (DS4AI-UPB/VANGUARD-CLEF2026-JOKER) and publish all prompts (rationale generation, hard-negative construction, judge) in the paper's appendix, with fixed seeds and full hyperparameters. The official test evaluation itself requires organizer-held qrels via CodaBench.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (pipeline diagram) cannot be verified from the extracted text; the pipeline description is reconstructed from Section 4 and the caption.
- Subscript notation in extracted tables (e.g., k1, BM25 k1/b) may contain extraction artifacts, though all MAP values are internally consistent across sections and the abstract.
- 'First place on the leaderboard' is the authors' claim at submission time and was not independently verifiable from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.