Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model
Simiao Ren, Kidus Zewde, Xingyu Shen, Xingyu Shen, Yuchen Zhou, Dennis Ng, Ankit Raj, Ankit Raj, et al.
cs.CL
Quick verdict
JevLite typed-decision readout on Qwen3-4B is non-inferior to an LLM judge with ~30x lower latency; candid about test-set exposure, but all data is synthetic.
Summary
This applied paper tests whether a Jev-style typed-decision readout suits per-turn scam-call screening. The authors LoRA-tune Qwen3-4B so that the temperature-scaled softmax over two answer-label logits yields P(scam) in a single forward pass (JevLite), trained with cross-entropy plus a Brier term on the label logits only, plus two teacher-labeled auxiliary intent questions and extra weight on gray scenarios. On 41 held-out CallScreenBench scenarios (82 synthetic calls, 577 turn-end decisions), a three-seed ensemble reaches AUROC .974 with calibration error .052, meets a pre-registered non-inferiority criterion against a MiniMax-M3 LLM judge at a .02 margin, commits no false alarms on legitimate calls (the judge classifies .823 of legitimate decisions correctly), decides 1.14 turns earlier under the same hang-up rule, and runs at 64.5 ms per decision on one RTX 3090 — roughly 30× faster than the judge via API and 4.9× faster than the same backbone fine-tuned to generate JSON. Matched ablations show the accuracy comes from fine-tuning, not the readout interface: a fine-tuned ModernBERT with matched auxiliary heads is not significantly worse and calibrates better. The authors candidly report that about ten recipes were scored on the test set before the selection rule was fixed (winner's-curse risk), that gray-family results rest on 3–5 scenarios with roughly 30 percentage points of seed variance, and that all callers are synthetic instances of known archetypes, so real-call validity remains untested.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A LoRA-tuned Qwen3-4B with the JevLite typed-decision readout reaches AUROC .974 and ECE .052 on 577 per-turn decisions from 41 held-out CallScreenBench scenarios.
- JevLite is pre-registered non-inferior (−.02 AUROC margin) to a MiniMax-M3 LLM judge while committing no false alarms on legitimate calls and deciding 1.14 turns earlier under the same hang-up rule.
- The readout costs 64.5 ms per decision on one RTX 3090 — roughly 30× faster than the judge via API and 4.9× faster than the same backbone fine-tuned to generate JSON.
- The accuracy comes from fine-tuning rather than the typed interface: a fine-tuned ModernBERT encoder with matched auxiliary heads is not significantly worse and calibrates better.
- The headline recipe was selected with test-set exposure, so point estimates carry winner's-curse inflation, and all callers are synthetic instances of known archetypes.
Strengths
Evaluation breadth beyond AUROC
Sections 4–5 report ECE, Brier, false alarms on legitimate calls, per-family accuracy, decision timing and latency with scenario-bootstrap CIs and seed replication — directly fixing the AUROC-only reporting gap the paper documents in prior scam-call work (Section 2).
Matched ablations that isolate the actual contribution
Table 4 trains parsed-label, label-token-readout, typed-decision and linear-head arms on identical backbone/data/schedule: the ordinary fine-tune's parsed label has ECE .107 and its verbalized confidence saturates near .95–1.0, while its label-token probability recovers AUROC .950 / ECE .050 — locating the typed interface's value in output guarantee, calibration and latency, not accuracy.
Unusual candor about weaknesses
Pre-registration and independent verification
Dual-use analysis
Weaknesses
Test-set exposure in recipe selection
Single-family judge/teacher/caller circularity
Synthetic-only, archetype-replay test set
Underpowered gray-family and subgroup claims
Fragile prompt and threshold behavior
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an applied empirical study from Scam.ai evaluating an open, Jev-style typed-decision readout (JevLite) for per-turn scam-call screening, with deliberately modest claims: no architectural novelty; the contribution is the application plus an evaluation reporting calibration, false alarms and timing alongside AUROC. The method (Section 3): LoRA (r16, α=32) on Qwen3-4B, trained with cross-entropy plus a Brier term restricted to two single-token answer-label logits; a validation-fitted temperature turns the label softmax into P(scam) in one forward pass. Two teacher-labeled auxiliary intent questions and a ×2 weight on gray-legitimate examples complete the recipe. Headline results (Table 2): on 577 turn-end decisions from 41 held-out CallScreenBench scenarios (82 synthetic calls), the three-seed ensemble reaches AUROC 97.4% and ECE 5.2%, is pre-registered non-inferior to a MiniMax-M3 LLM judge at a .02 margin, hangs up on no legitimate call, decides 1.14 turns earlier under a replayed rule, at 64.5 ms per decision on one RTX 3090.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is JevLite and how does it differ from asking an LLM to classify a call?
JevLite fine-tunes a small decoder (Qwen3-4B with LoRA adapters) so that after one forward pass over a transcript-plus-question prompt, the softmax over two single-token answer labels (A=yes, B=no), temperature-scaled on validation data, is a calibrated P(scam). No text is generated: the output is always a declared option, costs 64.5 ms per decision on one RTX 3090, and its calibration is part of the training objective (cross-entropy plus Brier on the label logits). An LLM judge instead generates reasoning and a JSON answer, taking about 2 seconds per decision through an API in this paper.
Did JevLite beat the LLM judge?
No superiority claim is made. A pre-registered non-inferiority test at a 0.02 AUROC margin was met (difference +.027, 95% CI [−.011, .065] on 577 per-turn decisions). Relative to the MiniMax-M3 judge, JevLite produced no false alarms on legitimate calls (the judge classified .823 of legitimate decisions correctly), decided 1.14 turns earlier on average under the same hang-up rule, and was about 30× faster. Importantly, a fine-tuned ModernBERT encoder with the same auxiliary labels was not significantly worse than JevLite, so the encoder is the paper's recommended choice when the question is fixed.
What are the main limitations of this paper?
Four stand out. (1) All callers are synthetic instances of known scam archetypes from CallScreenBench; no real calls or audio are tested. (2) The headline recipe was chosen after about ten candidate recipes had already been scored on the test set, so results risk winner's-curse inflation — disclosed by the authors. (3) The judge, the caller that generated the test calls, and the teacher that labeled auxiliary training questions are all MiniMax-family models. (4) Conclusions about ambiguous 'gray' calls rest on only 3–5 test scenarios with large seed-to-seed variance.
Is code or data available for this paper?
The authors state that code, LoRA adapters, per-decision predictions and the 82 generated test calls will be released, together with an independent verification script; CallScreenBench itself is a public benchmark. The appendix specifies prompts, LoRA configuration, seeds and software versions in enough detail for reimplementation. At review time the promised artifacts could not be verified as published.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 are available only as captions and in-text numbers; visual details (calibration bin shapes, latency whiskers, per-turn curves) could not be independently inspected.
- Code, LoRA adapters, per-decision predictions and the 82 test calls are promised 'to be released'; their availability could not be verified at review time.
- The extraction appears complete (full reference list and Appendices A–E present), so truncation risk is low; confidence is set accordingly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.