A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval
Santiago Poveda-Gutiérrez, Hideki Nakayama, Mayumi Bono
cs.CV · cs.CL
Quick verdict
Claims first open-vocabulary JSL sign lookup via captioning+retrieval; contends with I3D on seen classes; credible but single-seed, small-n, output collapse.
Summary
The paper reframes isolated sign language recognition as open-vocabulary retrieval: a 'reverse sign language dictionary' in which a vision-language model (InternVL3-8B) captions a sign-level clip from continuous Japanese Sign Language dialogue into a free-form procedural description, and a multilingual encoder (BGE-M3) retrieves the nearest entry from a 503-description vocabulary — no gloss supervision. On 1,300 segments (5 signers) from a JSL dialogue corpus, with a 2% top-10 chance floor, the authors compare off-the-shelf captioning, LoRA SFT with frozen vs adapted ViT, a closed-set I3D classifier, and an LLM-paraphrase matcher ceiling. Vision-adapted SFT raises TS1 top-10 from 4.5% to 49.0%, statistically tying I3D (52.0%) at top-5/top-10 while beating it at top-1 on every eligible test set (e.g., 29.0 vs 15.5 on TS1). Unseen-class (TS2) retrieval improves from 11.5% to 21.0% top-10 (p=0.0094), a regime closed-set classifiers cannot enter. View generalization is mixed: the pipeline wins TS3.1 but loses top-5/top-10 to I3D on TS3.2 (40.0 vs 75.0) without explanation. Honest limitations: a single training seed per configuration, n=100–200, all five signers shared across train and test, and over 90% of fine-tuned captions collapsing to near-verbatim training descriptions. Statistical reporting is strong (Wilson CIs, paired exact McNemar tests).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 5
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 18 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A captioning-plus-retrieval pipeline enables gloss-free, open-vocabulary sign lookup from continuous signing — the first such system, and the first for JSL, per the authors' knowledge claim (§I).
- Vision-adapted LoRA SFT raises TS1 top-10 retrieval from 4.5% (untrained) to 49.0%, statistically indistinguishable from closed-set I3D at top-5/top-10 on two of three eligible test sets, and ahead at top-1 on all eligible sets (e.g., 29.0 vs 15.5, p=6.6×10⁻⁵).
- Unseen-class retrieval improves significantly over the untrained pipeline (11.5% → 21.0% top-10, p=0.0094) in a regime (TS2) closed-set classifiers cannot enter by construction.
- The matcher is not the bottleneck: LLM-paraphrased gold descriptions are recovered 90–100% of the time (Table I ceiling), locating the residual gap in captioning quality.
- The fine-tuned captioner collapses toward its training vocabulary (>90% near-verbatim captions on TS1 vs 0% before SFT), which the authors attribute to data-scale overfitting rather than vision-tower adaptation.
Strengths
Genuinely novel problem framing
The reverse-dictionary pipeline (§II-A) replaces closed-set gloss classification with captioning plus description retrieval, enabling the TS2 unseen-class split (§II-B) that a closed-set classifier is structurally barred from — a clean conceptual contribution beyond incremental accuracy gains.
Evaluation design isolates failure modes
Four test sets separate unseen segments (TS1, n=200), singleton unseen classes (TS2, n=200), and view generalization (TS3.1/3.2, n=100 each with matched 100-description pools), plus a matched A/B control design (§II-C) where regime pairs differ only in ViT adaptation, and an LLM-paraphrase matcher ceiling that factors captioning error from retrieval error.
Unusually candid limitations section
Statistical hygiene above field norm
Informative negative ablation
Weaknesses
Single training seed per configuration
Unexplained TS3.2 reversal against the baseline
Output collapse strains the open-vocabulary narrative
Context leakage risk in seen-class results
No code, data, or prompt disclosure
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.03788, cs.CV/cs.CL) proposes a 'reverse sign language dictionary': instead of closed-set gloss classification, a sign-level clip from continuous Japanese Sign Language is captioned by InternVL3-8B into a free-form procedural description, which BGE-M3 embeds and cosine-ranks against 503 target descriptions (§II-A). The authors claim (§I) the first description-based, open-vocabulary sign lookup from continuous signing without gloss supervision, and the first for JSL — appropriately hedged ('to our knowledge'). Three concrete claims are supported to varying degrees: vision-adapted LoRA SFT lifts TS1 top-10 from 4.5% to 49.0% and ties closed-set I3D at top-5/top-10 on two of three eligible test sets; unseen-class retrieval improves 11.5%→21.0% top-10 (p=0.0094); and the matcher ceiling (90–100% recovery of paraphrased gold, Table I's Ceiling column) localizes the residual gap in captioning, not retrieval.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a reverse sign language dictionary?
It is a system for looking up a sign you just saw. Instead of classifying the video clip into a fixed list of gloss labels, a vision-language model writes a free-form description of how the sign is articulated (handshape, location, movement), and a text encoder then retrieves the closest entry from a dictionary of such descriptions. New signs can be added by writing a description, with no classifier retraining.
How well does this Japanese Sign Language recognition method perform?
On 1,300 sign segments from a JSL dialogue corpus (503-entry vocabulary, 2% top-10 chance), fine-tuning lifts top-10 retrieval on seen classes from 4.5% to 49.0%. That statistically ties a supervised closed-set I3D classifier at top-5/top-10 on two of three comparable test sets and beats it at top-1 on all three (e.g., 29.0% vs 15.5%).
Does it work for signs never seen during training?
Partially. On test items whose description has zero training examples, top-10 retrieval rises from 11.5% (untrained model) to 21.0% after vision-adapted fine-tuning (p=0.0094), well above the 2% chance floor but far below the ~99.7% matcher ceiling — so captioning quality, not retrieval, is the remaining bottleneck.
Is the code or dataset available for this paper?
The paper text contains no code or data release statement. It uses a subset of an existing annotated Japanese Sign Language dialogue corpus (Bono et al.), which has its own access conditions, and the exact captioning prompts are not published, so replication currently requires corpus access and re-deriving the setup.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 are present only as extracted captions; their visual content (e.g., the Fig. 2 error-bar comparison) could not be inspected, so figure-level claims rely on the caption and §III-A text.
- Table I arrived as flattened run-on text; all 36 numeric cells were legible, but row/column alignment was reconstructed from the header and caption wording rather than visual layout.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.