Investigating Temporal Motion Features for Pose-to-Text Indian Sign Language Translation

Summary

This WSLP 2026 shared-task paper compares T5-small, T5-base, and T5-large for pose-to-text Indian Sign Language translation and adds a motion-augmented variant that concatenates frame-to-frame pose differences with the 300-dimensional spatial features. A lightweight MLP pose encoder projects per-frame features into the T5 embedding space; models are fine-tuned on 16,980 iSign samples and evaluated on a 1,887-example 10% held-out split, distinct from the official 5,334-example test set. T5-small + Motion achieves the best BLEU (0.298 versus 0.189 for spatial-only T5-small, a ~58% relative gain) and the best ROUGE-2; T5-large attains the highest chrF (15.59) despite training only 3 epochs versus 10 for the others. The submitted system ranked 5th on the official leaderboard. The authors candidly note that the scale comparison is confounded — T5-large got fewer epochs and the intended learning-rate warmup was never applied — that each model was trained once without seed variance or significance testing, and that absolute scores are far below deployment quality. Remaining weaknesses: the motion ablation is tested only on T5-small, there are no external baselines or prior iSign results for context, ROUGE-2 values (0.25–0.77%) are implausibly low, and no official test-set numbers are reported behind the 5th-place rank. Code and models are public.