DiscoSign: Discourse-Aware Text to Sign Language Gloss Translation
Vasileios Baltatzis, Mert Inan, Connor Gillis, Raja Kushalnagar, Lorna Quandt, Leah Findlater, Colin Lea
cs.CL
Quick verdict
First discourse-aware text-to-ASL-gloss framework with novel metrics; big SCA/CGC gains but metrics partly self-enforced and judged by the same LLM.
Summary
DiscoSign is presented as the first systematic framework for discourse-aware English-to-ASL gloss translation. It targets three linguistically grounded phenomena: spatial coreference (entities keep consistent spatial indices like IX-3p:i), Question-Answer Clauses (rhetorical pseudocleft structures), and concept-gloss consistency. The pipeline translates sentence-by-sentence with one LLM call per sentence (Gemini 2.5 Pro; also Qwen3.6-35B-A3B), carries three state registries (spatial assignments, concept-gloss mappings, QAC decisions) as hard prompt constraints, and applies deterministic string-replacement post-processing that corrects violations. The authors introduce matching metrics — Spatial Coreference Accuracy (SCA), QAC Appropriateness (QACAp), and Concept-Gloss Consistency (CGC) — arguing chrF/COMET cannot capture discourse coherence. On Aesop's Fables (no gold ASL references), the framework lifts SCA from 0.29 to 0.84 and CGC from 0.59 to 0.97 with Gemini, with Qwen preserving the system ordering, while traditional metrics stay flat (chrF ~40). On sentence-level benchmarks (ASL STEM Wiki, a licensed 2,121-pair set), back-translation improves, but gloss-level COMET slightly drops on ASL STEM Wiki (0.54→0.51). Human evaluation (2 raters, 32 stories) validates SCA (ρ=0.52) and CGC (ρ=0.48) but not QACAp (ρ=-0.09). Main limitations: no gold discourse references, evaluation loops through the same LLM, QACAp unvalidated, and headline consistency is partly enforced by the system's own corrector.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- DiscoSign is the first systematic framework for discourse-level text-to-ASL-gloss translation, using explicit state registries and deterministic constraint enforcement.
- Discourse-aware processing substantially improves spatial coreference consistency (SCA 0.29→0.84) and concept-gloss consistency (CGC 0.59→0.97) with Gemini 2.5 Pro on Aesop's Fables, and the ordering replicates under Qwen3.6-35B-A3B.
- Traditional MT metrics (chrF/COMET) are insensitive to discourse coherence, motivating the proposed SCA, QACAp, and CGC metrics.
- The framework maintains competitive single-sentence gloss translation quality, with back-translation improving on both sentence-level benchmarks.
- The proposed metrics are validated by human evaluation for SCA (ρ=0.52) and CGC (ρ=0.48), while QACAp's appropriateness judgments remain rater-subjective.
Strengths
Linguistically grounded formalization
Section 3.1 grounds each phenomenon in sign-language linguistics (Wilbur 1994 on foregrounding; Caponigro and Davidson 2011 on QACs as pseudoclefts; Winston 1991 on spatial referencing) and formalizes module state with explicit constraints (Eqs. 1-3) — rare rigor for gloss MT work.
Metrics validated against humans, with honest negatives
Section 5.5 and Appendix B report a 2-rater, 32-story study: SCA correlates at story level (combined ρ=0.52, p=0.003; weighted κ=0.43) and CGC at ρ=0.48 (p=0.007); the non-significant QACAp result (combined ρ=-0.09, p=0.659) is reported transparently rather than buried.
Deterministic enforcement instead of pure instruction-following
Backbone generalization
Full prompt transparency
Weaknesses
Metric-evaluator circularity at the discourse level
Self-enforced headline metrics
Gloss-level regressions downplayed
QACM is the weakest module and its metric is unvalidated
No external baselines and validation opacity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This Apple/Northeastern/Gallaudet paper introduces DiscoSign (arXiv 2609.02796), presented as the first systematic framework for discourse-aware English-to-ASL gloss translation. It targets three linguistically grounded phenomena (Section 3.1): spatial coreference (entities keep stable spatial indices such as IX-3p:i), Question-Answer Clauses (pseudocleft topicalization per Wilbur 1994 and Caponigro/Davidson 2011), and concept-gloss consistency. Four contributions are claimed: the framework, an ASL-centric instruction design methodology, a three-metric evaluation suite, and ablations. The core empirical claims check out against Table 3 — large gains on the authors' own discourse metrics (SCA 0.29→0.84, CGC 0.59→0.97 with Gemini 2.5 Pro on Aesop's Fables) — but those gains are entangled with the system's own enforcement machinery and an LLM-centric evaluation loop, detailed below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DiscoSign and what problem does it solve?
DiscoSign is a framework for translating English text into ASL glosses (written labels for individual signs). Existing systems translate sentence-by-sentence, losing discourse-level consistency: entities get conflicting spatial pointing indices, the same concept is rendered by different signs, and rhetorical question-answer structures (QACs) are misplaced. DiscoSign carries three state registries across sentences as hard LLM prompt constraints and deterministically corrects violations afterward.
What results does DiscoSign achieve?
On Aesop's Fables with Gemini 2.5 Pro, it raises Spatial Coreference Accuracy from 0.29 (sentence-level baseline) to 0.84 and Concept-Gloss Consistency from 0.59 to 0.97, while standard MT metrics stay flat (chrF ~40), which the authors argue shows traditional metrics miss discourse quality. On sentence-level benchmarks, back-translation scores improve, though direct gloss matching shows small drops (e.g., COMET 0.54→0.51 on ASL STEM Wiki). The ordering of systems replicates with the open-weight Qwen3.6-35B-A3B.
What are Question-Answer Clauses (QACs) in ASL?
QACs are pseudocleft-like structures in ASL discourse where a rhetorical question is immediately answered, e.g., rendering I want coffee as I WANT WHAT? COFFEE. They serve emphasis, causal explanation, and topicalization functions, but are inappropriate at discourse beginnings or when answering genuine questions. DiscoSign includes a module that decides when to restructure sentences into QAC form.
Is DiscoSign's code or data available?
The paper announces no code or data release. It fully documents its prompts, JSON output schema, and post-processing rules (Appendix A). Two of its three datasets are public (ASL STEM Wiki; Aesop's Fables via Project Gutenberg), but the 2,121-pair licensed ASL dataset is not, and reproducing results requires access to Gemini 2.5 Pro (or Qwen3.6-35B-A3B for the open-weight comparison).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 3's exact per-point values are not recoverable from the text extraction; context-window claims rely on the in-text numbers (SCA 0.40→0.84, CGC 0.78→0.97, QACAp plateau at 0.76 with two prior sentences).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.