Building a Production Greek-English Speech Recognizer
Christos Petrocheilos, Cleopatra Papadopoulou, Chris Porikis, Ioakeim Perros, Ayoub Kirouane, Themistoklis Nikolis
eess.AS · cs.AI · cs.SD
Quick verdict
Industrial report on Sophea, a Greek-English production ASR: honest engineering, but the 6× frontier and 9/9 headline claims outrun single-run evidence.
Summary
This industrial report documents the engineering program behind Sophea, Kiefer's production Greek-English ASR service, evaluated against nine production gates: Greek and English WER ceilings, a 95% language-identification floor, and zero hallucination on non-speech audio. Across twenty-three training iterations over two architectures, no data composition passed all nine gates at once: the Greek noisy-environment gate needs roughly 1,500 steps of dense exposure, while English language identification tolerates only about 250 (about 1,250 rebalanced), a roughly 6× gap the authors frame as a capacity frontier at this scale. The pipeline work includes calibrating the UTMOS quality filter against in-domain anchors, cutting discarded Greek audio from 98.7% to 10.6%; a pre-registered ablation isolating a hallucination-causing data package; and a three-model ROVER ensemble raising gate coverage from 4–7 of 9 to a conditional 9/9 and cutting overlapping-speech WER from 53.35% to 37.87%. A separately-routed learned arbiter, sophea/asr-k1, posts 4.26% average WER on the public Open ASR Leaderboard and 25.88% on live Greek noisy-environment traffic (95% CI 24.44–27.19). The paper is notably candid, with negative results, instrument-failure post-mortems, and externally checkable leaderboard numbers. Limitations: the frontier claim rests on single-run evidence, core Greek benchmarks are proprietary, no weights or data are released, and one ensemble member has confirmed AMI pretraining exposure.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- At the tested 1.7B scale and training configuration, no data composition satisfies all nine production gates: Greek noisy-environment WER needs ≈1,500 dense-exposure steps while English LID tolerates ≈250 (≈1,250 rebalanced, at ~0.75 Greek WER cost).
- Calibrating generic quality scorers against in-domain anchors is decisive: a UTMOS threshold of 3.0 would discard 98.7% of scored Greek audio versus 10.6% at the calibrated 1.30.
- A three-model ROVER ensemble raises gate coverage from 4–7/9 to a conditional 9/9 and cuts overlap-only WER from 53.35% to 37.87% (bootstrap CI 11.0–19.5 absolute points, 18 AMI meetings).
- The learned per-clip arbiter K1 posts 4.26% average WER on the Open ASR Leaderboard's eight public English sets and 25.88% (CI 24.44–27.19) on live Greek noisy traffic, the first single served model under that 26% gate.
- Decode-time fixes outperformed retraining for three distinct defects (silence hallucination, code-switching, boilerplate), grounding a try-decode-time-first policy.
Strengths
Measurement-integrity culture with pre-registration
Section 4.4 runs a pre-registered multi-arm ablation that convicts one data package (boilerplate 24 → 20 vs 17–18 baseline) before any fix; Section 7 documents five instrument failures (32× loss-logging bug, curriculum-ordering loss jump, ITN-sensitive benchmark 4.46→8.42, false contamination matches, harness race condition), and the K1 re-decode nondeterminism (14.15 vs 14.21, 99.0% clip agreement) is disclosed rather than smoothed over.
Anchor-calibration principle is concrete and transferable
Section 4.2 / Figure 2: UTMOS anchor medians (Greek meetings 1.83, clean Greek read 2.72, community English 3.35) justify dropping the textbook 3.0 threshold to 1.30, cutting discarded Greek audio from 98.7% to 10.6%; the same principle is formalized for CTC forced alignment (τ = 1.5 × P95 of anchors).
External checkability where possible
Honest negative-results ledger and non-deployment rationale
Formalized trade-off with a useful control
Weaknesses
Headline impossibility rests on single-run evidence
Decisive gates are unverifiable
Abstract framing outruns the body's own caveats
No artifacts released
Gate-battery opacity and count-based metrics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an industrial systems report on Sophea, KIEFER SA's production Greek-English ASR service, organized around nine production gates: four Greek WER ceilings, three English WER ceilings, a 95% language-identification floor, and zero hallucination on a 1,500-clip silence/noise battery (Section 3, Appendix A). The central claims are: (1) a measured impossibility result — across 23 training iterations on a 1.7B bilingual model and a Whisper-based line, no data composition passes all nine gates, because the Greek noisy-environment gate (WER ≤ 25.25) needs ≈1,500 steps of dense exposure while English LID tolerates ≤≈250 steps (≤≈1,250 rebalanced, at a ~0.75-point Greek cost); (2) an anchor-calibration principle for automated scorers, most striking in the UTMOS fix (98.7% → 10.6% discard); (3) ensemble gains — three-model ROVER at conditional 9/9 gates and −29% relative overlap WER; and (4) K1's externally listed 4.26% leaderboard WER and 25.88% live Greek noisy WER. The paper explicitly scopes claim (1) to its scale and configuration (Sections 3, 11).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Sophea and who built it?
Sophea is a commercial Greek-English automatic speech recognition service built by KIEFER SA in Athens. The paper reports the engineering program behind it — 23 training iterations, a six-stage data pipeline, ensembling, and nine production gates — but releases no model weights, code, or training data.
What WER does Sophea ASR K1 achieve?
K1 is listed on the public Open ASR Leaderboard as sophea/asr-k1 (preview) with 4.26% average WER over the board's eight public English test sets as of 11 September 2026, and it scored 25.88% WER (95% CI 24.44–27.19, n=1,325 clips) on live Greek noisy-environment traffic through its production API.
Why couldn't one model handle both Greek and English?
In their controlled experiments, pushing Greek noisy-environment WER under the 25.25 gate required about 1,500 steps of dense noisy-Greek exposure, but English language identification fell below its 95% floor after about 250 such steps (about 1,250 with a rebalanced mix that stalled Greek accuracy short of its gate). The intervals never overlap, so the team shipped a routing layer with specialized models instead of one bilingual model.
Why does UTMOS-based audio filtering need calibration for Greek?
UTMOS was trained on English TTS-style judgments and systematically under-scores real Greek speech: clean Greek read speech scored a median 2.72 and real Greek meetings 1.83 under it. The conventional cutoff of 3.0 would have discarded 98.7% of scored Greek audio (including the FLEURS benchmark itself); a threshold of 1.30, calibrated on known-clean in-domain anchors, discards only 10.6%.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction appears complete through Appendix C (page 21); no truncated sentences detected. All numbers cited are from the text, tables, and table captions; figure-internal curve data (e.g., Figure 4 bars) are visible only as axis labels, which suffices for every claim made here.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.