Measuring Language Transfer in Robot Policies: Adding Greek to a Cosmos3 Vision-Language-Action Policy
Ayoub Kirouane, Georgios Giaples, Christos Petrocheilos
cs.RO · cs.AI
Quick verdict
Claims Greek VLA localization via machine-translated data with null-based measurement; unusually honest but single-language, machine-only, n=3-seed evidence.
Summary
The paper asks whether machine-rephrased Greek instructions can localize a vision-language-action robot policy without architecture changes, and concludes the real bottleneck is measurement, not translation. On the open Cosmos3 stack, an English-centric 2B text tower cannot acquire Greek through caption fine-tuning at any dose; swapping in an 8B multilingual tower yields coherent Greek-conditioned world-model generation (64% coherent, 0% good content match). In closed-loop LIBERO evaluation, a single-goal suite produces a false positive (84.6% Greek versus 82.6% under wrong instructions), while a ten-goal suite separates conditions: the bilingual policy reaches 48.6% Greek (61.3% three-seed mean) against a 4.4% wrong-instruction floor. On ninety tasks, the bilingual margin over the floor is 6.7-7.1 points across three seeds, whereas Greek-only training clears it by at most 2.7. Switching to an independently produced translator costs a mean 23.3 points; training on seven phrasings per task roughly halves that penalty. Warm-starting from a Greek-adapted world model (one run per arm) and unfreezing the text tower both degrade performance. Main limitations: a single language, a single model family, simulation only, all Greek machine-generated including the 'independent' set, and cross-policy contrasts bounded by a p=0.100 design floor with no multiplicity correction.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Single-goal robot benchmarks cannot attribute instruction-following; a wrong-instruction control and an independently constructed null (a policy proven unable to follow the language) are required, as shown by 84.6% Greek versus 82.6% wrong-instruction success on LIBERO-10.
- Target-language demonstrations are necessary but not sufficient: without Greek data a multilingual tower transfers nothing to the action pathway, and with Greek data alone the ninety-task margin over the wrong-instruction floor never exceeds 2.7 points versus 6.7-7.1 for bilingual training.
- Bilingual training on machine-translated Greek reaches 48.6% Greek success on ten goals (single run; 61.3% three-seed mean) and 27.4% on ninety tasks, with no architecture change.
- Policies overfit the training translator's phrasing - an independent translator costs a mean 23.3 points, about 40% of the headline level - and training on seven phrasings per task roughly halves this penalty.
- Warm-starting from a language-adapted world model (one run per arm) and unfreezing the text tower (three seeds) both degrade performance in both languages.
Strengths
Guaranteed-null measurement design
Three-way protocol (correct English / correct Greek / wrong instructions) plus an identically trained policy that provably cannot follow the language as a per-goal null (Section 5); the LIBERO-10 illusion (84.6% Greek vs 82.6% wrong) is exposed by a control costing one extra evaluation run (Table 1, Figure 5).
Statistical candor
Paired task-level bootstrap with quantified anti-conservatism at ten clusters (89-91% empirical coverage), exact sign-flip permutation corroboration, and Appendix A enumerating all 20 comparisons with a computed ~41% chance that at least one of five perfect-separation contrasts is spurious.
Seed-replication discipline
Matched perturbation controls
Retractions reported as evidence
Weaknesses
Narrow external validity
Machine-only Greek with judge/translator overlap
Cross-policy evidence at the design floor
Key intervention at n=1
Partial artifact release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an empirical measurement study rather than a systems paper. It asks whether machine-rephrased Greek instructions can localize a Cosmos3 vision-language-action (VLA) policy without architecture changes, and concludes the bottleneck is instrumentation, not translation. The spine is a set of controls, five of which destroyed the authors' own preliminary conclusions (Table 3). Central claims: (1) single-goal suites cannot attribute language-following - 84.6% Greek versus 82.6% under deliberately wrong instructions on LIBERO-10 (Section 5, Table 1); (2) target-language demonstrations are necessary but not sufficient - on LIBERO-90 the Greek-only margin over its wrong-instruction floor never exceeds 2.7 points across three seeds, against 6.7-7.1 for bilingual training; (3) policies overfit the training translator (mean 23.3-point drop, about 40% of the headline level); (4) warm-starting from a Greek-adapted world model (one run per arm) and unfreezing the text tower (three seeds) both degrade both languages.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can robot policies follow instructions in languages other than English?
Yes, but only with demonstrations in the target language, according to this study: a Cosmos3 policy trained with machine-translated Greek alongside English reached about half of its English success on ten LIBERO tasks, while a multilingual text tower without Greek demonstrations transferred nothing to behavior.
What is a wrong-instruction control in robot learning evaluation?
It means evaluating a policy while giving each task a different task's instruction. If success barely drops, the policy is reacting to the scene rather than the language. This paper measured 84.6% success under correct Greek instructions versus 82.6% under deliberately wrong ones on a single-goal suite, exposing that benchmark as non-discriminative.
How much does machine translation quality matter for training robot policies?
A lot: switching from the training translator to an independently produced Greek translation cost a mean of 23.3 success points (about 40% of the headline number), while comparable English rewordings cost under 5 points. The authors recommend evaluating on an independently produced instruction set and training on seven phrasings per task, which roughly halves the penalty.
Is the Greek robot policy from this paper publicly available?
Yes. Sophea-Nano-Policy-LIBERO-Greek-v1 is on Hugging Face (KIEFERSA) with a model card reporting the wrong-instruction control. Note it is the best of three training seeds (80.2% Greek on ten goals), not the recipe's average (61.3%).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction interleaves figure axis labels and captions with body text; all numbers cited in this review were verified against tables or inline prose, and no end-of-text truncation was detected (the reference list ends normally).
- The Greek instruction strings appear inline in the extraction (Figure 7, Appendix B); translation-adequacy judgments here rely on the authors' own commentary rather than independent verification of the Greek.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.