EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languages
Aleix Sant, Jordi Luque, Carlos Escolano
cs.CL
Quick verdict
Task-preserving data localisation reverses MT-induced instruction-following loss in 50 languages; solid evidence, but single seed and data unreleased.
Summary
This paper introduces EuroAlpaca, a pipeline and near-parallel instruction dataset (51,760 examples from Alpaca Cleaned) covering 50 European languages, plus European-IFEval, a harmonised multilingual benchmark aggregating existing IFEval-style resources. The pipeline routes each example via an LLM gate (quantised Gemma-4-31B) into direct field-wise translation (54.0%), field-preserving translation (24.6%, keeping labels, code, or fixed-language outputs unchanged), or task-level localisation (21.4%, rewriting examples whose constraints — rhyme, letter counts, idioms, grammar correction — do not survive translation), followed by coherence validation with target-language enforcement; NLLB-3.3B performs permitted MT. Across 600 LoRA fine-tuning runs (four 3-4B models × 50 languages × three supervision conditions), direct MT raises Aya ROUGE-L/F-BERT but cuts European-IFEval strict accuracy by 0.092 absolute (29.8% relative) versus the unadapted baseline; EuroAlpaca instead adds +0.040 (12.9% relative) and beats direct MT in all 200 model-language pairs. Ablations attribute most recovery to task-preserving routing (+0.095 over direct MT) and a further +0.037 to final validation. Against Okapi and MITS, EuroAlpaca wins on shared-language subsets, though Bactrian-X retains higher IFEval accuracy on its 20-language intersection. Limitations: single seed without significance testing, manual quality inspection only on a small sample, and neither dataset nor code publicly released at submission.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
5 strengths · 0 weaknesses · 0 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Directly machine-translated instruction data improves reference-similarity metrics (Aya ROUGE-L/F-BERT) but reduces European-IFEval strict accuracy by 0.092 absolute (29.8% relative) versus the unadapted baseline.
- EuroAlpaca reverses this degradation, improving instruction-following accuracy by 12.9% relative over baseline while also achieving the highest Aya scores, and beats Direct MT in all 200 model-language pairs.
- Task-preserving routing and localisation account for most of the recovery (+0.095 over Direct MT), with final coherence validation adding a complementary +0.037.
- Near-parallel alignment is compatible with competitive downstream performance, outperforming Okapi and MITS on shared-language subsets, though Bactrian-X retains higher IFEval accuracy on its 20-language intersection (0.401 vs 0.385).
Strengths
Unusually controlled large-scale experimental design
600 LoRA runs (4 models × 50 languages × 3 conditions, §5.1) with hyperparameters and evaluation held fixed within each model-language pair, isolating the supervision-data variable; per-language results for all cells are reported (Appendix C, Tables 15–33).
Clean ablation separating task-preservation from final validation
The Task-Preserved intermediate condition (§5.1, §6.2) separates routing/localisation gains (+0.095 IFEval over Direct MT) from final coherence validation (+0.037), supported by concrete worked examples (Tables 11–12).
Highly consistent headline effect
Transparent, honest reporting norms
Detailed reproducibility apparatus
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper contributes (1) EuroAlpaca, a task-preserving localisation of the 51,760-example Alpaca Cleaned corpus into 50 European languages and regional varieties; (2) European-IFEval, a harmonised aggregation of existing IFEval-style benchmarks for the same language set; and (3) 600 language-specific LoRA fine-tuning runs across four 3–4B instruction-tuned models. Central claims: direct field-wise MT of instruction data inflates reference-similarity metrics while degrading verifiable instruction following (−0.092 strict accuracy, −29.8% relative, macro across models, Table 4); EuroAlpaca reverses this (+0.040 absolute, +12.9% relative) while also attaining the best Aya ROUGE-L/F-BERT. Claims are scoped carefully: external comparisons (Okapi, MITS, Bactrian-X) are explicitly pairwise, and the authors decline to rank those resources against one another (§6.3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EuroAlpaca?
It is an instruction-tuning dataset of 51,760 examples derived from Alpaca Cleaned, localised into 50 European languages. A pipeline routes each example: direct machine translation where safe, preservation of task-critical fields (code, classification labels, fixed-language outputs) where translation would break the task, or full rewriting for tasks whose constraints (rhymes, letter counts, idioms, grammar errors) cannot survive translation.
What is the paper's main finding?
Fine-tuning on directly machine-translated instruction data improves similarity-to-reference scores (ROUGE-L, BERTScore) but cuts verifiable instruction-following accuracy on European-IFEval by 29.8% relative versus the unadapted model. EuroAlpaca's task-preserving data instead improves accuracy by 12.9% relative while also achieving the best similarity metrics, winning all 200 model-language comparisons against direct MT.
Is the EuroAlpaca dataset publicly available?
Not at submission time (double-blind anonymity). The authors commit to releasing both datasets, construction code, prompt templates, provenance manifests and evaluation scripts upon acceptance. The underlying European-IFEval sources (e.g., Marco-Bench-MIF, Multi-IFEval) are already public on Hugging Face.
Which models were evaluated?
LoRA fine-tuning (rank 16) of four instruction-tuned models in the 3-4B parameter range: Tiny-Aya-Global, Phi-4-Mini-Instruct, Ministral-3-3B-Instruct-2512, and Llama-3.2-3B-Instruct — one adapter per model-language-condition, 600 runs in total.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-6 are raster graphics; only partial embedded text (e.g., Figure 6's per-language gain values) was extractable, so figure-level details could not be fully verified.
- GitHub repository URLs appear line-wrapped in the extraction (e.g., 'Telefonica-Scientific-Res earch'); the exact URLs were not independently verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.