Language, Language Models, and What We're Talking About
cs.CL
Quick verdict
Position paper: most 'Italian' LLMs are English backbones tuned on translated/synthetic data; persuasive critique but anecdotal, proposal programmatic.
Summary
This CLiC-it 2026 position paper argues that contemporary 'Italian' large language models have little to do with the Italian language. Nissim reconstructs the Italian generative LM landscape from GePpeTto (2020) and IT5 (2022) — both trained from scratch on Italian corpora — through LLaMA-based adaptations (Camoscio, Fauno, Anita, Steered-ITA) built on machine-translated or synthetic instruction data, to Minerva (2024), trained from scratch on 2.5 trillion tokens (1.14T Italian). She documents that LLaMA-2 pre-training contained only about 0.11% Italian, that the Camoscio instruction set is a GPT-3.5 translation of Alpaca with checkable cultural and linguistic failures (Figure 3), and that evaluation relies largely on translated benchmarks such as MMLU-IT (Table 3). Connecting this to a linguistic history running from Saussure to usage-based approaches, she claims the field conflates two incompatible goals: building language technology and studying language. Her proposal is to formalise two research directions — deployable models openly producing artificial 'nobody's language', and separate ecologically faithful models as objects of linguistic inquiry (BabyLM as partial precedent). Limitations: the evidence is anecdotal (the author concedes cherry-picking dataset examples), no new experiments quantify how widespread the translation failures are, and the two-track proposal remains programmatic, without operational criteria or feasibility analysis.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 1
● low 2
4 strengths · 4 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Most post-2022 'Italian' LLMs (Camoscio, Fauno, Anita, Steered-ITA) are English-pretrained LLaMA backbones specialised via machine-translated or synthetic instruction data, and are not meaningfully models of Italian.
- Evaluation replicates the artificiality: the benchmarks commonly used (MMLU-IT, ARC_it, HellaSWAG_it) are mostly automatic translations of English originals, measuring displaced rather than native language use.
- The standard three-stage pipeline — translated/synthetic pre-training input, translated instructions, alignment toward external criteria — deliberately produces homogeneous 'nobody's language' far from natural language.
- NLP should formalise two separate research directions: deployable models that openly produce artificial language, and ecologically faithful models maintained as objects of linguistic study.
Strengths
Checkable evidentiary base, not vibes
Section 3 grounds the argument in verifiable numbers: LLaMA-2's pre-training was 89.70% English with ~0.11% Italian (quoted from [13], Table 10), Phi-3 descends from the fully synthetic TinyStories/CodeTextbook datasets, and Minerva's corpus split (1.14T Italian / 1.14T English / 200B code of 2.5T tokens) is stated precisely — the critique can be independently audited.
Vivid, reproducible failure taxonomy for translated data
Figure 3's four Camoscio failure types (cultural mismatch: Bloody Mary as Italian breakfast; lost wordplay: the anagram 'the eyes'/'they see'; literal annotation translation: 'Oggetto'→'Io'; translated output defying the instruction) and Table 3's MMLU-IT items ('Stivale di Denver' for 'Denver boot', 'Qual è il tuo alluce?' for 'hallux') make the abstraction concrete and instantly verifiable against public datasets.
Historically literate framing
Unusual epistemic honesty
Weaknesses
Anecdotal core evidence
No new experiments
Programmatic, unfalsifiable proposal
Rhetoric flattens real distinctions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a position paper by Malvina Nissim (CLCG, University of Groningen) for CLiC-it 2026 (Palermo, September 14–16, 2026; arXiv:2609.03577). Its scope is deliberately conceptual: using Italian language models as a case study, it asks whether systems built on English pre-training, machine-translated/synthetic instruction tuning, and heavy alignment deserve the label "models of Italian" — or "models of language" at all.
Four central claims emerge. First, the dominant post-2022 Italian LLM lineage (Camoscio, Fauno, Anita, Steered-ITA) consists of English-pretrained LLaMA backbones specialised at "Stage 2" with GPT-3.5-translated Alpaca instructions or synthetic self-chat data. Second, evaluation replicates this artificiality: benchmarks such as MMLU-IT, ARC_it and HellaSWAG_it are mostly translations of English originals. Third, the same pipeline — translated data, synthetic data, alignment toward external criteria — produces homogeneous "nobody's language", so the artefacts fail as models of natural language on both input and output ends. Fourth, NLP should formalise a split between technology-building and language-studying research lines. The paper is explicit that it offers "many questions without answers" and is "not something I am fully convinced about" — an unusual, commendable degree of epistemic modesty.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main argument of Nissim's 'Language, Language Models, and What We're Talking About'?
The paper argues that most 'Italian' language models are English-pretrained models fine-tuned on machine-translated or synthetic instruction data (e.g., GPT-3.5-translated Alpaca) and evaluated on translated benchmarks (e.g., MMLU-IT), so they have little to do with actual Italian. It calls on NLP to separate two goals: building deployable language technology — which may openly produce artificial language — and building ecologically faithful models for studying language itself.
Where was this paper published and was it peer reviewed?
It is a position paper by Malvina Nissim (University of Groningen) for CLiC-it 2026, the Twelfth Italian Conference on Computational Linguistics (Palermo, September 14–16, 2026), published under a CC BY 4.0 license and also posted as arXiv preprint 2609.03577 (cs.CL). Conference proceedings papers undergo peer review, though as a position paper it presents argumentation rather than experiments.
Why does the author use the phrase 'Italian models my ass'?
It is the deliberately provocative title of Section 3 (the author also offers the Italian rendering 'Modelli italiani un cazzo!'), signalling her thesis that LLaMA-based Italian adaptations — trained on GPT-3.5-translated Alpaca instructions and tested on translated MMLU-IT benchmarks — do not deserve the label 'Italian models'.
What evidence does the paper give that translated benchmarks are problematic?
It shows MMLU-IT items that are literal translations of US-specific questions — 'Brooklyn clamp' becomes 'Stivale di Denver', 'What is your hallux?' becomes 'Qual è il tuo alluce?' — and Camoscio instruction-data failures such as a Bloody Mary recommended as an Italian 7:30am breakfast and an English anagram puzzle ('the eyes'/'they see') that loses its point entirely in translation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure images (Figures 1, 2, 4) are not visually assessable from the text extraction; their content is taken from captions and in-text descriptions only (Figure 3's examples are transcribed in text and fully usable). No truncation detected — the text ends at the final reference [47].
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.