ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratch
cs.CL
Quick verdict
Honest $286 build of a 151M Turkish LLM; contribution is measurement practice—contaminated safety gates, seed variance—openly released, credible, modest.
Summary
ufakzeka-1 is a 151M-parameter (182M with embeddings) decoder-only Turkish language model pretrained from scratch on 13.5B openly licensed tokens and instruction-tuned for chat, at roughly $286 in cloud GPU, API and notebook costs. The stated contribution is not capability but the documented pipeline: a byte-level BPE tokenizer at 1.77 tokens per word, a three-stage pretraining schedule (Muon, then Hyperball Muon, then a 1.5B-token anneal extending context to 4,096), an SFT mixture of licensed and generated data, and an evaluation battery combining lm-evaluation-harness benchmarks, nine release gates, a rule-checked sweep of 5,508 conversations, LLM-judged conversations and hand tests — all separated from training data by 13-gram decontamination plus a checked-in invariant script. On Turkish benchmarks the base model sits within three points of five-times-larger Turkish models on ARC-easy and XCOPA while trailing by 5–8 points on TurBLiMP grammar. Three transferable findings are reported: a safety gate 'repaired' with training data written from its own probe questions read 64/64 versus an honest 34/64 under unseen paraphrases; three-seed variance (helpfulness 75.2–81.2) matched the spread across fourteen recipe checkpoints, making single-seed comparisons uninformative; and long-context identity tracking plus multi-turn arithmetic never improved with any data change, which the authors attribute to model size. Limitations include an unnamed judge model, an AI-rated hand test, small gate samples, and research-grade safety.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A 151M-parameter Turkish model pretrained from scratch on 13.5B tokens lands within three points of five-times-larger Turkish models on ARC-easy and XCOPA, trailing by 5-8 points on TurBLiMP grammar.
- A safety gate repaired with training data written from its own probe questions read 64/64 while the honest paraphrase-based score was 34/64, with nothing in the scores signalling the inflation.
- Training-seed variance (helpfulness 75.2-81.2 across three seeds) matched the spread across fourteen recipe checkpoints, so single-seed recipe comparisons at this scale are uninformative.
- Long-context identity tracking and multi-turn arithmetic did not improve under any data change across the last eight rounds, which the authors read as a limit of 151M parameters to be tested on a larger model.
- The complete pipeline - weights, data recipe, evaluation code, result files and spend ledger - is released under Apache-2.0 at roughly $286 total cost.
Strengths
End-to-end reproducibility
Weights (stock Qwen3 exports), GGUF f16/q8_0 verified for identical tokenization and greedy answers, data-generation and evaluation code, result files behind every Table 1 row, and the spend ledger are all Apache-2.0 (Sections 1, 8).
The contamination finding is genuinely valuable
Section 6 documents with numbers how a safety gate 'repaired' with training data written from its own probe questions read 64/64 while paraphrase-based honest measurement gave 34/64, and pairs the diagnosis with a code-enforced remedy (13-gram drop plus a checked-in invariant script run before each build).
Honest seed-variance reporting
Tokenizer choices justified by measurement, not assertion
Transparent cost accounting
Weaknesses
LLM judge is not identified
Hand test rated by the development AI
Several release gates ship with very small samples
Most log-likelihood benchmarks separate nothing
Seed-variance conclusion generalized from n=3
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a self-described pipeline report, not a capability claim: ufakzeka-1 is a 151M-parameter (182M with embeddings) decoder-only Turkish chat model pretrained from scratch on 13.5B openly licensed tokens and SFT-tuned for about $286 total. The authors explicitly disclaim capability as the contribution and stake three transferable findings on measurement instead: (1) probe contamination inflated a safety gate from an honest 34/64 to 64/64 (Section 6); (2) three-seed variance (helpfulness 75.2-81.2) matched the spread across fourteen recipe checkpoints, making single-seed comparisons uninformative; (3) long-context identity tracking and multi-turn arithmetic resisted every data change, read as a parameter-count limit. Claims are consistently scoped, and the paper is unusually its own best critic - the abstract leads with the failure story.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ufakzeka-1 and how much did it cost to build?
ufakzeka-1 is a 151M-parameter Turkish chat model trained from scratch on 13.5 billion openly licensed tokens. The whole project - pretraining, fine-tuning and evaluation - cost about $286, itemized from actual bills: $237 of Modal compute, $37 of LLM API calls, and about $12 of Colab units between August and mid-September 2026.
How good is ufakzeka-1 compared to other Turkish models?
On zero-shot Turkish benchmarks it is within three points of 750M-class Turkish models (Kanarya-750m, turkish-gpt2-large) on ARC-easy and XCOPA, trails them by 5-8 points on TurBLiMP grammar, and beats Qwen2.5-0.5B on every Turkish task except Belebele. It remains a research model: it invents facts, can corrupt its own arithmetic checks across turns, and cannot write code.
What is the main lesson of the paper for small-model projects?
Three lessons: evaluating a safety gate with questions that also appear in its training data inflated the score to 64/64 when the honest figure on unseen paraphrases was 34/64; training-seed variance was as large as the spread across every training recipe tried, so single-seed comparisons are meaningless at this scale; and two behaviours (long-context identity tracking, multi-turn arithmetic) resisted every data fix, which the authors read as a parameter-count limit.
Is ufakzeka-1 open source?
Yes. Apache-2.0 releases cover the base and chat weights (plus GGUF files at f16 and q8_0), the tokenizer and llama.cpp pre-tokenizer patch, the data-generation and evaluation code, the evaluation result files of the released checkpoint, and the spend ledger. A public demo runs at chat.ufakzeka.com.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All tables appear fully inline in the extracted text; no figures required visual analysis, so extraction loss is likely minimal.
- The judge model identifier is absent (by the paper's own admission), so the judged scores (79.2, 84.5) cannot be independently verified from the text.
- Result files for earlier candidates are not shipped, so Section 6's round-over-round numbers rest on the authors' reporting and could not be cross-checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.