Evaluating Open-Weight LLMs for Turkish Domain Documents Under Retrieval and Hardware Constraints
Imtiaz Ul Hassan, Öykü Akbulut, Onur Kaya, Ardhendu Behera, Swagat Kumar, Peter Matthew, Yonghuai Liu
cs.CL · cs.IR · cs.LG
Quick verdict
Careful benchmark of five 7-8B open-weight LLMs on Turkish document QA on a 6GB GPU; sound evidence-annotated protocol, but two documents limit reach.
Summary
The paper evaluates five open-weight 7B–8B LLMs (Trendyol-LLM-8B-T1, Qwen2.5-7B, Cosmos Turkish-Llama-8B, Mistral-7B, Kocdigital-LLM-8B) for question answering over long Turkish PDFs, run locally on a 6 GB RTX 3050 laptop GPU with 4-bit quantization via Ollama. The authors build two 100-question benchmarks: a primary set from a 109-page industrial R&D report and a robustness set from a 112-page public-sector report, each covering factual retrieval, numerical reasoning, multi-step inference, and abstention. The main methodological contribution is an evidence-annotated protocol: each question stores its supporting document figures, so evidence recall is measured independently of answer accuracy and retrieval failures can be separated from model failures without extra model calls. On the primary report, accuracy spans 49% to 75%, with Trendyol highest and Qwen2.5 offering the best accuracy-latency trade-off (65% at 1.69 s/question). Across seven lexical, dense, and hybrid retrievers at a fixed 3,500-character budget, tested with Wilson intervals and exact paired McNemar tests, none significantly beats character TF-IDF. Evidence recall is 57% on the primary report versus 91% on the secondary (31/54 and 70/77 questions), with document-dependent saturation. Limitations: only two documents, small recall denominators, no multiple-comparison correction, no code or benchmark release, and no per-model error-attribution cross-tabulation.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under identical retrieval, prompting, decoding, and 6 GB hardware conditions, five open-weight 7B–8B models show a 26-point accuracy spread (49%–75%) on a 100-question Turkish industrial-report benchmark.
- Recording each question's supporting evidence allows retrieval failure and model reasoning failure to be separated without additional model calls.
- None of seven lexical, dense, or hybrid retrieval configurations significantly outperforms character TF-IDF on either document (exact paired McNemar, all p>0.05).
- Evidence recall saturates differently per document — 57% vs 91% at a 3,500-character budget — so retrieval and effective context capacity bind only for some documents.
- Turkish-specific adaptation alone does not guarantee stronger document-grounded performance: general multilingual Qwen2.5-7B outranks both Turkish-adapted Llama models.
Strengths
Evidence-annotated protocol cleanly separates retrieval failure from model failure
Recording per-question source figures lets evidence recall be measured independently of model output (Sec. III-B, V), attributing errors without extra model calls — a cheap and transferable design.
Controlled comparison isolates generator variance
Retrieval runs once per question and the identical 3,500-character context, prompt, and decoding settings are shared across all five models; Q4_K_M quantization on a 6 GB RTX 3050 via Ollama at temperature 0 is a genuinely deployment-realistic configuration.
Appropriate and honest statistics
Cross-document replication with contrasting documents
Quantified accuracy-latency trade-off
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether five open-weight 7B–8B LLMs can reliably support document-grounded question answering over long, structurally complex Turkish PDFs when deployment is confined to a consumer laptop GPU — a setting motivated by confidential industrial documents that cannot be sent to cloud APIs. Two 100-question benchmarks are constructed and validated: a primary set from a 109-page industrial R&D report (452 chunks after layout-preserving extraction) and a robustness set from the 112-page 2024 KVKK public-sector activity report (348 chunks). Each set is banded 30 Easy / 30 Medium / 20 Hard / 20 Abstention, and each question records its expected answer, supporting evidence, and a grading rule. Central claims: (1) a 26-point end-to-end accuracy spread (49%–75%) across models under identical conditions; (2) an evidence-annotated protocol separating retrieval failure from model reasoning failure without extra model calls; (3) no significant advantage for any of seven retrieval configurations over character TF-IDF; (4) document-dependent saturation of evidence recall (57% vs 91%). Claims are consistently hedged — the authors explicitly scope the second document as a robustness check and warn the two recall values are not a controlled ranking.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which open-weight LLM is best for Turkish document question answering?
In this study, Trendyol-LLM-8B-T1 scored highest (75% on a 109-page industrial report, 82% on a public-sector report), but ran about 14x slower per question than Qwen2.5-7B, which offered the best accuracy-latency trade-off (65% at 1.69 seconds per question on the primary benchmark).
Do I need a dense or hybrid retriever for RAG over Turkish documents?
The paper found that none of seven configurations (character/word TF-IDF, BM25, multilingual E5-small/base, and two reciprocal-rank-fusion hybrids) significantly outperformed a simple character TF-IDF baseline at a fixed 3,500-character context budget. However, the tests had limited statistical power (only 54-77 recall-eligible questions), so absence of evidence is not proof of equivalence.
What GPU do you need to run 7B-8B LLMs locally for document QA?
The authors ran all five models on a 6 GB NVIDIA RTX 3050 laptop GPU using 4-bit Q4_K_M quantization, a 4,096-token context window, temperature 0 with a fixed seed, and the Ollama serving stack — a consumer laptop setup.
What is an evidence-annotated evaluation protocol for RAG?
It means each benchmark question stores the specific document content (e.g., source figures) its answer depends on. A pipeline can then measure whether that evidence made it into the retrieved context independently of whether the model answered correctly — separating retrieval failures from model reasoning failures without any extra model calls.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 appear only as captions in the extracted text; visual details (accuracy–latency scatter, band bars) could not be independently verified.
- Evidence-recall values for word TF-IDF and E5-small are not stated in the text, so the seven-way retrieval comparison cannot be fully checked.
- No supplementary material, prompts, or appendix could be inspected; the absence of a code/data statement is based on the provided text only.
- arXiv v1 (23 Sep 2026); page-limit formatting suggests tables are compressed and some pipeline parameters may live in unstated supplementary files.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.