Closing Cost-Quality Gap in Document VLMs: Difficulty-Aware Data Curation and Quality-Adjusted Deployment Economics
Maksim Evdokimov, Matvey Ivanov, Dmitrii Tsiupin, Olga Tsymboi, Anatolii Potapov, Aleksandr Ivanov
cs.CL
Quick verdict
Claims a 35B-A3B document VLM closes the cost-quality gap via difficulty-aware curation; plausible but rests on private benchmarks and unaudited cost math.
Summary
The paper presents a deployed document-understanding system for Cyrillic business documents (court files, invoices, certificates) built on Qwen3.5-35B-A3B, a 35B-total/3B-active MoE VLM fine-tuned with SFT on roughly 210K in-house documents plus synthetic supervision generated from 300K Common Crawl PDFs. A Difficulty-Aware Data Curation (DADC) pipeline keeps only hard, information-rich samples through text-layer validation against OCR, VLM-judged layout taxonomy and fact-extractability filters, and cross-model consistency verification, retaining 35-40% of candidates. The model averages 0.767 on non-fine-tuned benchmarks (+3.4 over its base, +2.4 over a reasoning-mode 397B baseline) and 0.814 including the fine-tuned group; it meets a 10-second P95 latency SLA on a single H100. Under a quality-adjusted cost model calibrated from 50-annotator production telemetry, expected per-document costs fall by over 80% versus human annotation, while the 397B baseline is cost-negative and breaches latency limits. Main limitations: all primary benchmarks and cost telemetry are proprietary; no confidence intervals, significance tests, or multi-seed variance are reported; base-model numbers differ inconsistently across Tables 1, 2, 3, and 9; the introduction's 53% cost figure cannot be reconstructed from Table 5; and the fine-tuned group (0.956) flatters the headline average.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- DADC-curated synthetic data plus in-house SFT lifts Qwen3.5-35B-A3B to 0.767 average on non-fine-tuned benchmarks, +3.4 over its base and +2.4 over the reasoning-mode Qwen3.5-397B-A17B-FP8 (Table 1).
- Unfiltered open-domain data degrades the in-house-only baseline (Avg w/o FT 0.753 to 0.661); only the full verified DADC pipeline turns open data into a positive scaling factor (0.768, Table 9).
- The quality-adjusted cost model yields cost-reduction ratios of 0.819/0.861 for the deployed model versus negative values for Qwen3.5-397B (-0.115/-1.582), making larger baselines economically unviable (Table 5).
- The deployed model meets a 10-second P95 latency SLA on a single H100 (up to 7.41s at 5 workers) while the 72B and 397B baselines breach it at 2-3 workers (Table 13).
Strengths
Curation ablation isolates each pipeline stage
Table 9 shows unfiltered open data degrades Avg w/o FT from 0.753 (in-house only) to 0.661; stage-1 filtering recovers 0.752, sampling 0.755, and only verification+sampling reaches 0.768 — evidence the difficulty filter, not raw data volume, drives the gains.
Deployment realism beyond leaderboard scores
Single-H100 serving with P95 7.41s at 5 workers under a 10s SLA (Table 13); reasoning baselines are excluded as non-deployable rather than silently compared; cost coefficients are fit from 3-month, 50-annotator telemetry (Appendix F.1: confirm 0.30 in [0.27-0.34], correct 1.08 in [1.04-1.17]).
Unusual pipeline transparency for an industry paper
Contamination control is specified, not asserted
Weaknesses
No uncertainty quantification
Numeric inconsistencies across tables
Headline average rides on the fine-tuned group
Unreconstructable 53% figure
Contradictory fine-tuning history
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This industrial systems paper from T-Tech describes a deployed Cyrillic-document VLM and the methodology around it: (1) DADC, a difficulty-aware curation pipeline converting Common Crawl PDFs into verified synthetic supervision; (2) SFT of Qwen3.5-35B-A3B-Base (35B total / 3B active) on ~210K in-house documents (60K judicial, 150K invoice instances) plus open-domain synthetic data, ~3B tokens total; (3) a closed-form quality-adjusted cost model calibrated from production telemetry. Central claims: 0.767 average on non-fine-tuned benchmarks (+3.4 over base, +2.4 over reasoning-mode Qwen3.5-397B-A17B-FP8, Table 1); single-H100 deployment within a 10s P95 SLA (Table 13); and expected cost reductions of 80%+ versus human annotation with 397B-class baselines cost-negative (Table 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DADC in this paper?
DADC (Difficulty-Aware Data Curation) is the paper's pipeline for turning raw Common Crawl PDFs into training data: it validates PDF text layers against OCR, filters pages by visual layout taxonomy and fact-extractability using a VLM judge, verifies generated question-answer samples via cross-model agreement, and keeps only 35-40% of candidates — the hard, information-rich ones. The paper's ablation shows unfiltered open data actually degrades performance.
How much cheaper is the model than human annotators?
In the paper's two production workflow groups, the quality-adjusted cost model estimates expected per-document cost reductions of 0.819 and 0.861 — over 80% versus the fully manual baseline. The coefficients (confirmation at 0.3x, correction at 1.1x of manual cost) were calibrated from three months of telemetry covering 50 professional annotators at the company, not from public data.
Is the model, code, or dataset publicly available?
No release is mentioned. All 12 primary benchmarks are internal company benchmarks and the cost telemetry is proprietary production data; only public third-party benchmarks (MWS, DocVQA, InfoVQA, OCRBench, OCRBench v2, CC-OCR) allow partial external comparison.
What base model is used and what hardware does it need?
The system fine-tunes Qwen3.5-35B-A3B-Base, a Mixture-of-Experts VLM with 35B total and 3B active parameters, via LoRA (0.8B trainable parameters). At inference it fits on a single H100 and holds a 10-second P95 latency SLA, whereas the 72B and 397B baselines needed 1-8 GPUs and breached the same SLA.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 6 (augmentation examples) is an image; its visual content could not be verified from text extraction.
- Only group-level aggregate scores are printed in the paper (no per-subbenchmark breakdowns); this is a property of the paper, not a text-extraction gap.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.