Fast And Accurate Text Content File Type Identification
Manu Nandan, Michael Brautbar, Edward Raff, Edward Raff
cs.LG
Quick verdict
Claims LABELER beats Magika on 54 text file types (Macro F1 0.98 vs 0.91, 3.7x faster); plausible but gains are confounded and evaluation lacks ablations.
Summary
The paper introduces LABELER, a compact dual-input convolutional neural network (~715 KB parameters) that identifies the type of text-content files — source code, configuration formats, markup — from content rather than extensions. A custom Rust tokenizer splits files into lines, sub-words at non-alphanumeric boundaries and camelCase components, mapping the 4,096 most frequent symbols to integer IDs; the network processes only the first and last 4 KB chunks through a shared embedding (d=16) and 1D convolution before concatenation and softmax. Training uses ~9.75 million permissively licensed files from The Stack/Stack V2 covering 54 languages, with c/java/js variants merged into family labels. Against the sole baseline Magika, the paper reports Macro F1 0.9804 vs 0.9057, Weighted F1 0.982 vs 0.921, and 1.05 ms vs 3.85 ms per file (LABELER on one CPU core, Magika unconstrained) — a 3.7x speedup while consuming 5.3x more input bytes. LABELER leads every supported class, with both models weakest on plain 'text' files (0.79 vs 0.31). Main limitations: input size, family grouping, training data and thresholds all differ from the baseline with no ablations isolating the tokenizer's contribution; ground-truth label construction is not described; the latency benchmark uses random bytes; no variance or significance reporting; 'unknown' predictions are counted inconsistently (FN vs FP); and no code release is mentioned.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LABELER achieves Macro F1 0.9804 versus Magika's 0.9057 across 54 text file types (Table I).
- LABELER infers in 1.05 ms per file on a single CPU core versus Magika's 3.85 ms, a 3.7x speedup while processing 5.3x more input bytes (8 KB vs 1.5 KB).
- The model is about 28% smaller than Magika (~715 KB vs ~1 MB parameters), with the Rust tokenizer wheel adding only 220 KB.
- A domain-specific tokenizer plus dual-chunk architecture yields higher sample efficiency than Magika, with higher recall at only 2,000 training files per type.
- LABELER covers text file types unsupported by Magika 1.0.1, including AppleScript, ColdFusion, ActionScript and Java Server Pages.
Strengths
Pareto framing with both axes measured
Table I reports accuracy and efficiency side by side (Macro F1 0.9804 vs 0.9057; 1.05 ms vs 3.85 ms per file), and Section IV-C describes the timing protocol (100k calls, first 100 excluded, single pinned core, Magika unconstrained) in enough detail to critique and replicate.
Domain-motivated tokenizer with worked example
Algorithm 1 (line splitting, non-alphanumeric splitting, camelCase splitting, lowercasing, frequency-ranked K=4096 vocabulary) is fully specified, and the define-vs-def vs word-piece fragmentation example in III-A concretely justifies rejecting standard NLP tokenizers.
Complete per-class results over all 54 types
Sample-efficiency analysis for rare types
Weaknesses
Headline comparison is confounded, no ablations
Ground-truth labeling never described
Timing methodology has realism and fairness gaps
No variance or significance reporting
Internal inconsistency on 'unknown' accounting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper proposes LABELER, a lightweight dual-input CNN plus a custom tokenizer for identifying the type of text-content files (54 source-code, configuration and markup classes) from content, motivated by security workflows where file extensions and magic bytes are untrustworthy (Section I). Central claims: Macro F1 0.9804 vs Magika's 0.9057 and Weighted F1 0.982 vs 0.921 (Table I); 1.05 ms vs 3.85 ms per file (Fig. 3), i.e. 3.7x faster on one CPU core while consuming 5.3x more input bytes; and a 28% smaller model (715 KB vs ~1 MB). The scope is deliberately narrower than Magika's: text files only, no binary types, with c/java/js collapsed into family labels.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the LABELER model in this paper?
LABELER is a small (~715 KB parameter) dual-input convolutional neural network with a custom tokenizer that reads the first and last 4 KB of a text file — source code, configuration, or markup — and predicts among 54 file types. The authors report higher accuracy and roughly 3.7x faster inference than Google's open-source Magika content-detection tool.
How much faster is LABELER than Magika?
The paper measures 1.05 ms per file for LABELER pinned to a single CPU core versus 3.85 ms for Magika version 1.0.1 via its Python API without CPU limits, on an Apple M1 Max — about 3.7x faster — while LABELER processes 8 KB of input versus Magika's 1.5 KB, a 19.6x per-byte advantage.
Which file types does LABELER support?
54 text file types, including mainstream languages (Python, Go, Rust, Java), markup and config formats (HTML, XML, YAML, TOML, JSON), and AppleScript, ColdFusion, ActionScript and Java Server Pages that Magika 1.0.1 does not support. C/C++, Java/Groovy and JavaScript/TypeScript are merged into family labels. Its weakest class is generic 'text' files (F1 0.79).
Is LABELER's code or model available?
The paper does not mention releasing code, weights, or data splits. Training data comes from the public BigCode Stack datasets and the baseline is the open-source Magika package, so an independent reproduction would require reimplementing the tokenizer and network from the paper's description.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3-6 are present only as axis labels and captions in the extracted text; exact curve values could not be analyzed, so Fig. 5-6 readings rely on the prose.
- The review is based on the provided v1 extraction; if appendices or supplementary material exist, they were not visible.
- All quantitative comparisons with Magika rely on the authors' own measurements; no independent verification was possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.