Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora
Chirag Garg, Eelaaf Zahid, Farhan Ahmed, Jay Pankaj Gala, Eric Butler, Heiko Ludwig
cs.LG · cs.AI
Quick verdict
Claims targeted crawling gives 70x math yield vs archive filtering and GSM8k parity; strong, but lacks a focused-crawler baseline and has probe circularity.
Summary
Data Scout (IBM Research) inverts the standard recipe for domain-specific pretraining corpora: instead of filtering a large web archive, it directs a targeted crawl. An LLM (phi-4) expands a root topic into a taxonomy and thousands of search queries; returned seed URLs are deduplicated, compliance-filtered, clustered by subdomain, and screened by a shallow probe crawl scored with a user-supplied domain classifier; only subdomains clearing a threshold receive a bounded deep crawl. On mathematics with the FineMath classifier as probe, 4,816 queries funnel down to 1,146 admitted subdomains and 9.56M crawled pages; 21.9% of scored pages reach the FM3+ bar versus 0.31% for a 100K RedPajama-v2 sample (70x), and 63.2% of a 1,000-URL sample is absent from CommonCrawl, mostly page-level rather than site-level gaps. A controlled ablation on 82 matched seeds shows in-subdomain pages are 28.5% FM3+ versus 1.33% for out-of-subdomain pages within the same registered domain (21x), justifying per-subdomain admission. Continued pretraining of Llama-3.2-3B on 1.9B tokens matches FineMath on GSM8k (28.7/28.6 vs 28.8/28.4 flex/strict). An astronomy replication gives 51.9% relevance (82x baseline). Limitations: no head-to-head against classical per-URL focused crawlers, evaluation reuses the admission probe, downstream testing covers only mathematics at 1.9B tokens, and Google Search API dependence complicates replication.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
6 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Domain relevance has a sharp discontinuity at the subdomain boundary: in-subdomain pages are 28.52% FM3+ versus 1.33% for sibling pages of the same registered domain (21x, Table 4, 82 matched seeds).
- Data Scout achieves 21.9% FM3+ relevance on its scored mathematics pages, 70x the 0.31% rate of a 100K RedPajama-v2 sample filtered with the same FineMath classifier (§4.1).
- 63.2% of a sampled 1,000 crawled mathematics URLs are absent from CommonCrawl entirely, driven by page-level depth gaps (59.9%) rather than undiscovered hostnames (3.3%) (Table 3).
- Continued pretraining of Llama-3.2-3B on 1.9B Data Scout tokens matches FineMath 3+ on GSM8k (28.7/28.6 vs 28.8/28.4 flex/strict, Table 5), despite 63.2% of the corpus being new to CommonCrawl.
- The pipeline generalizes: on astronomy with a different probe and threshold, 51.9% of 5.58M deep-crawled pages pass the classifier, 82x a matched RedPajama-v2 baseline (Appendix C).
Strengths
Well-controlled ablation of the core design assumption
Table 4 varies only crawl scope on 82 matched seeds with identical 5K-page budgets: in-subdomain 28.52% FM3+ (244,254 pages) vs out-of-subdomain 1.33% (106,870 pages) within the same registered domains — the 21x boundary effect is directly supported, and subdomain-scope also wins on total yield (23.90% vs 20.24%).
Uniqueness claim decomposed by mechanism, not just asserted
Table 3 splits the 63.2% CC-absence into 59.9% page-level gaps vs 3.3% hostname-level gaps (n=1,000), supporting the paper's 'depth, not discovery' conclusion with concrete counts (e.g., cs.cmu.edu indexed at 2.5M URLs but deep pages missed).
Full operational funnel transparency
Matched-budget downstream comparison with identical training config
Second-domain replication with a different probe
Honest limitation section
Weaknesses
No head-to-head against classical per-URL focused crawling
Probe circularity inflates the headline efficiency number
Downstream parity rests on single runs with no variance
Baseline and denominator caveats on the 70x figure
Internal inconsistency and thin astronomy reporting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Data Scout (IBM Research, arXiv 2609.05766) proposes inverting the standard recipe for domain-specific pretraining corpora: instead of filtering a large archive (CommonCrawl/RedPajama-v2), it directs a targeted crawl. An LLM (phi-4) expands a root topic into a taxonomy and thousands of search queries; seed URLs are clustered by subdomain and admitted to a bounded deep crawl only if a shallow probe crawl, scored by a user-supplied classifier, clears a threshold. The paper claims: (1) a sharp relevance discontinuity at the subdomain boundary (21x, Table 4); (2) 70x retrieval efficiency over archive filtering on mathematics (21.9% vs 0.31% FM3+, §4.1) with 63.2% of pages absent from CommonCrawl (Table 3); (3) GSM8k parity with FineMath at matched 1.9B tokens (Table 5); plus an astronomy replication at 82x (Appendix C).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 6 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How much better is targeted crawling than filtering a web archive?
On mathematics with the FineMath classifier, 21.9% of Data Scout's scored pages reach the FM3+ quality bar versus 0.31% of a 100K RedPajama-v2 sample scored with the same classifier — a 70x yield improvement. On astronomy with a different public classifier the gap is 82x (51.9% vs 0.63%). Note the baseline is a RedPajama-v2 sample, not CommonCrawl scored directly.
Is the crawled content already in CommonCrawl?
Mostly no: 63.2% of a 1,000-URL sample from the mathematics crawl is absent from every CommonCrawl snapshot. The dominant mechanism is page depth — for 59.9% of URLs the hostname is indexed by CommonCrawl but that specific page is not — while only 3.3% sit on hostnames CommonCrawl has never indexed.
Does the discovered data actually train competitive models?
At matched scale: continued pretraining of Llama-3.2-3B on 1.9B Data Scout tokens gives GSM8k accuracy of 28.7 (flexible) / 28.6 (strict), versus 28.8 / 28.4 for FineMath on the same 1.9B-token budget and hyperparameters. That is parity, not superiority — but the majority of Data Scout pages (63.2%) are not in CommonCrawl, so the corpus adds novel content.
Can I apply Data Scout to my own domain?
You need a domain classifier to serve as the probe — it is the only domain-specific component. The paper demonstrates mathematics and astronomy using public classifiers (FineMath, FineFineWeb-BERT) and open-sources the code, but the seed stage depends on the paid, rate-limited Google Search API (4,816 queries for mathematics), and a domain without an existing classifier requires building one first.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are image content: only their captions and in-text numbers are available in the extraction, so curve shapes, bin values, and error-bar magnitudes could not be verified against the plotted data.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.