Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web
Gonçalo Vinagre, Rui Pedro Guerra, Pedro Gomes, Miguel Moura Ramos, Duarte Miguel Alves, Afonso Simplício, Diogo Tavares, David Semedo, et al.
cs.CL · cs.AI
Quick verdict
41B-token PT-PT web corpus from Arquivo.pt; post-scraping block lifts yield +19.04%. Solid engineering, yet quality evidence rests on one test collection.
Summary
This paper introduces Fine PT-PT Web, a 41-billion-token European Portuguese corpus built from 411 TB of WARC data across 71 Arquivo.pt collections spanning 1997–2024. The pipeline, implemented on Datatrove and following FineWeb2's Portuguese configuration, applies URL filtering (removing .br and NSFW domains), Trafilatura extraction, GlotLID language identification (threshold 0.799), Gopher Repetition and FineWeb/Gopher Quality heuristic filters, MinHash fuzzy deduplication with rehydration weights, EuroFilter quality classification, and cross-collection deduplication. Its headline contribution is a post-scraping block — removing short (fewer than 10 alphabetic characters) and duplicated lines before heuristic filtering — that raises retained documents by 19.04% (9.19M to 10.94M) on the single FAWP21 test collection. The final corpus contains 24.6M documents, with media (41%) and blogs (43.7%) dominating tokens. Filter-stage ablations using AMALIA-9B, EuroLLM-9B, Llama-3.1-8B, and English-only OLMo-2-7B show removed documents have higher bits-per-byte, and memorization probes find over 99% of continuations matching at most six consecutive words (max 26). Two variety classifiers estimate PT-BR contamination at 13.6% and 17.1%. Key limitations: the 19.04% gain and all ablations rest on one 2015 collection; corpus quality is never validated by controlled training experiments; the contamination estimate is an ad hoc average without human validation; MinHash hyperparameters are unstated; and no dataset release URL appears in the text, only code repositories.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A post-scraping block that removes short lines and within-document duplicate lines before heuristic filtering increases final retained documents by 19.04% without compromising quality (measured on FAWP21).
- The six-stage pipeline converts 411 TB of Arquivo.pt WARC data into a 41-billion-token, 24.6M-document PT-PT corpus, a 51.3× token increase over the Arquivo.pt data used by GlórIA.
- Heuristic filters remove genuinely low-quality text, evidenced by higher bits-per-byte and lower top-1 accuracy on removed documents across four models including an English-only control.
- A model mid-trained on the corpus (AMALIA-9B) shows minimal verbatim memorization: over 99% of continuations match at most 6 consecutive words, with a maximum of 26.
- European Portuguese constitutes about 84.65% of the final corpus, per the average of two independent variety classifiers (individual estimates: 86.4% and 82.9%).
Strengths
Fills a real resource gap at scale
411 TB of WARC data become 41B tokens / 24.6M documents — a 51.3× token increase over GlórIA's Arquivo.pt subset (Table 3) and the first openly documented PT-PT-specific web pipeline (§2).
Quantified, staged ablation of the headline contribution
Yield effect decomposed step-by-step on FAWP21: 9.19M → 9.92M (short-line removal, +7.94%) → 10.94M (both steps, +19.04%), with per-filter drop counts (Fig 3, Fig 9) and concrete rescued-document examples (Appendix C, Table 5).
Cross-model filter validation with an English-only control
Memorization audit with worst-case transparency
Honest negative result and candid limitations
Weaknesses
All headline evidence comes from one collection
No controlled training validation of quality decisions
Contamination estimate is methodologically thin
Reproducibility and release gaps
Dedup and rehydration effects unvalidated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Fine PT-PT Web is a dataset-and-pipeline paper targeting European Portuguese (PT-PT) pre-training data. From 411 TB of raw WARC data across 71 Arquivo.pt collections (1997–2024, one-year embargo respected), the authors produce a 41-billion-token (GPT-2 tokenizer), 24.6M-document corpus. Three central claims: (1) a novel post-scraping block — removing lines shorter than 10 alphabetic characters and within-document duplicate lines before heuristic filtering — increases final retained documents by 19.04%; (2) the six-stage pipeline (URL filtering → Trafilatura extraction → GlotLID ≥ 0.799 → Gopher Repetition / FineWeb Quality / Gopher Quality heuristics → MinHash dedup with rehydration weights → EuroFilter classification → cross-collection dedup) yields a corpus 51.3× larger in tokens than the Arquivo.pt subset used by GlórIA (Table 3); (3) the corpus is optimized for LLM pre-training, supported indirectly by filter-stage prediction probes and a memorization audit of AMALIA-9B.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How large is the Fine PT-PT Web dataset and what data is it built from?
The corpus contains 41 billion tokens (counted with the GPT-2 tokenizer) across 24.6 million documents. It was built from roughly 411 TB of raw WARC archive data spanning 71 Arquivo.pt collections crawled between 1997 and 2024.
What does the post-scraping step in Fine PT-PT Web do and why does it matter?
It removes lines shorter than 10 alphabetic characters and duplicate lines within each document before heuristic quality filters run. On the paper's single test collection (FAWP21), this raised retained documents from 9.19 million to 10.94 million, a 19.04% increase, because documents no longer trigger filters that their raw scraped form would fail.
How much Brazilian Portuguese (PT-BR) content remains in the corpus?
Two independent Portuguese variety classifiers estimated 13.6% and 17.1% Brazilian Portuguese content respectively; their simple average is 15.35% PT-BR versus 84.65% PT-PT. The estimate has no human-labeled validation, which the review flags as a weakness.
Is the corpus or the pipeline code publicly available?
The pipeline code is openly available on GitHub (AMALIA-LLM/arquivo_processing and AMALIA-LLM/datatrove-amalia). The paper text itself names no download URL for the corpus, so its public release status is not stated in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Numeric values inside Figures 3, 6, 7 and 8 are not recoverable from the extracted text (only captions and axis labels appear); figure-level conclusions rely on the prose and captions.
- No truncation detected: the extraction includes the full reference list, Limitations, Ethical Considerations, and Appendices A–E, ending with the complete caption of Table 7.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.