Baszta: Data-Centric Fine-Tuning of a Polish Multi-Label Safety Classifier
Adam Górski, Mateusz Jąkalak, Rafał Jakubowski
cs.AI
Quick verdict
Reports a small but significant micro-F1 lead over Sójka on Polish safety classification; unusually honest self-critique, but OOD gains risk style overfitting.
Summary
This industrial empirical study fine-tunes HerBERT-base-cased (124M parameters) into a five-category Polish content-safety classifier (hate, vulgarity, sex, crime, self-harm) using a Focal (γ=1.5) + R-Drop objective over a 26,248-sample pool aggregating public corpora plus 1,294 style-matched synthetic jailbreak-style prompts, then compares it against Bielik Guard (Sójka, 0.1B) on the 520-sample adversarial Gadzi Język benchmark. Under a matched both-tuned paired bootstrap, the micro-F1 lead is significant but narrow (0.929 vs 0.903, p=0.011, marginal under Bonferroni across five tested configurations), while the advertised macro-F1 lead dissolves into an operating-point artifact once Sójka receives identical tuning (0.712 vs 0.782). The paper diagnoses the residual OOD gap as calibration rather than discrimination (high AUC, collapsed positive probabilities) and shows per-category temperature scaling transfers only when the calibration set contains safe text — fitting on the nearly all-positive benchmark flags 100% of safe inputs. A contamination audit caught PL-Guard leaking into training (899/900), and held-out evaluation on four public benchmarks yields a 2-2 split against Sójka. Key limitations: model selection on the same OOD benchmark, synthetic data style-adapted to the test distribution, exact-match-only contamination checking, macro estimated on 520 heavily skewed samples, and checkpoints/corpus not redistributed.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under a matched both-tuned paired bootstrap, the model's micro-F1 lead over Sójka on Gadzi Język is statistically significant but narrow (0.929 vs 0.903, p=0.011), while the advertised macro lead is an operating-point artifact (0.712 vs 0.782 once Sójka is tuned identically).
- The residual OOD gap is a calibration shift rather than a discrimination failure, and per-category temperature scaling recovers it only when the calibration set contains safe text (an all-positive fit flags 100% of safe inputs).
- Style-matched synthetic data is the decisive modelling ingredient (−16.2 pp OOD macro without it), and robustness must be selected for directly because standard improvements (per-class focal alpha, mean pooling) that raise in-distribution macro actively hurt OOD performance and calibration.
- A source-level contamination audit caught the PL-Guard benchmark leaking into training (899/900), and held-out evaluation on four public Polish benchmarks yields a 2-2 split against Sójka rather than a lead.
- Most of the task is lexical: a TF-IDF + logistic-regression baseline reaches within ~4.3 pp in-distribution (0.868 vs 0.9111 macro) and ~3.4 pp OOD at the deployable operating point of the fine-tuned transformer.
Strengths
Self-refuting honesty on headline claims
§4.5.2/Table 13 shows the macro lead (0.699 vs 0.619) collapses to −0.070 under matched tuning (0.712 vs 0.782), and the always-crime baseline (0.910 micro on the same split) is reported to expose that the significant micro lead (+0.026, p=0.011) barely clears a degenerate strategy — the paper dismantles its own abstract-level numbers.
Deployment-validity analysis rare in guardrail papers
§4.5.1 demonstrates that the F1-optimal fit on the all-positive benchmark drives the crime threshold to 0.025 (temperature 4.3), flagging 100% of safe text, then reports a balanced refit (0.952 macro, 5.5% safe-text FPR, Gadzi macro 0.496) — both operating points shown, not just the flattering one.
Contamination audit with a material catch
Statistical care unusual for industrial reports
Breadth of evaluation axes
Weaknesses
Model-selection bias on the primary benchmark
Synthetic data soft-adapted to the test distribution
Fragile per-category and macro statistics
Reproducibility limited by artifact policy
Strongest competitor result understated at summary level
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Baszta is an industrial empirical study (Billennium S.A.) that fine-tunes allegro/herbert-base-cased (124M) into a five-category Polish content-safety classifier (hate, vulgarity, sex, crime, self-harm) with a Focal (γ=1.5) + R-Drop (α=0.20) objective over a 26,248-sample pool aggregating ~nine public corpora plus 1,294 style-matched synthetic imperative prompts (§3.2–3.3). Its central claims are measured and, unusually, partly self-negating: (1) under a matched both-tuned paired bootstrap on Gadzi Język, the micro-F1 lead over Bielik Guard/Sójka survives but is narrow (0.929 vs 0.903, +0.026, p=0.011); (2) the advertised macro lead (0.699 vs 0.619) is an operating-point artifact — with identical tuning Sójka reaches 0.782 against their 0.712 (§4.5.2, Table 13); (3) the residual OOD gap is calibration, not discrimination (§4.3), recoverable by per-category temperature scaling only when the calibration set contains safe text (§4.5.1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does the new Polish safety classifier beat Bielik Guard (Sójka)?
Only narrowly and conditionally. On the 520-sample adversarial Gadzi Język benchmark, with both systems given identical threshold tuning, it leads micro F1 by 0.929 to 0.903 (p=0.011, marginal after multiple-comparison correction). The advertised macro-F1 lead disappears under the same tuning (0.712 vs 0.782), and on four external public benchmarks the result is a 2-2 split.
Why is micro F1 misleading on Gadzi Język?
The benchmark is 97% crime-positive, so a degenerate classifier that flags crime on every input already scores 0.910 micro F1 — above Sójka's tuned 0.903 and only 1.9 points below the new model's 0.929. Macro F1 is the column that distinguishes real models from that strategy, but it is statistically noisy at only 520 samples.
What was the data contamination finding?
PL-Guard, on paper the ideal external Polish benchmark, had been ingested into the training pool: 899 of 900 test rows and 765 of 900 adversarial rows matched training text exactly. It was excluded from all held-out claims. The headline Gadzi Język set was verified clean (0/520 exact matches against train + validation pools).
Is the model deployable as-is?
Not at its headline operating point: fitted on the almost all-positive benchmark, it flags crime on 100% of safe text (crime threshold 0.025, temperature 4.3). A balanced refit on a calibration set containing safe text reaches 0.952 macro with a 5.5% safe-text false-positive rate but drops adversarial macro to 0.496, below Sójka's 0.619 — no single point optimizes both.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 (reliability diagrams, comparison bar charts) appear only as captions in the extracted text; their visual content could not be verified, though the underlying numbers are present in tables.
- This is an internally reviewed industrial report ('Reviewed by' two colleagues at Billennium), not a venue peer-reviewed manuscript.
- Code is 'available on request' per Appendix B and checkpoints/corpus are not redistributed, so reproducibility assessments rest on the protocol description alone.
- Extraction appears complete (references intact, sections continuous), but any data shown only inside figure images could not be checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.