Correction as Annotation: Bootstrapping a Dependency Parser for Documentary Medieval Latin
cs.CL · cs.DL
Quick verdict
Expert-corrected bootstrapping lifts medieval-Latin LAS .48→.92 in 33h; strong case study, but single-annotator gold and stopping-rule claim need scrutiny.
Summary
The paper shows that five off-the-shelf Latin UD models fail on 160 late-medieval Marseille inventories (1258–1446): the best labelled attachment score is 0.62 and the best morphology-aware (MLAS) score is 0.24, and performance does not track genre or period proximity. Instead of annotating from scratch, the author bootstraps an in-domain parser by correcting model output: over nine iterations, each model pre-annotates a 200-sentence batch (sampled randomly within length strata, model-blind), an expert corrects morphology then syntax in brat, and a fresh Stanza model is trained from scratch on the cumulative pool. 33.2 hours over 1,804 sentences raise universal PoS accuracy from 0.80 to 0.98 and LAS from 0.48 to 0.92 (MLAS 0.82), surpassing all five baselines while training on ~97% fewer tokens than the largest. Token-level correction effort falls from 54% to a 14–18% plateau, proposed as a gold-standard-free progress metric. Limitations: the 200-sentence gold standard and all annotations come from the single author, so no inter-annotator agreement exists; the two largest gains coincide with word-vector rebuilds, a confound the author flags but cannot disentangle; the stopping-criterion claim is asserted rather than validated, since stopping was subjective and the plateau is non-monotonic; significance procedures are described but their outcomes are not shown in the provided text. Data, code, and all nine models are released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- None of five off-the-shelf Latin UD models is usable on the DALME-Marseille inventories (best LAS 0.62, best MLAS 0.24), and genre or period proximity does not predict transfer.
- Nine iterations of correction-based bootstrapping (1,804 sentences, 33.2 hours) raise UPoS from 0.80 to 0.98 and LAS from 0.48 to 0.92, outperforming all five baselines while using ~97% less training data than the largest.
- The per-round token-level correction rate falls from 54% to a 14–18% plateau and can serve as a gold-standard-free progress metric and stopping criterion.
- Residual errors concentrate in vernacular lemmatization gaps and genuinely ambiguous structures, with about one third adjudicable differently by a second annotator.
Strengths
Thorough baseline diagnosis with a quantitative account of why transfer fails
Five UD Latin treebank models evaluated under CoNLL-2018 metrics (Table 2) and re-checked on Stanza 1.14 (Table 3); the corpus comparison quantifies the gap (mean sentence length 6.87 vs 11.0–27.4; nominal:verbal ratio 13.7:1 vs 1.7–3.4:1; §4.7), and §4.6 shows genre/period proximity fails as a transfer predictor.
Unusual transparency about failures and confounds
The early UPoS dip is traced to adposition collapse (98.1% baseline → 3.7% at s2 → 97.2% at s6, explaining ~85% of the dip, §6.3.1); the coincidence of the two largest gains with word-vector rebuilds is flagged as undisentangleable (§6.3.1); lemmatization is excluded from headline claims as lexicon-driven (§8); LAS at s8 (0.924) exceeding s9 (0.917) is reported.
Contamination and consistency controls rare in single-author projects
Actionable cost accounting
Complete open release
Weaknesses
Single-annotator evaluation loop
Small, fragile evaluation set
Stopping-criterion claim outruns the evidence
Key design confound acknowledged but unresolved
Missing comparison arms
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a real gap: NLP for specialized documentary corpora, here 160 late-medieval Marseille inventories (1258–1446, 47,391 words, 6,903 sentences) mixing Medieval Latin and Occitan. Three claims are staked out: (1) no readily available Latin UD model is usable on this material — best LAS 0.62 (LLCT), best MLAS 0.24 (UDante), Table 2 — and transfer does not track genre or period proximity (§4.6); (2) a human-in-the-loop bootstrapping loop, in which expert corrections of model output *are* the annotation, reaches research-grade accuracy (UPoS 0.80→0.98, LAS 0.48→0.92, MLAS 0.82) in 33.2 hours over 1,804 sentences, beating all five baselines with ~97% fewer training tokens than ITTB (Table 8); (3) the per-round token-correction rate (54% → 14–18% plateau) is an operational progress metric and stopping criterion. Claim (3) is the weakest link: the experiment itself stopped for subjective reasons (§5.9) and §6.4.2 explicitly warns against generalizing a stopping rule, putting the abstract at odds with the body.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How long does it take to build a usable parser for a low-resource historical corpus?
In this study, 33.2 hours of expert correction across 1,804 sentences lifted a Medieval Latin parser from LAS 0.48 to 0.92; the full correction pass averaged 1.11 minutes per sentence (falling from 2.57 to 0.50), covering POS, morphology, lemma, head and dependency relation.
What does 'correction as annotation' mean?
Instead of annotating text from scratch, an existing model pre-annotates each batch, a human expert corrects the output, and the corrected data trains the next model; the corrections themselves become the training data, so annotation effort doubles as model improvement.
Is the correction rate a reliable stopping criterion for annotation projects?
The paper proposes it (effort plateaued at 14–18% of tokens), but the evidence is weak: annotation actually stopped for subjective reasons, the plateau fluctuates rather than steadily decreasing, and the author warns against deriving a general stopping rule from this single run.
Are the corpus, models and code from this paper publicly available?
Yes — the 1,804 annotated sentences, the 200-sentence gold standard, the domain lexicon, and all nine trained models are released under open terms (GitHub repository plus a Zenodo archive), and tables/figures regenerate from the released data by one command.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The embedded text is a sampled excerpt with three marked '[...sampling gap...]' sections; outcomes of the declared significance tests and parts of §§5.1–5.3, 5.9–6.2 and 7.2–7.4 were not visible, so no judgment is made about content that may lie in those gaps.
- Figures 1–6 appear only as captions; their visual content was not part of the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.