Custom Named Entity Recognition and Topic Classification for Global Health Publications
Genis Skura, Antoine Geissbühler, Jean-Luc Falcone
cs.AI · cs.CL
Quick verdict
Geneva master's thesis builds a spaCy NER + topic-classification pipeline for global health; practical and candid, but evaluations use single-annotator data.
Summary
This University of Geneva master's thesis (2023) builds NLP components to annotate global health publications in the Implementome knowledge base under scarce-annotation, low-compute constraints. First, skip-gram word2vec models trained on progressively larger in-house corpora (305k, 910k, 8.9M tokens) are compared with BioWordVec for tag discovery; the specialized models win on author-built odd-one-out tests (0.58-0.66 versus 0.487) and perplexity, though test material came from the models' own corpora, a bias the author acknowledges. Second, on 1,000 sentences annotated by the author, lenient token-level IOB scoring gives the RoBERTa-based spaCy transformer 0.80 micro-F1 versus 0.65-0.69 for CNN models, at 82 versus 5-6 seconds; the transformer is discarded, CNNs are fine-tuned on Prodigy annotations, and a disease recognizer trained on the NCBI Disease Corpus (81.33% test F1) completes a ~7-second-per-document pipeline with frequency filtering and MeSH enrichment. Third, on 1,000 self-written sentences over 50 topics, BART-large-MNLI zero-shot reaches 95.2% single-label and 88% multi-label accuracy versus 59%/32% for a MiniLM few-shot baseline, but costs ~400 seconds per document. Limitations: single-annotator gold data, handcrafted test sets, lenient scoring, no significance testing, a subjective ~91% pipeline accuracy estimate, and no released code or data.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Broader or externally pretrained corpora do not necessarily yield more useful domain-specific tag associations than small specialized corpora for global health tag discovery (§2.4.3, §5).
- A RoBERTa-based spaCy transformer reaches 0.80 lenient micro-F1 versus 0.65-0.69 for CNN models on 1,000 annotated sentences, but at 82s versus 5-6s per 1,000 sentences (§3.2.3).
- Fine-tuned CNN spaCy models plus a from-scratch disease recognizer (81.33% test F1 on NCBI Disease) yield a ~7s/document pipeline the author judges ≈91% accurate (§3.3-3.5).
- BART-large-MNLI zero-shot (95.2% single-label, 88% multi-label) clearly beats MiniLM few-shot (59%/32%) on handcrafted 50-topic tests, but at ~400s/document it is impractical for the deployment setting (§4.3).
Strengths
End-to-end deployable pipeline with documented engineering
§3.5 details pdfminer-vs-PyPDF2 benchmarking, reference stripping via string.find, Punkt sentence tokenization, frequency-threshold filtering (entities kept at >=1/3 of the max count, §3.5.3) and MeSH enrichment; ~7s per 10-15 page publication makes it genuinely usable.
Transparent speed/accuracy trade-off analysis
The §3.2.3 table (0.65/0.69/0.69/0.80 micro-F1 at 5.1-6.0s vs 81.8s per 1,000 sentences) is used to justify deploying fine-tuned CNNs instead of the transformer, explicitly tied to LMIC connectivity and model-size constraints (§3.1) rather than chasing SOTA.
Honest acknowledgment of evaluation biases
Externally anchored disease recognizer
Well-reasoned hyperparameter and preprocessing choices
Weaknesses
Single-annotator gold standard with no agreement statistics
Cross-protocol score comparisons
Handcrafted, single-author test sets for classification
No statistical rigor anywhere
Missing baselines and no release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a June 2023 University of Geneva master's thesis (arXiv 2609.24625) by Genis Skura, motivated by incomplete manual annotation in the Implementome global-health knowledge base built with WHO partners on neo4j (§1.1). It spans three NLP studies — word2vec tag discovery (Ch. 2), named entity recognition (Ch. 3), and multi-label topic classification (Ch. 4) — unified by a deployment goal: a fast, compact annotation pipeline for low-resource settings, with LMIC connectivity and disk constraints explicitly cited (§3.1). The headline claims: specialized word2vec beats broad coverage for tag usefulness; transformer NER reaches 0.80 lenient micro-F1 versus 0.65-0.69 for CNNs but costs 82s versus 5-6s per 1,000 sentences; a deployed CNN pipeline with a disease recognizer (81.33% test F1 on NCBI Disease) supports document indexing at ~7s per publication; and BART-MNLI zero-shot beats MiniLM few-shot 95.2% versus 59% single-label accuracy. Within their narrow, self-built evaluation settings the experiments support these claims, but those settings limit external validity considerably.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy does this paper's NER pipeline achieve?
On 1,000 author-annotated sentences with lenient token-level IOB scoring, the RoBERTa-based spaCy transformer scored 0.80 micro-F1 versus 0.65-0.69 for CNN models. The deployed fine-tuned CNN reached about 0.8 under strict scoring, the disease recognizer scored 81.33% test F1 on the NCBI Disease Corpus, and the full document pipeline was judged roughly 91% accurate by the author's own pragmatic (subjective) criterion.
Why didn't the author use the more accurate transformer model in production?
The transformer processed 1,000 sentences in about 82 seconds versus 5-6 seconds for CNN models and needs GPU resources. The thesis prioritizes speed, small model size, and accessibility for low-connectivity, low-resource global health settings, so it fine-tuned the faster CNN models instead.
Is code or data available for this paper?
No. The paper describes no released code, corpora, or annotation sets, and its annotation tool (Prodigy) is commercial. The only fully public component is the NCBI Disease Corpus, so the NCBI disease-recognizer result (81.33% F1) is the main number others could attempt to replicate; most comparisons cannot be reproduced from the paper alone.
Which topic classifier performed better, few-shot or zero-shot?
BART-large-MNLI zero-shot clearly won on the author's 1,000 handcrafted test sentences over 50 topics: 95.2% single-label and 88% multi-label accuracy versus 59% and 32% for the MiniLM few-shot classifier. However, zero-shot inference took about 2 seconds per sentence (~400 seconds per document on a GTX 1650 Ti), which the author deemed impractical for deployment.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several sections were elided in the extracted text (parts of §2.2, §2.4.3, the §3.2.3 per-label result tables, §3.5.4-3.5.5); judgments about those parts rely on surrounding prose.
- All figures, tables, and benchmark screenshots could not be analyzed; per-label metrics and exact fine-tuned scores are known only from text descriptions.
- This is a June 2023 master's thesis posted to arXiv in 2026; it was not peer reviewed and reflects a 2022-2023 tooling landscape.
- The size and composition of the '≈91%' pipeline benchmark document set are not stated in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.