From UNDRR Reports to Event Records: Schema-Constrained LLM Extraction of Georeferenced Disasters
Camilla Andreozzi, Phuong-Anh Nguyen-Le, Zhijing Jin, Revati Mani
cs.CL · cs.AI · cs.IR
Quick verdict
LLM pipeline extracts 3,572 georeferenced disaster records from UNDRR reports (86.0 F1); careful work, but pooled scoring leaves event linkage unverified.
Summary
The paper presents an LLM pipeline that converts narrative documents from PreventionWeb, UNDRR's disaster-risk-reduction knowledge hub, into candidate georeferenced event records using a controlled hazard vocabulary, a fixed schema via OpenAI Structured Outputs, and Nominatim/OpenStreetMap geocoding. Applied to 10,000 documents (12,000-character input cap), it retained 3,572 records from 1,913 documents, resolved 81.2% of 5,463 location mentions, and mapped 50 hazard strings to nine families. Evaluation used a 217-document stratified reference set (171 positive) annotated by the authors: GPT-5, the production configuration, reached 86.0% pooled attribute F1 (precision 84.2, recall 87.9) versus 44.2% for a spaCy-gazetteer baseline, with 95.1% event-detection F1; GPT-5.4 scored the highest pooled F1 (86.6) among ten LLMs. A 75-mention geocoding audit found 77.3% correct place and extent and 21.3% correct place but wrong extent. The central limitation, stated prominently by the authors, is that pooled document-level scoring cannot assess assignment of attributes to individual events; author-only annotation without inter-annotator agreement and an event stratum drawn from production output weaken ground-truth independence. Proprietary models via OpenRouter limit exact reproduction; prompts, schema, and outputs are promised upon publication.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A schema-constrained LLM pipeline can convert narrative UNDRR documents into auditable, georeferenced candidate event records at corpus scale (3,572 records from 10,000 documents across 24 hazard types).
- GPT-5 reaches 86.0% pooled document-level attribute F1 on 171 event-bearing windows, 41.8 points above the spaCy-gazetteer baseline, with 95.1% detection F1 on the 217-window set.
- The pipeline resolves 81.2% of 5,463 location mentions to OpenStreetMap geometries, and 77.3% of audited resolutions are correct in both place and extent.
- Verbatim evidence occurrence (72.0% for GPT-5, 47.2% for GPT-5.4) measures textual traceability but does not establish attribute support.
- Pooled document-level evaluation by design does not assess assignment of attributes to individual events, which remains an open problem for multi-event documents.
Strengths
Exceptional evaluation-scope honesty
The abstract itself states pooled scoring is 'without assessing their assignment to individual events'; §5 and Limitations repeat that 'no event extracted' means 'not assessed', structural compliance is not semantic support (§4.1), and geocoding success is not correctness (§7.2). This candor is rare in LLM-extraction papers.
Controlled ten-model comparison with raw counts
All ten LLMs and the NER baseline ran under identical prompt, schema, hazard vocabulary, 12,000-character window and geocoder (§5); Table 11 publishes full TP/FP/FN/TN counts per field, and the baseline's 100% verbatim-evidence rate is correctly framed as a copy-by-construction control.
Empirical grounding of the geocoding stage
Actionable production failure taxonomy
Full methodological disclosure
Weaknesses
Evaluation cannot answer the core question
Weak ground-truth independence
Sampling circularity
Quality-blind prompt selection
No inferential statistics on model differences
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: From UNDRR Reports to Event Records (arXiv:2609.23853)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper actually build and achieve?
It builds an LLM pipeline that reads narrative documents from PreventionWeb (UNDRR's disaster-risk-reduction knowledge hub) and outputs candidate disaster-event records with hazard type, event dates, affected locations, impacts and evidence snippets. On 10,000 documents it produced 3,572 records from 1,913 documents and resolved 81.2% of location mentions to OpenStreetMap geometries; on annotated test documents GPT-5 scored 86.0% pooled attribute F1 versus 44.2% for a rule-based spaCy-gazetteer baseline.
Can I use the extracted disaster dataset as a reliable event database?
Only as candidates for human review. The scores are document-level pooled metrics that do not check which attributes belong to which event, records are not deduplicated across documents describing the same occurrence, evidence snippets may be paraphrases, and a geocoding audit found 21.3% of resolved locations had the right place but the wrong extent. The authors explicitly state the outputs require source-based verification before use.
Which language model performed best in the comparison?
Ten LLMs were compared under the same prompt and schema. GPT-5.4 had the highest pooled F1 (86.6%), while GPT-5 (the production model) scored 86.0% with the best complete-match rate (44.4%) and much higher verbatim evidence occurrence (72.0% vs 47.2%). The authors caution that differences under one F1 point should not be read as a ranking.
Is the code or data available?
At review time, no artifacts were released: the paper states that prompts, schema, code and derived outputs will be released upon publication. The prompts and output schema are reproduced in the appendices, and the corpus source (PreventionWeb) is public, but the models are proprietary via OpenRouter, so exact reruns are not guaranteed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 3 (workflow example with map, temporal bar charts) are only recoverable from captions; the map judgment and exact bar values could not be visually verified.
- All appendices A–H and references appear complete in the extracted text, so truncation risk is low; Table 11's column alignment was cross-checked against Table 1.
- The review relies on the authors' description of annotation procedure; no annotation files were available to verify label quality.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.