LLM Agents as Computational Typologists
Changbing Yang, Christopher Hammerly, Freda Shi, Jian Zhu
cs.CL
Quick verdict
LLM agent codes typological features from grammars; evaluation is careful and honest, but 3 of 5 models beat the majority-vote baseline in the best setting.
Summary
The paper introduces AutoTypologist, an LLM agent that performs evidence-grounded linguistic typology over 25 open-source reference grammars covering 14 language families. Digitized grammars are chunked (≤1,000 words, with GPT5.4-mini-generated summaries) and paired with automatically extracted, manually reviewed interlinear glossed text (IGT). A ReAct-style agent plans an investigation, retrieves grammar sections, runs IGT analyses, records cited claims in an evidence graph, and passes conclusions through a same-model auditor. Two tasks are evaluated. In Typological Feature Coding (195 Grambank features, 25 languages, three information conditions), grammar access works best (Qwen3.6-27B: 70.7 weighted / 59.2 macro F1, versus 62.5/41.5 for a majority-vote baseline); IGT-only is clearly harder, and Grammar+IGT is not significantly better than Grammar-only (language-clustered bootstrap, Holm-corrected). Even in the best condition only 3 of 5 models beat the majority-vote baseline on weighted F1. In Typological Hypothesis Testing (10 Greenberg universals plus adversarial variants, 10 languages), the agent separates supporting evidence from counterexamples, but the task has no gold standard, and one traced run falsely refutes a true universal. Manual adjudication of 180 errors yields five failure modes. Limitations include the contested status of Grambank as gold standard, possible pretraining exposure to the public grammars, and a self-audit ablation confined to a single language.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AutoTypologist is the first LLM agent for evidence-grounded typological analysis, combining grammar retrieval, IGT analysis, and ReAct-style reasoning with citation-forced evidence graphs.
- Under Grambank feature coding, reference-grammar prose enables the best agreement with experts (Qwen3.6-27B: 70.7 weighted / 59.2 macro F1), IGT-only remains hard, and adding IGT to grammar is not significantly better.
- The agent can test Greenberg universals by assigning per-language supporting/counter/irrelevant/insufficient judgments and discriminates adversarial from standard hypotheses.
- A released corpus of 25 open-license reference grammars (14 families) with structured chunks, IGT, and metadata enables evaluation of evidence-grounded typological reasoning.
Strengths
Statistically grounded headline findings
Condition comparisons use paired language-clustered bootstrap (10,000 resamples) with 95% CIs and Holm-corrected p-values (Appendix L, Table 16), properly establishing both that Grammar beats IGT-only for 4/5 models on weighted F1 and that Grammar+IGT never significantly beats Grammar-only.
Rare depth in manual error analysis
180 prediction-gold mismatches (10 per language x condition, 6 languages x 3 conditions, pooled over three models) adjudicated against original grammar texts (Section 8), yielding five concrete failure modes with named examples, e.g., Papuan Malay GB520 (adverbs misread as auxiliaries) and Chacobo GB092 (ergative clitic =wa misread as verbal agreement).
Inspectability by construction
Honest negative results and hard baselines
Reproducibility hygiene
Weaknesses
Missing non-agentic baseline
Modest absolute performance and weak calibration
Hypothesis testing lacks ground truth
Unaddressed contamination for feature coding
Thin ablation scope and a shared-index channel
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper presents AutoTypologist, described as the first LLM agent for evidence-grounded typological analysis, together with two formalized tasks - TYPOLOGICAL FEATURE CODING (assigning Grambank feature values from a reference grammar) and TYPOLOGICAL HYPOTHESIS TESTING (evaluating Greenberg-style universals by aggregating per-language evidence) - and a released corpus of 25 openly licensed reference grammars across 14 language families with structured chunks, extracted interlinear glossed text (IGT), and metadata. Headline claims: grammar prose enables coding with high expert agreement; IGT-only inference remains difficult; the agent can identify supporting evidence and counterexamples for universals. All three are largely supported, with the qualification that 'high agreement' means at most 70.7 weighted F1 against Grambank's own codings, under a 62.5 majority-vote baseline.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AutoTypologist?
An LLM agent presented in this paper that analyzes reference grammars of mostly low-resource languages: it plans what evidence it needs, retrieves relevant grammar sections, analyzes interlinear glossed example sentences, and outputs typological feature judgments (such as Grambank codes) with cited evidence and confidence scores, using a ReAct-style reasoning loop plus a self-audit step.
Can LLMs do linguistic typology as well as experts?
Not yet. With full grammar text, the best model agrees with expert Grambank codings at 70.7% weighted F1 (a majority-class baseline reaches 62.5%), working only from annotated examples is much weaker, and the paper's own traces show confident mistakes (e.g., refuting a true universal), so expert validation remains necessary; fieldwork cannot be replaced.
What data does the paper use?
25 openly licensed reference grammars covering 14 language families (e.g., Aguaruna, Japhug, Yakkha, Pite Saami), digitized, chunked, and paired with manually reviewed interlinear glossed text. Agent codings are evaluated against expert Grambank annotations, and 10 Greenberg universals plus deliberately falsified adversarial versions are tested over 10 languages.
Does adding example sentences (IGT) to grammar text improve LLM typological coding?
No - a statistically tested finding of the paper is that Grammar+IGT is not significantly better than grammar text alone for any of the five evaluated models, and it can even hurt (e.g., -7.7 weighted F1 for Qwen3.5-9B), suggesting the models rely more on descriptive prose than on bottom-up induction from examples.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 6 states that some grammars are 'highlighted in red' as not used; color coding is unrecoverable from text extraction, so which duplicate source was excluded for languages with two listed sources (e.g., Karbi, Moloko) cannot be identified from the provided text.
- Figure 2 (per-domain weighted F1) and Figure 3 (token-vs-F1 scatter) are described qualitatively in the text; their plotted data-point values are not recoverable from the extraction, so domain-level numbers cannot be independently checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.