Inspicio: Open-Vocabulary, LLM-Based Sense Retrieval for Historical Languages
cs.CL · cs.AI
Quick verdict
Zero-shot English-pivot pipeline links historical-language tokens to OEWN synsets; 96% Recall@50, 6×6 grid, but no significance tests or defined dev split.
Summary
Inspicio is a zero-shot retrieval pipeline that links tokens in Latin, Ancient Greek, and diachronic Italian to synsets of the Open English WordNet without any source-language sense inventory or word-to-sense mapping. For each token (given with gold lemma, sentence, and language tag), an instruction-tuned LLM produces literal and natural English translations, one to three dictionary-style definitions, and up to five candidate English lemmas. A hybrid retriever combines dense similarity between definitions and OEWN synset documents, sparse lemma lookup, definition-rank-decay scoring, lemma boosting (gamma=0.8, beta=0.65), and MMR re-ranking (lambda=0.8), returning top-50 synsets with logged intermediate artifacts for auditability. Evaluation spans a 6x6 grid of LLMs and sentence embedders on a new 150-token Latin/Ancient Greek perception-verb set (kappa=0.895), the PREMOVE motion-verb dataset, and 100 diachronic Italian tokens (kappa=0.914). The best configuration (DeepSeek V4 Pro + KaLM-Embedding-Gemma3-12B) reaches 96% Recall@50 on perception verbs, 81.65% on PREMOVE, and 91% on Italian. Ablations show the translation stage (-4 points) and lemma boost (-2 points) contribute, while removing MMR or rank decay does not lower headline recall. Main limitations: hyperparameters were tuned on an undefined development subsample, all results are single stochastic runs without significance testing, the headline metric is Recall@50 rather than top-1 accuracy, and no external baseline is compared.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Inspicio links tokens in context to OEWN synsets with no source-language sense inventory or word-to-sense mapping, in a fully zero-shot regime (Sections 1, 3).
- The best configuration, DeepSeek V4 Pro with KaLM-Embedding-Gemma3-12B, reaches 96% Recall@50 on a new 150-token Latin/Ancient Greek perception-verb set (Section 5, Table 1).
- The hybrid design contributes measurably: removing the translation stage drops Recall@50 to 92% and removing the lemma boost to 94% (Table 2 ablations).
- The same configuration transfers out-of-domain and cross-lingually: 81.65% Recall@50 on PREMOVE and 91% on diachronic Italian (Table 2).
- Every prediction is auditable because translations, definitions, lemmas, and per-definition score contributions are logged per token (Section 3.7).
Strengths
Precise open-vocabulary problem framing
Sections 1-2.3 position the work exactly at the gap: classical WSD assumes a source inventory and word-to-sense mapping, and Word Sense Linking (Bejgu et al., 2024) degrades when that mapping is incomplete; Inspicio drops both assumptions and routes through OEWN, matching the pivot used by the Latin and Ancient Greek WordNet bootstrapping projects.
New manually annotated bilingual resource with reported reliability
Section 4.1 contributes a 150-token Latin/Ancient Greek perception-verb set (72+78), balanced across periods and genres via the PREMOVE Base Corpus, double-annotated by trained linguists with kappa=0.895; the Italian set adds kappa=0.914. Code, prompts, and evaluation data are stated to be released (Section 1, anonymous GitHub link).
Exhaustive 6x6 model grid with actionable findings
Unusually candid ablation reporting
Auditability by design and concrete error analysis
Weaknesses
Undefined development/test boundary
No statistical treatment of a stochastic pipeline
Headline metric is candidate recall, not disambiguation
No external baselines
Weakened OOD/cross-lingual claims and unaccounted inputs
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Inspicio (arXiv 2609.00998)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Inspicio pipeline and what problem does it solve?
Inspicio (arXiv 2609.00998) is a zero-shot retrieval system that links words in Latin, Ancient Greek, and Italian sentences to sense entries (synsets) of the Open English WordNet. Word-sense disambiguation normally requires a sense inventory in the language being analyzed, which historical and low-resource languages lack. Inspicio sidesteps this by routing through English: an LLM translates the sentence and proposes dictionary definitions plus English lemmas, which are matched against WordNet synsets using dense embeddings, sparse lemma lookup, and MMR diversification — no source-language inventory needed.
How accurate is Inspicio?
In the paper's best configuration (DeepSeek V4 Pro with the KaLM 12B sentence embedding model), 96% of gold WordNet synsets appear within the top-50 retrieved candidates (Recall@50) on a 150-token Latin/Ancient Greek perception-verb test set, 81.65% on the PREMOVE motion-verb dataset, and 91% on 100 diachronic Italian tokens. These are candidate-pool recall figures, not final top-1 disambiguation accuracies, and they come from single runs without statistical significance testing.
Is the Inspicio code and data available?
Yes. The paper states that code, prompts, and evaluation data are released on GitHub via an anonymous.4open.science link. This includes the new 150-token perception-verb annotation set with gold Open English WordNet synsets and the full prompt templates (paper Appendix A). Caveat: the pipeline depends on specific current-generation API LLMs (e.g., DeepSeek V4 Pro, Kimi K2.6), so exact reproduction may drift as those models are updated.
Does Inspicio actually pick the correct word sense?
Not by itself. Inspicio is a high-recall candidate generator: its 96% headline means the correct synset is somewhere in the top 50 results, not necessarily ranked first. Numeric top-1 performance is only shown as heatmap figures, and the authors explicitly leave an LLM-based reranking stage that selects the final sense as future work (Section 6.2). For lexicographers, the logged intermediate outputs make each prediction inspectable and correctable.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's per-cell Recall@1/@10/@20 values exist only as heatmap images; the extracted text provides numeric Recall@50 (Table 1) and Recall@50/@20 (Table 2), so top-of-list performance could not be assessed numerically.
- Figure 1 (architecture diagram) is an image; the methodology assessment relies on the complete Section 3 prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.