Dynamics of meaning: Towards the Evaluation of Diachronic Semantic Change in Sinhala
Nevidu Jayatilleke, Nisansa de Silva
cs.CL
Quick verdict
Promising LOO-based drift attribution for Sinhala, but evaluation is purely intrinsic, genre-confounded, and key metrics are internally inconsistent.
Summary
The paper presents a multi-stage framework for detecting lexical semantic change in Sinhala, a low-resource language, using the authors' SiDiaC corpus (dated subset: 59 books, 67,005 words; six usable century slices between the 13th and 20th centuries). SiDiaC-v.2.5 adds lemmatisation (SinMorphy) and hybrid POS tagging. Century-specific Word2Vec and fine-tuned FastText embeddings are aligned via similarity-matrix alignment (SMA) and two Orthogonal Procrustes (OP) variants; across 15 alignments and 210 consistent words/261 lemmas, OP shows higher mean similarity (e.g., FastText Neural-OP 81.04%) and fewer flagged dips, so the authors prefer OP and FastText. A μ−0.8σ dynamic threshold, checked against the Gaussian tail expectation (flag rates 16.19–20.57% vs 21.19% theoretical), selects drift candidates for qualitative neighbour inspection. Second, contextualised embeddings from a continually pre-trained Llama-3.1-8B (best PPL 2.18) support a Leave-One-Out 'Bidirectional Semantic Impact Pruning' that attributes centroid drift to specific high-impact sentences and computes a 'persistent shift'. Centroid distances are small (Table 18 range 0.0154–0.0340), drift reductions mostly 11–18%, and case studies (guṇa, mārga, vahanse) attribute 27–46% reductions to register shifts. Key limitations: no gold standard or at-scale human evaluation (validation is the authors' own 'small-scale manual validation'), a circular threshold check, genre/topic confounds (one 13th-century work dominates the mārga case), the 8% headline reduction in §4.2 sitting oddly beside Table 18's implied ~14%, and no significance testing.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Orthogonal Procrustes alignment provides more stable neighbourhood tracking than Similarity Matrix Based Alignment for century-specific Sinhala Word2Vec/FastText embeddings (Tables 3, 11).
- Under extreme per-century data scarcity, fine-tuned FastText embeddings are more stable than freshly trained Word2Vec, though their high similarity may stem from a shared pretrained base.
- Semantic drift measured in continually pre-trained Llama-3.1-8B representations is unevenly distributed: of 112,945 lemma instances, 21,738 drifted, and removing high-impact 'informer' sentences reduces measured drift by roughly 8–18% depending on aggregation.
- A dynamic μ−0.8σ threshold flags a stable 16.19–20.57% of the consistent vocabulary across four independent alignment pipelines, close to the Gaussian expectation of 21.19% (Table 10).
- Sinhala diachronic change in this corpus is better characterized as localized contextual innovation (e.g., guṇa expanding from royal-heroic to devotional and courtly-praise registers) than as uniform lexical replacement.
Strengths
First substantial computational diachronic semantic analysis of Sinhala, with public resources
Builds SiDiaC-v.2.5 (lemma + POS annotations via SinMorphy and a hybrid TnT/Sinling pipeline) on the dated subset of SiDiaC-v.2.0 (59 books, 67,005 words, 13th–20th centuries), released on GitHub and Hugging Face (footnotes 5, 6, 8) — a genuine contribution to an underserved language (Class 2 per Ranathunga & de Silva 2022, Table 4).
Systematic multi-pipeline comparison with consistently reported numbers
Six model-to-mapping combinations × 15 alignments are reported end-to-end: mean similarities (Table 3, e.g., FastText Neural-OP 81.04% vs Word2Vec SMA 30.23%), per-century-pair flag counts (Table 11), and all 30 ordered permutations for the LOO stage (Table 18), enabling scrutiny.
Novel and interpretable drift-attribution method
Candid limitations and failure reporting
Weaknesses
No extrinsic or at-scale human evaluation
Circular threshold validation
Genre/topic confound with 'semantic change'
Internal numeric inconsistencies and no uncertainty quantification
FastText 'stability' may be partly an artifact
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tackles diachronic lexical semantic change in Sinhala from the 13th to 20th century, using the dated subset of SiDiaC-v.2.0 (59 books, 67,005 words) enhanced into SiDiaC-v.2.5 with SinMorphy lemmas and a hybrid TnT+Sinling POS pipeline (§3.1, App. C). After excluding the sparse 5th, 12th, and 16th centuries, six century slices remain (Fig. 1: e.g., 13th c. 13,617 words; 18th c. 4,707). The authors claim: (i) Orthogonal Procrustes (OP) alignment is more stable than Similarity Matrix Based Alignment (SMA); (ii) FastText is more stable than Word2Vec under scarcity; and (iii) their novel LOO-based "Bidirectional Semantic Impact Pruning" shows drift concentrates in a minority of high-impact usages. Claims are honestly scoped as a "preliminary framework," and the Limitations section candidly admits data scarcity, historically untested SinMorphy, and modern-biased POS tagging.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SiDiaC-v.2.5?
An enhanced version of the Sinhala Diachronic Corpus released with this paper: the dated subset of SiDiaC-v.2.0 (59 books, 67,005 words spanning the 5th–20th centuries) annotated with lemmas (via the SinMorphy morphological analyser) and POS tags (via a hybrid TnT + Sinling pipeline). The authors make it publicly available on Hugging Face, with the framework code and prior corpus versions on GitHub.
What is the Leave-One-Out (LOO) 'Bidirectional Semantic Impact Pruning' method in this paper?
A diagnostic that, for each lemma, measures how much removing one sentence changes the cosine distance between the embedding centroids of two centuries. Sentences whose removal most reduces the distance (within 50% of the maximum effect) are flagged as 'informers' of semantic drift; removing them permanently yields a 'persistent shift', and the percentage difference between the original and reduced distances is the drift reduction attributable to those specific usages.
What did the paper find about semantic change in Sinhala?
That measured drift between centuries is small (centroid cosine distances of roughly 0.015–0.034) and concentrated in a minority of high-impact contextual usages rather than spread uniformly: about 19% of lemma instances (21,738 of 112,945) were flagged as drifted, and removing informer sentences reduced measured drift by roughly 7–18% depending on the century pair, with case studies showing register expansion (e.g., 'virtue' widening from royal praise to devotional and courtly-poetic praise).
Which alignment method worked best for low-resource diachronic embeddings in this study?
Orthogonal Procrustes (OP) alignment outperformed Similarity Matrix Based Alignment: mean cross-century similarity of consistent words reached 81–82% under OP with fine-tuned FastText versus about 30–56% for SMA, and OP flagged fewer spurious 'similarity dips'. The authors caution, however, that FastText's high stability may partly reflect fine-tuning from a shared modern pretrained model rather than genuine temporal closeness.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with '[...sampling gap...]' breaks: the formal Orthogonal Procrustes description in §3.3.2, part of Appendix E's alignment-error analysis, Appendix A/B, and mid-alphabet reference entries were skipped; judgments about those parts rely only on surviving summary text.
- Figures 1–3 were extracted as garbled/glued text; Figure 1's numeric labels were readable, but plot geometry and figure-level details could not be verified.
- Claims that no significance testing or no at-scale human evaluation exists are based on all text visible in this excerpt; equivalent content could in principle sit inside a sampling gap.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.