Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus
cs.CL
Quick verdict
Transparent reliability audit finds chance-level machine agreement on inferential Turkish narrative features, but single human raters and severe class imbalance cap certainty.
Summary
This empirical reliability report asks whether humans agree with the machine-generated annotation layer shipped with the Objective Projection corpus, a Turkish narrative dataset carrying rule-based flags for six craft features. Three studies are run: Study 1 (120 scenes) scores the rule-based detector against blind labels from the scheme's own author; Study 2 (100 disjoint scenes) scores the detector plus Gemini 2.5 Flash and Grok against an independent non-expert rater whose labels were locked before any machine ran; Study 2b repeats the identical protocol with Claude Fable 5 (High) and ChatGPT 5.5. On materialized metaphor, the feature closest to the methodology's theoretical core, five machine labellers mark 0, 1, 40, 72 and 78 of 100 scenes positive against a human count of 9; Cohen's kappa is at or near chance (0.000-0.027) for five of six labellers across both human references and scene sets, while raw agreement (74.7-84.5%) is exposed as a class-imbalance artefact. Claude and ChatGPT achieve above-chance agreement on atmosphere contradiction, the only rule with a usable class distribution, partially correcting the author's earlier verdict. The key limitation is that each study has one human rater - one the scheme's author - and human-human agreement is never measured, so the author declines to choose between 'the feature resists machines' and 'the definition is not operational', naming a second rater and balanced evaluation sets as prerequisites.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On materialized metaphor, five machine labellers (rule detector, Gemini 2.5 Flash, Grok, Claude Fable 5, ChatGPT 5.5) return positive counts of 0, 1, 40, 72 and 78 out of 100 scenes against a human count of 9, showing they are not applying the same rule.
- Cohen's kappa is at or indistinguishable from chance (0.000-0.027) for five of six machine labellers on materialized metaphor across both human references and both disjoint scene sets, with Gemini's 0.185 as the single exception resting on one positive judgement.
- Raw agreement of 74.7-84.5% is a class-imbalance artefact rather than evidence of competence, and ranking systems by it ranks them by their willingness to say absent.
- The shipped applied_rules annotation layer should not be treated as ground truth for the corpus's inferential features, under either candidate explanation.
- The paper deliberately does not decide between 'the feature is genuinely inferential' and 'the rule definition is not operational', because no second independent human rater exists to separate the two.
Strengths
Reflexive integrity unusual for a negative-result report
Three conflicts of interest are declared on page 1 (author designed the rules, sole rater in Study 1, Claude assisted the manuscript); the negative result is published against the author's own corpus; provenance gaps are disclosed (Gemini/Grok per-scene labels lost, two Table 2 cells derived); and the paper self-corrects an earlier internal record (Gemini materialized-metaphor positives ~2 corrected to 1, Section 7.3).
Arithmetic checks out and statistics are handled honestly
All confusion tables and kappas reconstruct exactly from reported counts and marginals (e.g. Claude materialized metaphor: TP 8, FP 70, FN 1 vs H+=9 yields kappa 0.027 and 29% raw agreement); undefined kappa is reported as n/a rather than 0; the prevalence problem is grounded in Feinstein & Cicchetti and Byrt et al. rather than asserted.
Procedures that guard against post-hoc bias
Reports the disconfirming within-paper result
Weaknesses
Single unvalidated human rater per study undermines the central claim
Two of five labellers lack per-scene data on the Study 2 material
Class imbalance makes most of the design uninformative
Uncontrolled model conditions and unverifiable run identities
Closed ecosystem
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does it mean that Cohen's kappa was near zero while raw agreement was high?
In this paper, machines and humans both said 'absent' on most scenes because five of the six features are rare, so percent agreement looked high (74.7-84.5%) even though their actual positive judgements rarely coincided. Kappa corrects for agreement expected by chance, and it landed at 0.000-0.027 for five of six labellers on materialized metaphor - meaning agreement no better than random once the imbalance is accounted for.
Can LLMs reliably detect 'materialized metaphor' (show-don't-tell) in Turkish text?
According to this study, no. Five systems - a rule-based detector, Gemini 2.5 Flash, Grok, Claude Fable 5 (High) and ChatGPT 5.5 - were asked the same question on the same 100 Turkish scenes. They marked 0, 1, 40, 72 and 78 scenes positive respectively, against a human count of 9, and agreement with the human was at chance level. The author cannot yet tell whether the feature is inherently hard to detect or the rule's definition is too vague; a second human rater would be needed to decide.
Should I use the applied_rules annotations in the Objective Projection dataset?
The paper's practical conclusion: not as ground truth for the four inferential features (materialized metaphor, micro-focus, temporal anchor, atmosphere contradiction). The surface features behaved well - simile detection agreed perfectly, and the emotion rule over-triggers (12 machine flags vs 2 human in Study 1). Machine flags for inferential features disagree wildly across systems and with a human reader.
How were the three studies designed?
Study 1: the rule-based detector versus blind labels from the scheme's own author on 120 scenes. Study 2: the detector plus Gemini 2.5 Flash and Grok versus an independent non-expert rater on 100 different scenes, with the human labels locked before any machine ran. Study 2b repeated the identical protocol with Claude Fable 5 (High) and ChatGPT 5.5. Each study has a single human reference, and human-to-human agreement was never measured.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract as embedded is truncated mid-sentence ('Two readings survive: that the fea...'); all numbered sections, tables and references are present.
- The Hugging Face evaluation/ directory that the paper's reproducibility claims rest on could not be inspected from the text; reproReadiness assumes the files are as described.
- Only mean/variance-free summary tables are available in the text; no figures exist to analyze, and per-scene label-level patterns cannot be checked.
- Model identities (e.g. 'Claude Fable 5 (High)', 'ChatGPT 5.5', rejected 'Gemini 3.6') are accepted as reported by the author and were not verifiable from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.