Levent Bulut

—

Unclaimed · AI-structured profileORCID OpenAlex

Computational narratology researcher auditing LLM-as-judge bias and annotation reliability for narrative text

papers here

reviewed by Evolve

first-author

corresponding

—

works · OpenAlex

h-index · OpenAlex

Bibliometric figures come from OpenAlex (2026-10-15) and cover the author's entire public record, not just the papers listed here. Reproducibility signal comes from Evolve's own reviews.

Research profile

Output—

Output is marked as missing: Evolve's pipeline has not verified this researcher's full publication count. This axis would measure cumulative works, output trajectory over time, and venue spread; none of these can currently be scored, so no estimate is offered.

Impact—

Impact is marked as missing: no verified citation aggregates are available for scoring. This axis would measure citation counts, citing works, and influence on later research; the only recorded signals are an h-index of 1 and an i10-index of 0.

Focus0/100

Focus scores 0/100, reflecting an evenly split field distribution (0.5 computational linguistics / NLP annotation methodology, 0.5 LLM evaluation) and topic mass spread across five clusters: Narrative Theory and Analysis (21), Literary Theory and Cultural Hermeneutics (8), Aesthetic Perception and Analysis (6), Computational and Text Analysis Methods (4), and Heart Rate Variability and Autonomic Control (3). The first three are adjacent narratology themes; the last sits outside the reviewed NLP scope.

Collaboration0/100

Collaboration scores 0/100: OpenAlex records 0 distinct coauthors and the frequent-coauthor list is empty. Both Evolve-reviewed 2026 papers are single-author outputs with the researcher as first and corresponding author. This is a genuinely solo working pattern so far, though with only two reviewed papers and a one-year record, it may reflect early-career stage rather than a fixed preference.

Activity93/100

Activity scores 93/100, the strongest signal in this profile: countsByYear records 23 works in 2026, and recent3yWorks is 23 — meaning the entire recorded recent output lands in a single year. Both catalog papers appeared on arXiv in 2026. This concentration indicates an intense current publishing pace, though the one-year window makes longer-term pacing unverifiable.

Reproducibility38/100

Reproducibility scores 38/100 (reproMean 38), computed from exactly 2 reviewed papers. Readiness differs sharply: the Turkish-corpus inter-rater reliability audit is rated medium — a transparent audit, though single human raters and severe class imbalance cap certainty — while the summarization-bias framework is rated low, being a pre-registered protocol with zero new data. The construct is sound but evidence density is limited.

Research summary

Levent Bulut is an early-stage computational linguistics researcher whose profile is name-resolved from public bibliographic data (OpenAlex identity, ORCID 0009-0007-7500-2261); no institutional affiliation is on record. The Evolve catalog contains two 2026 arXiv papers, both first-authored and corresponding-authored, sitting at the intersection of LLM evaluation and NLP annotation methodology — the two fields split the recorded field distribution evenly (0.5/0.5). One paper proposes a conceptual framework and a registered test protocol for summarization bias, arguing that large language models directionally collapse shown-mode prose into told-mode labels, with implications for LLM-as-judge evaluation and reward models. The other is a transparent inter-rater reliability audit of LLM and rule-based annotation for inferential narrative features on a Turkish corpus, reporting chance-level machine agreement under single-rater, class-imbalanced conditions. OpenAlex activity metrics are high: 23 works recorded in 2026 (recent3yWorks: 23), yielding an activity score of 93, while distinct coauthors count zero. Topic mass concentrates on Narrative Theory and Analysis (21), Literary Theory and Cultural Hermeneutics (8), and Aesthetic Perception and Analysis (6), with smaller clusters in Computational and Text Analysis Methods (4) and Heart Rate Variability and Autonomic Control (3). Bibliometric impact is thin (h-index 1, i10-index 0), the output and impact axes are marked missing, and the reproducibility mean across the two reviewed papers is 38.

Collaboration pattern

The recorded collaboration pattern is strictly solo: OpenAlex lists 0 distinct coauthors, the frequent-coauthor list is empty, and both Evolve-reviewed papers are single-author works with Bulut as first and corresponding author. The 0/100 collaboration score follows directly from this. Given that the verifiable record spans essentially one publication year (2026), the solo pattern should be read as an early-career snapshot rather than an established long-term collaboration style.

Research keywords

LLM evaluationLLM-as-judgesummarization biasreward modelsLLM-as-annotatorinter-rater reliabilityCohen's kappaannotation qualityclass imbalancecomputational narratology

Highlights

Chance-level machine agreement on inferential narrative features
First-authored reliability audit of LLM and rule-based annotation on a Turkish corpus: a transparent inter-rater analysis (Cohen's kappa) finds machine agreement near chance on inferential narrative features, with single human raters and severe class imbalance capping certainty. Evolve rates its reproducibility readiness medium.
A registered test protocol for summarization bias in LLMs
Conceptual framework arguing that large language models directionally collapse shown-mode prose into told-mode labels, paired with a pre-registered, falsifiable test protocol relevant to LLM-as-judge evaluation and reward models. Evolve praises the protocol design but notes zero new data and only self-citations; reproducibility readiness is low.

Timeline

2026Publishes a pre-registered conceptual framework for summarization bias — LLMs' directional collapse of shown-mode prose into told-mode labels — on arXiv as first and corresponding author.
2026Releases a three-study inter-rater reliability audit of LLM and rule-based annotation for inferential narrative features on a Turkish corpus.
2026OpenAlex records 23 works within this single year (recent3yWorks: 23), the basis of the 93/100 activity score.

Papers in the Evolve catalog

2026-09-18first authorcorresponding
Summarization Bias: The Directional Collapse of Objective Projection into Told-Mode Labels in Large Language Models --- A Conceptual Framework and Registered Test Protocol
2026-09-12first authorcorresponding
Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus

About this profile

This profile is AI-structured by Evolve.Pub from public bibliographic data (OpenAlex) and Evolve's own paper reviews. It is not written or endorsed by the researcher unless the profile is claimed. Radar dimensions are computed from real bibliometrics with fixed formulas — raw figures are shown alongside; missing dimensions mean insufficient data, never zero. Errors? Claim the profile to correct or hide it.

Bibliometric metrics come from OpenAlex only (identity: openalex); institution and country are not on record, so this profile is name-resolved from public bibliographic data plus ORCID 0009-0007-7500-2261.
The output and impact axes are marked missing; the only impact signals available are h-index 1 and i10-index 0, so no citation-based scoring is offered.
OpenAlex topic data includes Heart Rate Variability and Autonomic Control (3 works), which lies outside the reviewed NLP scope and may reflect topic-assignment or name-disambiguation noise.
The reproducibility mean (38) rests on only 2 reviewed papers, and the activity signal covers a single year (2026, 23 works), limiting generalization.