Orfeas Menis Mastromichalakis

Research profile

Research summary

Orfeas Menis Mastromichalakis is a researcher at Instituto de Telecomunicações whose recorded country code is GR, working primarily on evaluation in natural language processing. His assigned field weight splits three ways: machine translation evaluation (0.5), AI safety for LLM agents (0.25), and NLP model interpretability (0.25). Indexed topics span Ethics and Social Impacts of AI (12), Explainable AI (12), NLP techniques (10), topic modeling (7), and semantic web and ontologies (5). The Evolve catalog includes four of his 2026 arXiv papers: a first-authored benchmark of gender bias in machine translation evaluation metrics across occupations, an empirical study of counterfactual self-explanations in LLMs, the AURA-Eval framework for acting under risk awareness in LLM agent trajectories, and the community-built Last Translation Benchmark. Bibliometric signals show a steep activity ramp — annual works rising from 2 in 2020 to 14 in 2026, with 26 works in the last three years (activity 96/100) — and broad collaboration, with 75 distinct coauthors (collaboration 86/100). Citation-based indicators remain early-career (h-index 4, i10-index 2), and Evolve's reproducibility assessment averages 55/100 across the four cataloged papers, each rated medium readiness. The profile is name-resolved from OpenAlex bibliographic data.

Research keywords

Highlights

• First-authored benchmark exposes gender bias in WMT26 MT evaluators

  Using an occupation-balanced GAMBIT+ subset, this first-authored study benchmarks WMT26 machine translation evaluators and finds a masculine preference in 126 of 131 statistically significant runs, with effects varying widely across metrics; Evolve calls it a rigorous study.

• AURA-Eval probes risk awareness in LLM agent trajectories

  This framework augments LLM agent tool-use trajectories to test acting under risk awareness, showing unsafe acts rise when no safe path exists; Evolve finds validation solid, but notes LLM-judge reliance and a single seed source temper the claims.

• Counterfactual self-explanations in LLMs improve with model scale

  This co-authored empirical study reports a ρ=0.87 correlation between flip rate and model scale for LLM counterfactual self-explanations; Evolve deems the trends credible but cautions that aggregate-level statistics temper the causal claim.

• Community-built Last Translation Benchmark separates frontier models from humans

  The benchmark combines 3,456 hard translation exceptions with rule-based verifiers to distinguish frontier models from human rankings; Evolve views it as solid despite reliance on unquantified LLM verifiers.

Timeline

1. 2026First-authored arXiv study benchmarks gender bias in machine translation evaluation metrics across occupations using the GAMBIT+ subset
2. 2026Co-authored empirical study of counterfactual self-explanations in LLMs, reporting a ρ=0.87 flip-rate correlation with model scale
3. 2026Co-authored AURA-Eval, an evaluation framework for acting under risk awareness in LLM agent trajectories
4. 2026Co-authored the community-built Last Translation Benchmark with 3,456 hard exceptions and rule-based verifiers

Papers in the Evolve catalog

• 2026-09-18first author

  Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations

• 2026-09-15

  An Empirical Study of Counterfactual Self-Explanations in LLMs

• 2026-09-06

  AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories

• 2026-09-03

  Last Translation Benchmark

Frequent co-authors