DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactions
Rem Hida, Masahiro Kaneko, Daisuke Oba, Danushka Bollegala, Naoaki Okazaki
cs.CL
Quick verdict
Response-conditioned multi-turn protocol reveals late, non-monotonic, re-emerging LLM bias; sound design but single generator-judge LLM and thin uncertainty.
Summary
The paper introduces DyMT-ESB, an evaluation protocol probing social bias in multi-turn LLM interactions where each user query is generated from the evolving dialogue history, anchored on a seed stereotype sampled from FairMT-Bench (240 seeds across age, appearance, disability, gender, race, religion). Unlike scripted benchmarks, follow-up turns adapt to the target model's responses, enabling variable-length dialogues. Using gpt-4o-mini as both query generator and LLM-as-a-Judge, the authors evaluate six target LLMs (Gemma3-12B, Qwen3-8B, Llama3.1-8B, Phi-4, Olmo3-7B, Gemini-2.5-Flash-Lite) against zero-shot and re-run FairMT-Bench baselines in 5- and 10-turn settings. Headline results: cumulative bias of 12.7-20.0% by turn 5 (Assessment framing 9.2-16.3%, Implicit 15.0-25.8%) arises in highly coherent dialogues (4.59-4.97 vs 3.03-3.50 for FairMT-Bench) with near-zero refusals (0.2-1.7%); model rankings shift between protocols; 7.92-11.25 additional percentage points of dialogues first show bias in turns 6-10; per-turn bias fluctuates and re-emerges (4.58-10.00% of dialogues); bias persists after post-hoc ShieldGemma filtering. Human validation on 125 turns by three authors yields Gwet's AC1 of 0.70-0.76. Main limitations: English-only, one generator/judge LLM, per-model query generation confounding cross-model comparison, no dataset release, and a pipeline never independently rerun.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 5
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Social bias emerges in coherent, low-refusal, response-conditioned multi-turn interactions, with 12.7-20.0% cumulative bias by turn 5 versus 1.67-12.92% zero-shot across six LLMs.
- Bias can emerge late: an additional 7.92-11.25 percentage points of dialogues show their first biased response only between turns 6 and 10.
- Per-turn bias is non-monotonic and re-emerges after disappearing in 4.58-10.00% of dialogues, so bias must be analyzed as a turn-level dynamic phenomenon.
- Model-level bias rankings differ between FairMT-Bench and DyMT-ESB (e.g., Olmo3-7B higher on FairMT-Bench, Gemma3-12B higher on DyMT-ESB), showing protocol sensitivity of bias comparisons.
- Under post-hoc ShieldGemma filtering, biased outputs remain observable among non-blocked DyMT-ESB turns, unlike FairMT-Bench where most dialogues are blocked by turn 5.
Strengths
Response-conditioned protocol enables genuinely new measurements
Because follow-ups are generated from dialogue history (Sec. 2.2, Eq. 2), the paper can quantify late emergence (Table 3: 7.92-11.25 pp of dialogues first biased in turns 6-10), non-monotonicity (Fig. 2), and re-emergence (Table 4: 4.58-10.00%) - phenomena structurally invisible to fixed-turn scripted protocols.
Carefully anchored baseline comparison
FairMT-Bench is re-run on stereotype-matched seeds rather than quoting the original paper (Sec. 3.4), and one non-alignable category was excluded instead of forcing an unfair comparison; this strengthens the protocol-level contrast in Table 2.
Judge validation uses an imbalance-appropriate metric plus transparency
Multi-pronged confound controls in appendices
Guardrail-filtering analysis adds deployment relevance
Weaknesses
Generator-judge circularity
No uncertainty on the headline dynamic claims
Cross-model comparison confounded by per-model queries
Reproducibility gaps
Small, non-blind, author-only human evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DyMT-ESB?
DyMT-ESB is a protocol for measuring social bias in multi-turn LLM conversations. It starts from a seed stereotype sentence and has a separate LLM generate each new user question based on the conversation so far, so follow-ups adapt to the model's own answers. Dialogues can run a variable number of turns, and every model response is labeled biased or not by an LLM judge that was validated against human annotation.
How does DyMT-ESB differ from FairMT-Bench?
FairMT-Bench pre-scripts user turns and feeds them regardless of what the model says, while DyMT-ESB generates each user turn from the evolving dialogue history. In this paper's experiments, DyMT-ESB conversations are much more coherent (average 4.59-4.97 vs 3.03-3.50 on a 5-point scale) and produce far fewer refusals (0.2-1.7% vs 19-63% of dialogues), yet bias still appears in roughly 13-20% of dialogues by turn 5.
What are the paper's main findings about when bias appears in a conversation?
Across six LLMs (Gemma3-12B, Qwen3-8B, Llama3.1-8B, Phi-4, Olmo3-7B, Gemini-2.5-Flash-Lite), bias emerges even without adversarial prompts; 8-11 additional percentage points of dialogues first show bias only between turns 6 and 10 (religion-related stereotypes showed the most late emergence); per-turn bias fluctuates rather than growing steadily, and bias that disappears can reappear in later turns.
Is code or data available for DyMT-ESB?
Code is available on GitHub (rem-h4/dymt-esb), including the prompt templates. The generated dialogues and datasets are not released, because they may contain harmful content, and the reported numbers come from a single run of the pipeline, so exact replication is not guaranteed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-6 are recoverable only as embedded axis-label text; exact per-turn values and heatmap patterns could not be independently verified from the extraction.
- Table 2 contains a few two-decimal cells (e.g., Phi-4 Jailbreak Tips 24.24, Gemini-2.5 refusal 22.11) amid otherwise one-decimal reporting, likely extraction noise; cited ranges avoid relying on these cells.
- Review based on arXiv v1 (16 Sep 2026); seed counts and human-evaluation sizes are taken from prose and could not be checked against released artifacts since no data is released.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.