ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation

Summary

The paper introduces ProMediConv, a benchmark that frames legal dispute mediation as a proactive, multi-stage, party-aware dialogue task with 11 mediator strategies and four party behavior-pattern (BP) states adapted from the Transtheoretical Model, plus MAD, a fine-grained metric capturing average initial-to-final BP shifts within a dialogue (motivated by an ESConv analysis where identical turn counts and success hide divergent emotional gains). From 972 real cases OCR-extracted from 15 Chinese mediation practitioner books, a five-agent LLM pipeline (text purifier, legal counsel, secretary, dialogue rebuilder, BP classifier — Qwen2.5-14B zero-shot, 84.3 macro-F1) reconstructs dialogues with utterance-level strategy and BP annotations. The authors benchmark Qwen2.5, Llama-3.1, GLM-4, GPT-3.5-Turbo, two legal-specific LLMs, and prompt-based planning methods, and train ProMediAgent (RoBERTa-large planner, Qwen2.5-7B generator; SFT + REINFORCE), which leads overall (SR@t 0.9071, AT 10.34, SSR 0.9428, MAD 0.9893) — although ICL_AIF on Qwen2.5-7B attains higher MAD (1.0855), a gap the authors attribute to their reward lacking BP signals. Main limitations: every outcome is judged by an LLM reward model inside an LLM-simulated party environment (with acknowledged reward hacking); dialogues are LLM reconstructions of book narratives, not verbatim transcripts; the judge/reward LLMs and sample count are unspecified; no GPT-4-class model is evaluated; and coverage is limited to the Chinese legal context.