Who Are They to Each Other? Multi-Agent Reasoning for Speaker Relationship Inference
Yaohan Guan, Yen-Ju Lu, Yuzhe Wang, Junhyeok Lee, Jesus Villalba, Laureano Moro Velazquez, Thomas Thebaud, Najim Dehak
cs.MA · cs.CL · cs.SD
Quick verdict
Training-free multi-agent debate/compete lifts speaker-relationship inference over baselines in most settings, but margins are small and significance untested.
Summary
The paper introduces two training-free multi-agent reasoning frameworks for inferring speaker relationships from spoken conversations: Multi-Role Multi-Agent Debate (MRMAD), which assigns agents complementary roles (linguist, sociologist, psychologist) or social-theory-grounded perspectives, and Multi-Agent Compete (MAC), where agent answers compete through pairwise LLM-judge adjudication. Experiments use 607 filtered dialogues from the Seamless Interaction dataset (441 familiar, 166 stranger), across text (GPT-5-mini), audio and audio+text (GPT-audio-1.5, Qwen2.5-Omni-7B), for binary familiar-vs-stranger and fine-grained relationship classification. Reported gains over zero-shot and debate baselines are consistent but small: e.g., MAC improves binary macro F1 by 3.3 points over the strongest baseline in audio+text with GPT-audio-1.5 (66.5%), while text-modality multi-class macro F1 stays around 11-12.4% for all methods; audio-only results are mixed, with baselines winning several metrics. A 27-annotator human study finds low inter-annotator agreement (Krippendorff's alpha 0.17-0.34); models beat averaged human votes in text-included settings, while humans lead binary macro F1 in audio-only. Key limitations: no significance testing or run variance despite temperature-1 sampling, tiny per-category test sizes (coworkers n=15), a single licensed dataset, and no code release.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Structured inference-time interaction among LLM agents (role-based debate and competition) improves speaker relationship inference over zero-shot prompting and prior debate baselines in most modality-model settings.
- Gains are largest and most consistent with audio+text input, smaller in text, and mixed in audio-only, which the authors attribute to debate needing quotable, transcript-like evidence.
- GPT-5-mini outperforms GPT-5.4 variants at all reasoning effort levels, suggesting the task rewards integration of distributed conversational evidence over longer deductive chains (Table II).
- Speaker relationship inference is intrinsically ambiguous even for humans (Krippendorff's alpha 0.171-0.340); models beat averaged individual human votes in text-included settings while humans retain better balanced binary decisions in audio-only.
Strengths
Genuinely understudied task at a real literature gap
Related Work shows prior speaker-relationship work stops at feature-engineered models (LIWC/LDA in Stark 2012, boosting on acoustic features in Yella 2014, BERT-based PRIDE); the paper is the first systematic multi-modality evaluation of LLM and multi-agent methods on this task, with binary and fine-grained settings.
Principled, transparent dataset curation
Section III documents filtering explicitly: 56 Unknown-labeled and 44 empty-transcript dialogues removed, familiar-generic excluded with a conceptual argument (residual category without a defining criterion), yielding one shared 607-dialogue test set (441/166) used across all modality-model blocks for fair comparison.
Theory-grounded mechanism, not just role labels
Candid reporting of mixed results and a valuable negative result
Human reference study with agreement statistics
Weaknesses
No statistical rigor behind headline claims
Near-chance absolute fine-grained performance in text
Missing matched-compute and supervised baselines
Unexamined judge biases and no ablations
Fragile per-category evidence and single-source evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets speaker relationship inference — deciding whether two speakers are familiar or strangers and, if familiar, the relationship subtype — from spoken conversations, entirely at inference time without task-specific training. It claims that structuring multi-agent interaction improves over zero-shot prompting and prior debate baselines *in most cases*, with the largest gains in audio+text, and adds a human study showing the task is hard even for people. All experiments use a filtered 607-dialogue test set from the Naturalistic subset of Meta's Seamless Interaction dataset: 441 familiar (friends n=273, family n=102, romantic n=51, coworkers n=15) and 166 stranger dialogues, evaluated in text (GPT-5-mini), audio and audio+text (GPT-audio-1.5 and Qwen2.5-Omni-7B) at temperature 1.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is speaker relationship inference?
It is the task of predicting the social relationship between two speakers from their conversation: first whether they are familiar or strangers, and if familiar, the subtype (e.g., friends, family, romantic partner, coworkers). Cues include what is said (words, topics) and how it is said (tone, prosody, turn-taking).
Can LLMs infer relationships from conversations better than humans?
In this paper's 50-conversation study, the best LLM setups beat averaged individual human votes when text was available (zero-shot reached 76.0% binary accuracy vs 70.6% for humans). But in audio-only conditions humans achieved better balanced binary F1 (62.8% vs 56.9% for the best model). Both humans and models find the task hard, with low human inter-annotator agreement.
What is Multi-Agent Compete (MAC)?
It is a training-free protocol where three LLM agents independently propose relationship answers, an LLM judge runs all pairwise comparisons to eliminate the weakest agent, the two survivors revise after seeing the tournament broadcast, and a final head-to-head decides; the winner's revised answer becomes the prediction.
What data and models were used, and is the work reproducible?
Experiments use the Naturalistic subset of Meta's Seamless Interaction dataset, filtered to 607 dyadic dialogues (441 familiar, 166 stranger), with GPT-5-mini for text and GPT-audio-1.5 / Qwen2.5-Omni-7B for audio and audio+text. The dataset is available from Meta under a research license, but the paper publishes no code, repository, or prompt templates, and the main models are proprietary APIs, so exact reproduction is difficult.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is present only as a caption in the extraction; the framework diagrams were assessed from the Section IV prose alone.
- Agent role and judge prompt templates are genuinely absent from the entire provided text (no appendix in the extraction), so prompt-level reproducibility could not be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.