Speak for Me: Giving LLMs the Situational Awareness to Participate in a Meeting
Muneeb Khan, Frederic Kirstein, Terry Ruas, Bela Gipp
cs.AI · cs.CL
Quick verdict
结构化会议状态将 AMI 上的代理静默率从 51.4% 降至 2.5%；协议稳健，但 LLM-judge 信用和离线回放削弱了在线代理的可信度。
Summary
本文针对在线会议委派问题，即 LLM 智能体代表缺席参会者并决定何时发言及发言内容。作者指出仅基于 prompt 的代理在 AMI 语料库中对于参会者的发言机会保持沉默的比例达 51.4%，并提出 CAPA，这是一种固定权重的感知-行动-再校准架构，用于维护显式的、受 schema 约束的会议状态（议题、决策、未决问题、立场、覆盖范围、发言权），由 Perceiver 更新，由 Controller-Generator 对执行，并由使用 turn forecaster 和两个 LLM judges 的 Recalibrator 进行修复。他们引入了一种基于参会者自有意念单元的情节级评估协议，其受 schema 约束的 judges 通过三位人工标注者的验证（Cohen's κ = 0.71）。在 137 次 AMI 会议（1,126 个情节）中，CAPA 将未尝试情节的比例从 51.4% 降低到 2.5%，大致使宽松召回率（loose recall）翻倍（26.1%→52.2%），将严格召回率（strict recall）从 10.7% 提升到 25.1%，并将 Decision F1 从 38.1 提升到 63.0，同时将幻觉率控制在 0.6%；消融实验将这一提升归因于显式状态而非更长的原始上下文。主要局限性：CAPA 近一半的干预（45.3%）未被记入，相同的 LLM 家族同时担任代理和 judge，评估是在离线因果回放中进行的，其中代理的话语无法影响真实对话，且代码仅在录用后提供。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Prompt-only delegates fail at opportunity recognition, staying silent on 51.4% of the represented participant's talking opportunities on AMI (Sec 1, Table 2).
- An explicit, continuously maintained meeting state — not longer raw context — is the lever that closes the recognition gap (Sec 6.1, Tables 8/15).
- Under the episode-level protocol, CAPA cuts not-attempted episodes from 51.4% to 2.5%, roughly doubles loose recall to 52.2%, and holds hallucination at 0.6% across 137 AMI meetings (Table 2).
- The bottleneck for LLM delegates is the binary floor-taking decision rather than generation timing; recalibration refines selection while the state carries the threshold (Sec 5.2, 6.2).
- State-driven modular design converts opaque omission failures into module-attributable selection errors (Sec 5.3).
Strengths
Architecture derived from documented failure modes
Table 1's six desiderata are each traced to a prompt-only failure documented by Hu et al. 2025; the POMDP belief-policy decomposition (Sec 3.2-3.3) is instantiated concretely with schema-constrained modules, and pre-generation commitment of proposition zt is a sensible anti-hallucination device.
Human-validated LLM judges, rare in this literature
Sec 4.3/App C: three trained annotators, calibration round plus adjudication, 150 idea units and 150 same-proposition decisions, aggregate Cohen's kappa = 0.71 — a stronger validation than most LLM-judge evaluation pipelines report.
Large-scale gains with credible interval reporting
Module-attributable error analysis
Unusual transparency on cost and prompts
Weaknesses
Judge-agent circularity with partial human validation
Offline causal replay cannot support online claims
Nearly half of interventions go uncredited
Thin baseline set for the core causal claim
Code released only after acceptance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses participant-specific meeting delegation: an LLM proxy attends on behalf of an absent stakeholder and must decide, at every turn, whether to speak and what to contribute. Building on failure modes documented by Hu et al. (2025) — silent abstention on 51.4% of talking opportunities, surface-cue-bound timing, and content suppression — the authors argue the root cause is missing state tracking and propose CAPA, a fixed-weight perceive-act-recalibrate architecture. Three claims are tested: (i) an explicit, schema-constrained meeting state (topics, decisions, open questions, stances, coverage, floor; Table 1 desiderata v-vi) closes a recognition gap that raw-context scaling cannot; (ii) the bottleneck is the binary floor-taking decision, not generation timing; (iii) modular state-driven design converts opaque omissions into module-attributable selection errors. A second contribution is an episode-level evaluation protocol anchored on participant-owned idea units with human-validated LLM judges (kappa 0.71).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does CAPA solve in meetings?
CAPA is an architecture that lets an LLM agent represent an absent meeting participant. It tracks topics, decisions, open questions, stances, coverage, and speaking floor in an explicit state, and decides when to speak and what to contribute in that person's style, instead of staying silent or interrupting at the wrong moment.
How much better is CAPA than a plain prompted LLM on AMI meetings?
On 137 AMI meetings, the prompt-only delegate missed 51.4% of the absent participant's talking opportunities; CAPA reduced this to 2.5%, roughly doubled credited recovery (loose recall 26.1% to 52.2%), raised floor-taking F1 from 38.1 to 63.0, and kept hallucination at 0.6%.
Were the results verified by humans?
Partly. The evaluation's LLM judges for idea-unit extraction and same-proposition matching were validated against three human annotators with Cohen's kappa of 0.71. Quality flags such as hallucination, redundancy, and off-topic were scored by LLM judges; human agreement for those specific flags is not reported in the paper text.
Is the code and data available?
The evaluation uses the public AMI Meeting Corpus (CC BY 4.0). The authors state that CAPA's code, prompts, judges, and analysis scripts will be released on GitHub under an MIT license after paper acceptance, so the implementation is not yet downloadable.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked '[...sampling gap...]' breaks; parts of Methodology 3.3, human-eval Section C detail, ablation Tables 8-9, Sections D-F, and portions of Appendix J were skipped, so judgments on those rely on in-text summaries.
- Ablation effects (e.g., loose recall -22.4% without state, -3.9% without recalibration) are cited from the in-text numbers of Sections 6.1-6.2, not from the Tables 8-9 themselves, which fall in a sampling gap.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.