Disclosure-Gated User Simulation for Companion-Agent Evaluation
cs.CL · cs.AI · cs.HC
Quick verdict
Specifies and ablates CompanionBench's disclosure gate, showing it is load-bearing downstream; rigorous and candid, but core depth labels and self-judging weak.
Summary
This paper fully specifies the disclosure gate that the CompanionBench benchmark's user simulator previously described in only ~400 words: five ordered gates mapped onto three observable depth layers, an eight-class transition function, rupture-triggered asymmetric retreat, seven guards, and a deterministically compiled per-example gate legend. Simulators at 35B/122B are trained so gating behavior is learned from the corpus's synthetic branch while real production conversations supply linguistic texture; after training, per-example gate information is not needed at runtime. Intrinsic audits show the mid-layer disclosure contrast rising from +0.090/+0.140 (untrained backbone, Chinese/English) to +0.530/+0.680, with permutation tests and a mis-mounting placebo excluding annotation-convention and format explanations. Downstream, across 12 systems under test, stripping gate information from training displaces rankings 3 positions (three times the seed noise band) with no detectable per-system score change, whereas a prompted gpt-5.1 environment shifts every score up by 0.521 with a nearly unchanged ranking. Only the released 122B English simulator M1b passes both acceptance criteria (ρ=0.993 with the reference). Limitations: the main judge doubles as the external reference simulator; human agreement on core depth labels is 35% and 88.6% of core labels are inferred with low confidence; the data-branch ablation confounds optimizer step counts; the released Chinese simulator lacks downstream validation; and real-corpus selection inflates observed depth.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Gating behavior is internalized into the weights: stripping gate information at inference changes the mid-layer disclosure contrast by only −0.022 (CI [−0.093, +0.053], n=321).
- The synthetic corpus branch supplies the gating: real-only training yields +0.080 versus +0.610 for synthetic-only on the contrast (∆=+0.532, p=0.0001).
- The gate is a load-bearing environment component: removing gate information from training displaces 12-SUT rankings by 3 positions (3× the seed noise band) with no detectable per-system score change (Table 4).
- Only the released 122B English simulator M1b meets both acceptance criteria — order-preserving (ρ=0.993 with the reference) and scale-stable (0/12) — while a prompted gpt-5.1 environment shifts all rubric totals +0.521 with a nearly unchanged ranking.
- Human raters prefer M3b over real conversations in blind paired preference (68.3%) yet can discriminate it from real users (37.8% 2AFC, below 50%), so no indistinguishability is claimed.
Strengths
Unusually complete falsification machinery
Depth-label and condition-label permutation controls (§6.1, D.3), a mis-mounting placebo A3 that keeps prompt shape while corrupting gate–content pairing (Appendix J), a split-call judge check against common-method variance (§6.2.1), and a second cross-family judge re-running six configurations (Appendix C.4).
Empirical noise floor instead of α-level rhetoric downstream
§6.4 judges the 3-position displacement of stripping against a seed re-run of the same environment (ρ=0.986, 1-position band, top-5 overlap 5/5), and two explicit acceptance criteria (order-preserving ρ≥0.95, scale-stable) make claims falsifiable (Table 4, Fig 3).
Candor about negative and self-limiting results
Executable, field-level specification
Human anchors with no LLM judge
Weaknesses
Judge–reference identity overlap
Core-layer construct fragility
Confounded data-branch ablation
Thin downstream sampling
Selection-inflated human depth baseline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a specification-and-audit companion to a published benchmark: the disclosure-gated user simulator that CompanionBench (Liu et al. 2026) used as its evaluation environment — there described in ~400 words — is here given a full executable specification (§3), a corpus-construction recipe (§4), intrinsic and human validation (§6.1–6.2), four ablations (§6.3), and a downstream sensitivity analysis (§6.4). Central claims: (1) gating can be trained into weights from a synthetic corpus branch and then operates without runtime gate information; (2) the gate is load-bearing — removing it from training displaces 12-SUT rankings by 3 positions against a 1-position seed noise band while per-system scores stay put; (3) a prompted gpt-5.1 environment preserves ranking but shifts all scores +0.521, invisible to rank-only checks; (4) only the released 122B English simulator M1b passes both acceptance criteria (ρ=0.993 with the reference). The two-layer evidence design (intrinsic conformance vs downstream displacement) is explicitly derived from the user-simulation-as-instrument lineage (§2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a disclosure gate in LLM user simulation?
A mechanism that makes a simulated user release personal information in ordered layers (surface → mid → core) only once the companion agent's behavior has earned it, with retreat (guards, withdrawal) when the agent misbehaves. It prevents a system under test from scoring highly merely by asking many questions instead of making the user willing to speak.
What is CompanionBench and how does this paper relate to it?
CompanionBench is a published benchmark for evaluating emotional-companionship agents. The simulator specified in this paper is the evaluation environment CompanionBench already used; that paper described the gating mechanism in about 400 words, and this paper adds the full specification, ablations, human studies, negative controls, and downstream sensitivity analysis, plus released 122B simulators.
Can the simulated user be told apart from a real person?
Yes, by design of the reported results: raters identified simulator M3b's utterance as the real person's in only 37.8% of two-alternative forced-choice judgments (CI 31.9–43.5%, significantly below 50%), so the authors do not claim indistinguishability — although in blind paired preference 68.3% of judgments chose M3b as more like a real person.
Why does the 122B scale matter if 35B already shows gating?
35B is sufficient for the gating behavior itself (the intrinsic readings saturate), but only the 122B model reproduces the published leaderboard comparably — M1b correlates at ρ=0.993 with the reference environment versus 0.853 for the 35B M1 — which is why the 122B English model is the released evaluation environment.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: several middle passages (parts of §4–§5, §8, and appendix fragments, e.g., C.1, E, H1 rows) were skipped by sampling; no conclusion in this review relies on them, but judgments touching skipped content are made from the surrounding visible text only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.