Message capacity and claim wording set the transition points of collective truth-finding in language-model networks
cs.MA · cs.CL · nlin.AO · physics.soc-ph
Quick verdict
Models LLM collectives as binary-neuron networks; blind capacity prediction failed, but honest failure analysis and qwen3:8b validation (15/16) impress.
Summary
The paper asks whether one number — the message capacity bounding how many peers' messages each LLM agent reads — together with single-agent response measurements predicts where an LLM collective settles on a binary claim. Communication networks are generated by a crowding acceptance process, and 31,824 randomized single-shot queries show that llama3.1:8b's snap judgment is well described as a logistic function of a weighted inbox sum with divisively normalized weights (a stochastic binary neuron). Mean-field theory predicts a critical crowding level alpha* of about 0.435 (mean in-degree 6.4 of 31) above which the wrong-consensus basin should vanish. This pre-registered prediction failed in 1,414 episodes: the correct side won below 50% from every start, even with 75% correct seeds. Replay analysis traces the failure to 'field transport' — per-claim thresholds do not transfer across claim sets, while message weights do. A pre-registered discrimination using negated rewrites and true claims shows the threshold follows the assertion, not the truth. On qwen3:8b the pipeline predicted claim-dependent bistability and observed transition points matched computed positions (15/16 cells; mean absolute difference 0.04); llama3.3:70b showed no assertion bias. Limitations: the fold in capacity itself was never observed; a single REFUTES-skewed claim set; quantized 8B models; local, non-registry pre-registration; and a small unexplained residual (collectives 0.05-0.17 more correct than predicted).
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An 8B LLM's snap judgment of a claim is well captured by a logistic of a divisively normalized weighted sum of its inbox — a stochastic binary neuron (31,824 queries; cell RMS 0.054 vs 0.118 under constant weights).
- From single-agent coefficients and network degree statistics alone, mean-field theory computes a critical message capacity (alpha* = 0.435, mean in-degree 6.4 of 31) at which the wrong-consensus basin should vanish — and this blind, pre-registered prediction failed in llama3.1:8b collectives.
- The failure mode is 'field transport': per-claim thresholds do not transfer across claim sets, while message weights do.
- The threshold follows the assertion, not the truth, in llama3.1:8b (negated rewrites flip outcomes); the polarity channel is absent in qwen3:8b and undetected in llama3.3:70b.
- In qwen3:8b, transition points predicted from single-agent data alone matched observations in 15 of 16 conditions, with mean absolute error 0.04 and rank correlation 0.96.
Strengths
Honest pre-registered failure
alpha*=0.435±0.03 and its decision criteria were registered before any collective episode; when the prediction failed (0 of 2,000 claim-clustered bootstrap resamples crossed the 20% criterion, Sec. 4.2/S5.3), the failure is reported prominently and then quantitatively diagnosed — rare and credibility-enhancing.
Productive failure decomposition
Replaying every realized inbox through the measured response function separates the weights (validated: Brier 0.10-0.14 vs 0.17-0.25 climatology, ECE 0.03) from the field (failed: experimental-claim fields 0.16-0.55 vs calibration mean 0.94), cleanly isolating 'field transport' as the failure mode (Sec. 4.3).
Causally clean polarity discrimination
Out-of-sample validation on a second model
Theory closure validated
Weaknesses
Title-evidence gap
Confounded claim set
Scaffold sensitivity of the pivotal quantity
Local-only pre-registration
Quantized serving, single-sample judgments
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper (arXiv 2609.19183) asks whether an LLM collective debating a binary claim can be characterized before it runs, using only single-agent measurements. Two quantities carry the theory: (i) a message capacity alpha that generates the communication network (an agent that has accepted r sources accepts the next with probability e^(-alpha*r)); and (ii) a measured single-agent response function — a logistic of a divisively normalized weighted inbox sum, i.e., a stochastic binary neuron. From 31,824 randomized single-shot queries on llama3.1:8b, the mean-field map (Eqs. 2-3) predicts a critical crowding level alpha* = 0.435 (mean in-degree 6.4 of 31) above which the wrong-consensus basin should vanish. The paper's distinguishing feature is candor: the pre-registered prediction failed, and the failure is decomposed into mechanism ('field transport') and design artifact (finite horizon). Supporting results: a pre-registered polarity discrimination (Sec. 4.5), a second-model validation of predicted transition points (Sec. 4.6), and a 70B single-agent probe.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures appear only as interleaved text descriptions in the extraction; visual content (Fig. 2-7) could not be independently verified.
- The GitHub repository is stated as 'to be released'; its availability and contents could not be checked (no network access).
- Extraction is otherwise unusually complete (main text plus full SI including per-cell outcome tables), supporting high confidence.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.