What You Can't See Is Still What You Learn: A Preregistered Sixty-Society Confirmation That Evidence Masking Drives Compositional Generalization
cs.AI · cs.LG
Quick verdict
Preregistered study: evidence masking drives near-ceiling compositional generalization in a small LM relay; rigorous and transparent, but single-world scope.
Summary
The paper preregisters and runs a confirmation study of whether restricting each module's evidence access improves compositional generalization in a modular language-model system. Sixty four-cell 'societies' share a frozen Qwen2.5-0.5B-Instruct backbone with one shared rank-8 LoRA and communicate through learned continuous packets; the task is ordered composition of twelve affine maps modulo 17. Five conditions cross evidence masking, ownership markers, and neutral-filler replacement across six initialization clusters and two data orders. With markers present, masked systems reach ceiling (median depth-3 accuracy 1.000) versus roughly 0.14 for globally visible systems; all twelve matched pairs clear the preregistered 0.20 margin (median paired differences +0.846/+0.859), and the unmarked replication also passes. Marker-following fails in every global system, leaving role equalization unresolved; the neutral-filler arm splits (7 full generalizers, 4 non-acquirers, 1 shallow-partial), and its decomposition contrasts are inconclusive. Packet interventions in all eighteen audited masked systems show perfect value-level fidelity, though success-conditioned audits do not establish mediation. Main limitations: a single task world, one 0.5B backbone, an evaluation-embargo violation with pre-verdict outcome inspection, and unretained admission records.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under the tested architecture, task world, and training budget, evidence masking yields near-ceiling held-out compositional accuracy (median depth-3 = 1.000) versus roughly 0.14 under global visibility, with all twelve matched pairs exceeding the frozen 0.20 margin.
- The masking advantage replicates without ownership markers (median paired differences +0.884/+0.917) on a fresh preregistered task world, passing confirmation criteria that the parent cohort formally missed by 0.0012.
- No globally visible system passes the counterfactual marker-following check, so whether usable role information eliminates the masking gap remains unresolved rather than refuted.
- All eighteen audited masked systems show perfect value-level packet intervention fidelity (same-value preservation and counterfactual following of 1.000), though these success-conditioned audits do not establish that the packet code mediates generalization.
- Seven of twelve neutral-filler systems fully generalize under global attention geometry, but the preregistered decomposition contrasts are inconclusive about how much removal of foreign task content drives the masking advantage.
Strengths
Frozen preregistration with verifiable provenance
The protocol, a 38-file SHA-256 code inventory, contrast definitions, and decision rules were hash-frozen and Software Heritage-archived (snapshots dated 2026-09-11) before seed generation; the verdict script ran exactly once under an atomically locked exclusive lock on the compiled bundle (Appendix A).
Large, uniformly consistent behavioral effect
All 12 matched R+/G+ pairs clear the frozen 0.20 margin at both depths (smallest margins +0.71 at depth two, +0.39 at depth three), with median paired advantages +0.846/+0.859; masked arms sit at median 1.000 versus 0.083-0.154 for global arms (Table 1), and the unmarked C6 replication passes at +0.884/+0.917.
Controls that bound the explanation
Candid disclosure of protocol deviations
Value-level intervention evidence with honest scoping
Weaknesses
Single task world and model scale behind a general title claim
Evaluation-embargo violation weakens the preregistration protection
Marker manipulation cannot answer the paper's own attribution question
Low statistical power and bimodality in the descriptive tests
Incomplete audit coverage and unverifiable fleet admission
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
In a preregistered experiment with sixty small language-model systems, modules that were masked from reading other modules' evidence reached near-perfect accuracy on held-out compositions of learned arithmetic operations (median 1.000 at depth three), while systems that could read all evidence reached only about 0.14. All twelve matched comparisons exceeded the preregistered 0.20 accuracy margin, and the result replicated without ownership markers.
Does this mean restricting input access always makes AI models generalize better?
No. The result is demonstrated for one small task world (ordered compositions of twelve affine maps modulo 17), one 0.5B-parameter backbone, and a fixed relay architecture with four cells. The authors explicitly scope the claim to this tested regime and propose follow-ups in other worlds, at other scales, and with less scaffolding before any general recipe is claimed.
Is the code, data, and model checkpoint availability good?
Yes. The preregistration, per-society raw evaluation counts, mechanism-audit outputs, secondary statistics, and all sixty final training checkpoints (Hugging Face repository tokenosopher/populus-evidence-partitioning-checkpoints) are public, with SHA-256 inventories and Software Heritage snapshots of the frozen code.
Was the study truly preregistered and free of p-hacking?
The protocol, code hashes, contrast definitions, and decision rules were frozen and third-party archived before any random seed was generated, and the verdict script ran once under an exclusive lock. However, the authors disclose that the planned cohort-wide evaluation embargo was violated: most results were computed and inspected before the verdict, though no outcome-dependent protocol change is reported and all numerical gates were fixed beforehand.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are present only as extracted text fragments; per-trajectory score distributions could not be visually inspected, so figure-level claims rely on the reported tables.
- External artifacts cited as evidence (SECONDARY_STATS_v2.json, the GitHub evidence repository, Hugging Face checkpoints) were not inspectable; statements about their content rest on the manuscript's description.
- Tables were reconstructed from running text in the extraction; minor column misalignment in Table 4 is possible, though all quoted numbers were cross-checked against surrounding prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.