Two tests of phase-structure features for transition prediction
cs.CL
Quick verdict
Null-result report: phase features add ~+0.001 AUROC, failing prespecified gates; unusually candid, but Study 2's gate was retrospective, not confirmatory.
Summary
This report from ENTRUST AI's founder tests whether phase-structure features derived from hidden-state trajectories improve endpoint prediction over a combined baseline, extending the author's prior rotary-attention framework (arXiv:2607.25507). Study 1 froze a contradiction-category pipeline before sealed scoring: on 1,136 eligible cases from Qwen2.5-3B-Instruct, adding the fitted phase feature PC-2 yielded a paired AUROC difference of +0.00087; the 99% bias-corrected interval [−0.0046, +0.0069] included zero and the prespecified +0.05 threshold failed. A replication role (Qwen2.5-1.5B, 1,063 cases) showed +0.00019, passing only the direction check. Study 2 retrospectively developed fifteen treatments (layers 24–36, meanmax, concat) on 1,415 answer-change comparisons with twenty-repeat five-fold grouped cross-validation; a 9 September 2026 execution applied a three-condition gate to b0–b4 development statistics and no treatment advanced (best: layer 25, +0.00027, favorable in three of five seed blocks). Unusually candidly, the paper discloses that the Study 2 gate was applied retrospectively — the signed protocol's b5 selection and b6 evaluation were never completed — and that August records do not prove an earlier notebook execution. Key limitations: development statistics double as tuning estimates, no whole-seed-block holdout, an entirely self-referential evidence chain, and single-author commercial affiliation.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Adding the fitted phase feature PC-2 to a combined baseline improves sealed contradiction AUROC by only +0.00087 on 1,136 Qwen2.5-3B cases, with the 99% BCa interval including zero and the prespecified +0.05 threshold unmet.
- A smaller replication model (Qwen2.5-1.5B, 1,063 cases) shows a same-direction +0.00019 increment, satisfying only the direction condition.
- Under the applied three-condition gate on b0-b4 development statistics, zero of fifteen Study 2 treatments advanced; the best, layer 25, gained approximately +0.00027 with a favorable sign in three of five seed blocks.
- The Study 2 gate was applied retrospectively; the signed protocol's b5 selection and b6 evaluation were never completed, and August records do not evidence an earlier notebook execution.
- The results do not test the rotary score identity, its local pre-softmax bound, or execution-boundary governance effectiveness.
Strengths
Exceptional provenance candor
Sections 4.4, 6.1 and 7.1 explicitly separate the executed 9 September verification from the August records, correct the earlier publication's implication of an Artifact 5 execution ('The author reports no Artifact 5 execution before 9 September 2026'), and state the gate was not prospectively locked — disclosure well beyond field norms.
Genuinely sealed Study 1 design
Pipeline frozen 11 August 2026 with hashed, reload-verified model roles (Qwen2.5-3B revision aa8e7253..., 1.5B revision 989aa798...); sealed scoring restricted to written-answer-resolved cases; 10,000-resample whole-quartet bootstrap (seed 260725507) preserves eligibility structure within quartets.
Honest scope discipline
Complete numerical accounting
Weaknesses
Study 2 is not confirmatory
Power/effect mismatch
Self-referential evidence chain
PC-2 is not one construction across studies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tests two operationalizations of "phase-structure" features from the author's prior theoretical framework (arXiv:2607.25507) against a combined baseline in a synthetic authority-conflict task (status register vs. repeated unverified advisory, Qwen2.5-3B and 1.5B backbones): (1) casewise contradiction — written answer differing from the governing target — under sealed scoring (Study 1), and (2) answer-identity change across adjacent pressure levels 0→2, 2→4, 4→6 (Study 2). The central claims are negative: PC-2 adds +0.00087 paired AUROC on 1,136 primary cases (99% BCa interval [−0.0046, +0.0069]), failing both primary advancement conditions; the 1.5B replication adds +0.00019 (direction only); and in Study 2 zero of fifteen treatments passed the applied gate (best: layer 25, +0.00027, 3/5 blocks). Sections 7 and 8 explicitly disclaim testing the rotary score identity, the pre-softmax bound, or governance effectiveness.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did the phase-structure features improve prediction in this paper?
No. In Study 1, adding the fitted phase feature PC-2 to a combined baseline changed sealed AUROC by only +0.00087 on 1,136 cases (99% CI included zero; the +0.05 threshold failed); in Study 2, none of fifteen treatments passed the applied gate, the best increment being +0.00027 at layer 25.
Was this study preregistered?
Partially. Study 1's pipeline was frozen before sealed scoring, but Study 2's selection gate was applied retrospectively: the signed protocol called for b5 selection and b6 evaluation, which were never completed, and the paper openly states the gate was not fixed before development results were known.
Which language models were used?
Qwen/Qwen2.5-3B-Instruct (primary role) and Qwen/Qwen2.5-1.5B-Instruct (replication role), each pinned to a specific Hugging Face revision hash in the acquisition manifest.
Does this paper disprove the author's rotary-attention phase theory (arXiv:2607.25507)?
No. The paper explicitly states the experiments do not test the rotary score decomposition, its pre-softmax bound, or the governance framework; they only evaluate two specific operationalized hidden-state feature constructions against a combined baseline.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract in the supplied metadata is cut mid-sentence ('Neit...'), but the embedded full text contains the complete abstract and ends naturally at reference [7] (page 12); no body-text truncation was detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.