Complex-valued Phase-Coherent Transformers
cs.LG
Quick verdict
Proposes scaled-cosine scores as the fix complex Transformers need; supported by clean isolations, a Path-X solve and candid claim retractions, at small scale.
Summary
Complex-valued Transformers have inherited softmax attention over raw complex inner products, which stays near chance on ordinary sequence tasks. The paper argues, via a one-variable isolation across four diagnostic tasks and two gates (sigmoid and row softmax), that L2-normalising queries and keys—turning the score into a cosine similarity held at order-one scale by √dh—is a precondition for training: without it, every gate tested stays at chance on ListOps and Needle-in-a-Haystack, and with it all train. The mechanism is measured directly: unnormalised scores are row-constant (≈10⁻⁴) at initialisation, freezing softmax attention at uniform; the failure is optimisation, not expressivity. The resulting PCT family matches or exceeds the strongest real baseline (real screening) on long-range memory, positional retrieval, hierarchical reasoning, frequency-domain classification and RadioML; holds accuracy to depth 20; and decreases loss log-linearly over a 61-fold parameter range. A variant pairing complex screening with a phase-coherent recurrence solves Path-X at 92.71±0.89 with 91.6% complex-valued parameters; fixing phase to zero never leaves chance. Main limitations: results come from small-scale diagnostic tasks with one configuration each, some decisive cells are single-seed, held-out benchmark margins are modest, and FFT-MNIST is evaluated on training tensors.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Query–key normalisation yielding an order-one scaled cosine score is a precondition — not merely a stabiliser — for training complex attention across tasks, shown by a one-variable ablation over four tasks and two gates (Table 1).
- The failure of standard complex attention is an optimisation effect: unnormalised scores are row-constant (≈10⁻⁴) at initialisation, freezing attention uniform; the unnormalised model class contains the normalised solutions.
- Within the PCT family the choice of gate is secondary to the score path: sigmoid, L2-softmax and screening gates are task-dependent siblings (Table 2), and every gate-shape hypothesis reduces to passing gradient at initialisation (Table 3).
- PCT matches or exceeds the strongest real-valued baseline (real screening) across the tested task categories, shows no degradation to depth 20, and decreases loss log-linearly (−0.081/decade, R²=0.999) over 0.10M–6.32M parameters.
- A complex-screening plus phase-coherent-recurrence hybrid solves Path-X at 92.71±0.89 with 91.6% complex-valued parameters; fixing eigenvalue phase to zero never leaves chance, and a narrow phase band (θ∈[0,π/50]) yields 0.999 train but 0.53 test.
Strengths
Clean one-variable isolation with a directly measured mechanism
Section 2.2 / Table 1 toggles only query–key normalisation under sigmoid and row-softmax gates on Copy (d=1,000), ListOps (L=1,024), Needle (L=2,048) and FFT-MNIST, three seeds each. Section 2.7 measures the mechanism: unnormalised scores are row-constant to 10⁻⁴ at initialisation (row s.d. 6.5×10⁻⁵), attention stays uniform after 30K steps, and a temperature sweep τ∈{0.25,1,4} leaves trajectories identical; the unnormalised model class contains the normalised one, so the failure is demonstrably optimisation, not expressivity.
Unusual transparency and self-correction
Appendix E itemises withdrawn claims from the previous version (competition-for-mass explanation, LRA-Text 1.000 from four-sample train-split evaluations now superseded by 0.665 held-out, single-batch depth readings 0.78–0.81 corrected to 0.62–0.65 at 2,048 samples), each replaced by pre-registered re-measurement. Appendix F releases per-run configs, metric trajectories and summaries; Appendix C enumerates every cell×task with archive records.
Breadth with parameter-matched real baselines
Depth robustness and scaling with explicit fit statistics
Formal guarantees with honest scoping
Weaknesses
Small-scale, single-configuration scope
Single-seed cells at decisive comparisons
RadioML is mixed but headlined as categorical parity
Path-X model is a hybrid whose complex core is standard SSM machinery
Uneven evaluation hygiene across tables
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper argues that complex-valued Transformers fail on ordinary sequence tasks not because complex arithmetic is unsuitable but because they inherit softmax attention over the raw complex inner product. Its central, carefully delimited claim (Section 1, Section 2.2): a scaled cosine score — L2-normalised queries and keys, score Re⟨q̄,k̄⟩√dh — is a *precondition* for training, not a stabiliser. Supporting claims: the gate is nearly free given the score (Section 2.3); PCT matches or exceeds real screening across categories (Section 3.2); robustness to depth 20 and log-linear scaling (Section 3.3); and a hybrid that solves Path-X with a measurable 91.6% complex-parameter share (Section 4). The authors explicitly scope the work to small-scale diagnostic tasks, one configuration each, in a solve-versus-chance regime, and decline to rank variants on real datasets — an honest frame the review takes seriously.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why do complex-valued Transformers fail to train on ordinary tasks?
According to this paper, because they inherit softmax attention over the raw complex inner product. At standard initialisation that score is nearly constant within each attention row (≈10⁻⁴ variation), so attention starts and stays uniform and gradients that select tokens essentially vanish. It is a training (optimisation) failure, not a limit on what the architecture can express.
What is 'phase-coherent attention' (PCT)?
A complex attention layer that (1) uses complex-linear projections, (2) L2-normalises queries and keys so the score becomes a cosine similarity (reading the direction/phase pattern of the vectors, ignoring magnitudes), held at order-one scale via a √dh factor, and (3) applies a real-valued gate — element-wise sigmoid by default, with L2-softmax and screening as siblings. The family is provably invariant to global phase rotation of the input.
What is the Path-X result?
Path-X (16,384-token connectivity task from Long Range Arena) is solved at 92.71±0.89% by a hybrid model combining the paper's complex screening attention with a phase-coherent complex diagonal recurrence (LRU-style). 91.6% of its trainable parameters are complex-valued. This is below S5 (98.58%) and MEGA (97.98%) but above S4-v1 (88.10) and S4D-LegS (91.95). Fixing the recurrence's phase to zero never beats chance, and a too-narrow initial phase band causes memorisation (0.999 train vs 0.53 test).
Is the code and data available?
Yes. The paper states that the Lean 4 proofs, analysis scripts, and per-run configurations, metric trajectories and summaries for every listed test are in a public GitHub repository (github.com/leohio/phase-coherent-transformer-r-d), and Path-X weights for all three seeds are on the Hugging Face Hub (complexedleo/pcr-screening-pathx).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on the extracted text only; figures could not be inspected and the public archive/Lean development were not independently verified.
- Abstract-level parity claims are categorical; at row level real screening beats PCT on RadioML L2 (0.389 vs 0.352) — the text discloses this.
- The ListOps result uses the authors' synthetic generator and is explicitly not comparable to published LRA ListOps numbers; Path-X is the only full-length LRA-protocol result.
- FFT-MNIST values are measured on training-tensor samples per the authors' own footnote, so absolute accuracies should not be read as held-out performance.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.