AdamX: Cosine similarity meets gradient descent
Francisco Caldas, Ruben Belo, Cláudia Soares
cs.LG · math.OC
Quick verdict
AdamX augments AMSGrad with a bounded cosine-alignment controller; results are competitive but its edge over AMSGrad itself is thin and untested for significance.
Summary
AdamX is a first-order optimizer that multiplies the Adam/AMSGrad-style update by a bounded controller γt = exp(λ·cos(gt, gt−1)), scaling steps up when consecutive gradients align and down when they oppose, while retaining AMSGrad's monotone second-moment envelope; λ=0 recovers AMSGrad. The authors prove O(√T) convex OCO regret for a simplified momentum-free variant, showing the rate is unaffected with constants scaling as e^λ, and candidly note the full momentum version retains a linear worst-case bound. Empirically, AdamX is compared against ten optimizers on MNIST (3-layer CNN, 0.994 target) and CIFAR-10 (CifarNet, 0.84 target), five seeds, reporting epochs to test-threshold under default hyperparameters. AdamX attains the best CIFAR-10 mean (6.6 vs AMSGrad 7.8) but trails Yogi/AMSGrad on MNIST (13.2 vs 9.2). Main limitations: empirical evidence spans only two small CNNs; AdamX's margin over AMSGrad — exactly its λ=0 case — lies within one to two seed standard deviations with no significance test; the new hyperparameter λ receives no sensitivity analysis; the abstract's 'variance rectification scheme' has no explicit counterpart in the method section; and some Table 1 entries conflict with the narrative.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A bounded multiplicative cosine-similarity controller on consecutive gradients (γt = exp(λct)) can be added to an Adam/AMSGrad-style update without changing implementation complexity or scalability.
- A simplified momentum-free AdamX-OCO variant retains the standard O(√T) convex regret rate, with constants scaling by e^λ (Theorem 1), so the alignment mechanism does not worsen the rate.
- Under default hyperparameters, AdamX reaches test-accuracy thresholds in the fewest epochs on CIFAR-10 (6.6±0.54 vs AMSGrad 7.8±1.30) and remains competitive on MNIST (13.2±2.00 vs Yogi/AMSGrad 9.2).
- A variance rectification scheme promotes smoother optimization during early training (abstract claim; not defined in the method section).
Strengths
Simple, drop-in mechanism with clean failure mode
Eq. (5)–(6) define a controller bounded in [e^-λ, e^λ], independent of gradient scale, computed per parameter group; λ=0 exactly recovers AMSGrad (Section 2), making integration and ablation trivial.
Honest scoping of the theory
Section 3 explicitly separates the analyzed OCO variant (momentum removed, monotone envelope qt on the alignment-scaled step) from the practical Algorithm 1, and admits in 'Momentum and the Full AdamX Algorithm' that the momentum-bias term (Eq. 20) can be linear in T.
Sound core experiment protocol
Public code and experiments
Weaknesses
Empirical scope too narrow for the headline claim
Improvement over the λ=0 ablation is thin and untested
New hyperparameter λ never analyzed
Abstract/method mismatch on 'variance rectification'
Internal inconsistencies in reported results
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AdamX (Section 1–2) inserts a bounded cosine-similarity controller into an Adam/AMSGrad-style update: ct = ⟨gt, gt−1⟩ / max(∥gt∥∥gt−1∥, δ), γt = exp(λct) ∈ [e^-λ, e^λ], multiplying the moment-normalized step, on top of a coordinate-wise monotone second-moment envelope (Eq. 7). λ=0 recovers AMSGrad exactly — which makes AMSGrad simultaneously the paper's closest prior and its own ablation. The contributions are the controller, an OCO regret analysis of a simplified variant, and a benchmark against ten optimizers. The abstract additionally advertises a "variance rectification scheme" for early training that never appears as an explicit component in Section 2 — only the AMSGrad envelope and γ1=1 exist. The evaluated scope is two small CNN tasks: MNIST (0.994 test-accuracy target) and CIFAR-10 (0.84 target), 100-epoch caps.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AdamX and how does it differ from Adam?
AdamX is a first-order gradient optimizer that augments an Adam/AMSGrad-style update with a multiplicative controller built from the cosine similarity of consecutive gradients: when successive gradients point in similar directions the effective step grows (up to e^λ), and when they oppose it shrinks (down to e^-λ). It keeps AMSGrad's monotone second-moment denominator; setting λ=0 recovers AMSGrad exactly.
Does AdamX outperform Adam?
In the paper's two benchmarks (MNIST and CIFAR-10 with small CNNs, five seeds, default hyperparameters), AdamX reaches the accuracy thresholds faster than Adam: 13.2 vs 28.0 epochs on MNIST and 6.6 vs 9.6 on CIFAR-10. However, it trails Yogi and AMSGrad on MNIST (9.2 both), and its edge over AMSGrad on CIFAR-10 is within about one standard deviation with no significance test reported.
What convergence guarantees does AdamX have?
The paper proves O(√T) convex online-learning regret for a simplified variant without momentum that uses a monotone envelope on the cosine-scaled step size; the alignment mechanism does not change the rate, only the constants (which scale with e^λ). For the full algorithm with momentum, the worst-case bound remains linear, which the authors acknowledge.
Is code for AdamX available?
Yes. The abstract states that code and experiments are available at github.com/FranciscoCaldas/adamX. The specific λ values and per-seed results would need to be confirmed in that repository.
What are the main limitations of the AdamX paper?
Empirical evidence covers only two small CNN benchmarks (MNIST and CIFAR-10); the improvement over AMSGrad — the method's own λ=0 special case — is within one to two seed standard deviations and untested for significance; the new hyperparameter λ has no sensitivity analysis; the 'variance rectification scheme' promised in the abstract is not defined in the method section; and some Table 1 entries conflict with the text.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 were extracted as numeric labels only; curve shapes (e.g., per-epoch loss trajectories) could not be inspected.
- In the extracted Table 1, the MNIST Yogi train-loss cell reads '9.2 ± 0.734' — identical to its epochs entry; this may be a PDF-extraction artifact rather than an error in the paper, and is flagged as such.
- The public GitHub repository is cited by the abstract but was not examined, so reproducibility claims rest on the paper text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.