AI & Research · arXiv Machine Learning · publications

Learning Rate Transfer in Normalized Transformers

Source excerpt

This page presents AI and research content to help researchers, students and AI tools read, cite and verify the information.

arXiv:2604.27077v1 Announce Type: new Abstract: The Normalized Transformer, or nGPT (arXiv:2410.01131) achieves impressive training speedups and does not require weight decay or learning rate warmup. However, despite having hyperparameters that explicitly scale with model size, we observe that nGPT does not exhibit learning rate transfer across model dimension and token horizon. To rectify this, we combine numerical experiments with a principled use of alignment exponents (arXiv:2407.05872) to revisit and modify the $\mu$P approach to hyperparameter transfer (arXiv:2011.14522). The result is a novel nGPT parameterization we call $\nu$GPT. Through extensive empirical validation, we find $\nu$GPT exhibits learning rate transfer across width, depth, and token horizon.

Your DzCademia workshop

Check a publication method or promise

A video or journal-finding tool can shortlist titles, not guarantee acceptance. Check identity, scope, author instructions and possible fees at official sources. Do not choose solely on speed promises or one indicator.

A practical exercise

Compare two journals against your research question: fit, article types, instructions and verified terms. Keep unknowns as questions rather than assumptions.

Record reference points

Topic: Learning Rate Transfer in Normalized Transformers
Source website: arxiv.org
Publication date at source: 2026-05-01
Collection date on DzCademia: 2026-05-01

My preparation plan

An indicative personal list, separate from official documents or requirements. Checkmarks stay on this device; no account needed.

Identify the journal and official site.Read the scope and sample papers.Check instructions and possible charges.Reject unsupported publication guarantees.Prepare an original, compliant manuscript.

0 / 5 steps checked

Read the source

DzCademia advice and exercises are separate from the source summary and requirements. This structured aid uses automated assistance and is neither scientific validation nor a selection promise. Source consulted on 2026-10-28.

View the original source

Official or original source: arXiv Machine Learning. Always check the details against the primary source.

Back to AI & Research

Learning Rate Transfer in Normalized Transformers

Check a publication method or promise

A practical exercise

Comment with Google

Your browsing, your choices