Muon is an optimization method for training artificial neural networks. Its distinguishing operation is to approximately orthogonalize a momentum-based update for each selected weight matrix before applying it. Rather than independently rescaling individual coordinates, as in Adam, Muon transforms the singular-value structure of an entire update matrix. It is used particularly for hidden-layer weights in deep learning, including transformer models, while parameters such as biases and embeddings are commonly handled by a separate optimizer.
The name expands to Momentum Orthogonalized by Newton–Schulz. The method was introduced by Keller Jordan in 2024 and subsequently attracted interest as an alternative to adaptive, coordinate-wise optimizers for language-model training.
Basic update rule
Consider a weight matrix , with gradient obtained by backpropagation from a loss function. A representative Muon update begins by accumulating momentum:
where controls the persistence of past gradients. Implementations may omit the factor , or construct a Nesterov-style look-ahead direction from the momentum buffer and current gradient.
The resulting direction is passed through an approximate matrix-orthogonalization procedure:
The weight update then takes the form
where is the learning rate and is an implementation-dependent scaling factor. Some implementations additionally apply decoupled weight decay, a form of regularization:
Momentum conventions, dimension-dependent scaling, weight decay, and the exact orthogonalization procedure differ across implementations. Consequently, the name Muon identifies a family of closely related update rules rather than a single universally fixed formula.
What orthogonalization means
The mathematical target can be described using the singular value decomposition. For a nonzero update matrix , write its compact decomposition over its nonzero singular values as
The corresponding idealized orthogonalized direction is
This transformation preserves the left and right singular vectors but replaces every nonzero singular value with one. It therefore removes the relative magnitudes of the singular values while retaining their associated directions.
For a full-column-rank matrix with ,
For a full-row-rank matrix with ,
A rectangular matrix cannot generally have both orthonormal rows and orthonormal columns. In rank-deficient cases, the expression above is a partial isometry: the corresponding products are projections rather than full identity matrices.
This operation is closely related to the polar decomposition. For example, when has full column rank,
Importantly, Muon orthogonalizes the update direction, not the weight matrix itself. It does not impose an orthogonality constraint on the learned weights.
Geometric interpretation
In mathematical optimization, an idealized orthogonalized gradient can be interpreted as a steepest-descent direction under a spectral-norm constraint. For a matrix , the direction is a solution of
where .
Ordinary gradient descent instead corresponds to a Frobenius-norm constraint on the update. Muon thus changes the geometry used to define an update of bounded size. This interpretation explains the idealized matrix transformation; it is not, by itself, a convergence guarantee for momentum-based training of nonconvex networks.
Newton–Schulz approximation
Computing an exact singular value decomposition for every matrix at every training step would generally be expensive. Muon instead uses a small, fixed number of Newton–Schulz iterations, which rely chiefly on matrix multiplication and are well suited to graphics processing units.
A classical iteration illustrating the principle is
With suitable initialization and scaling, this iteration drives nonzero singular values toward one. An initial normalization such as
places the singular values in a controlled range; zero inputs require separate handling.
Practical Muon implementations often use higher-degree polynomial updates, for example
The coefficients and iteration count are selected to produce a useful approximation with few matrix multiplications. They need not reproduce the convergence behavior of the classical cubic iteration. In particular, some aggressively tuned, finite-step procedures yield singular values near one rather than an exact polar factor.
Thus, “orthogonalized” describes the intended transformation, not a guarantee that every computed update has exactly orthonormal rows or columns. Reduced-precision arithmetic, including bfloat16, introduces further approximation.
Parameter selection and scaling
Muon is primarily designed for matrix-valued hidden-layer weights. In a transformer, these include matrices used in attention projections and feed-forward blocks.
Training systems commonly divide parameters into two groups:
- Muon parameters: selected hidden-layer weight matrices.
- Auxiliary-optimizer parameters: biases, normalization parameters, input embeddings, output projections, and other excluded tensors.
The second group is often trained with Adam or an AdamW-style update. These exclusions are conventions rather than a mathematical requirement: an embedding table is also a matrix, but its role and gradient structure differ from those of a dense hidden-layer transformation.
Shape-dependent scaling matters because an exact orthogonalized matrix of rank satisfies
Its total and per-coordinate update magnitudes therefore depend on matrix dimensions. Implementations may compensate for aspect ratio or use a target root-mean-square update scale. Learning-rate values are not directly comparable across variants unless these conventions also match.
For higher-order tensors, such as weights in a convolutional neural network, applying Muon requires choosing a matrix reshaping. Different reshaping choices define different transformations.
Comparison with other optimizers
Momentum gradient descent
Momentum gradient descent smooths gradients over time but retains the relative singular values of the resulting matrix direction. Muon adds a matrix-level transformation that approximately equalizes those singular values.
This can increase the relative contribution of directions associated with smaller nonzero singular values. It can also amplify directions that contain noise; orthogonalization does not distinguish useful signal from stochastic variation.
Adam and related adaptive methods
Adam maintains coordinate-wise estimates of first and second moments. Its effective rescaling depends on individual parameter coordinates.
Muon instead uses the joint matrix structure of an update. Its core state ordinarily includes a momentum buffer but no Adam-style elementwise second-moment buffer. That can reduce optimizer-state storage for the parameters it handles, although total training memory also includes weights, gradients, activations, and any auxiliary optimizer state.
Muon is not generally described as a Hessian-based second-order optimizer. Although its transformation contains an inverse-square-root-like matrix operation, it does not explicitly estimate the Hessian or the Fisher information matrix.
Computational considerations
Muon's added cost is dominated by the matrix products used during orthogonalization. For a matrix of shape , an iteration based on forming a Gram matrix on the smaller dimension has a rough arithmetic cost of
Actual runtime depends on matrix shape, precision, kernel efficiency, iteration count, and hardware utilization. The operation is more expensive than simple elementwise optimizer arithmetic, but it may remain a modest fraction of a large model's forward and backward computation.
In distributed training, matrix-level transformations also interact with parameter sharding. Orthogonalizing separate shards is generally not equivalent to orthogonalizing the full matrix. Implementations must therefore choose between preserving the full-matrix operation, communicating additional data, or accepting a modified blockwise procedure.
Evidence and limitations
Muon has been investigated particularly in language-model training, where reported experiments have shown advantages over Adam-based baselines in some configurations. Such results depend on architecture, training duration, tuning, numerical implementation, and the resource budget used for comparison.
Several distinctions are important when interpreting these results:
- Steps versus time: fewer optimization steps do not necessarily mean lower wall-clock training time.
- Training loss versus downstream behavior: a lower training loss does not automatically establish better generalization or performance on every task.
- Optimizer versus training recipe: changes in learning-rate schedules, weight decay, initialization, or parameter grouping can affect comparisons.
- Exact theory versus practical computation: theoretical properties of the polar factor do not automatically hold for a short polynomial approximation.
- Transfer across settings: results from pretraining do not establish the same advantage for fine-tuning, sparse models, or every neural-network architecture.
Muon also does not remove the need to tune hyperparameters. Its matrix-aware update introduces additional implementation choices, especially orthogonalization coefficients, iteration count, scaling, and parameter grouping, that affect both optimization behavior and computational cost.