Deriving Muon From Mirror Descent: The Proximal Geometry Behind Orthogonalization

Muon is an optimizer that has recently attracted widespread attention, and its core orthogonalization operation has been interpreted in several ways from the perspective of constrained optimization. However, a natural question arises: Muon itself is an unconstrained optimizer — can its orthogonalization operation be derived naturally from a pure unconstrained optimization framework? Starting from the perspective of Mirror Descent, this article provides a concise and definitive answer — the key lies in a class of proximal geometry induced by the gradient itself. Furthermore, we find that Muon is not an isolated method, but a special case within a broader family of spectral transformation optimizers.

Muon: The New Optimizer That Shook Adam's Throne

At the end of 2024, Muon burst onto the scene^[1].

Prior to this, Adam and its variant AdamW had dominated deep learning optimization for nearly a decade. From early deep neural networks to today's large-scale Transformers, training almost all mainstream models relied on the Adam family of optimizers. The emergence of Muon truly shook this long-stable landscape for the first time.

In less than two years, Muon has been used to train ultra-large-scale frontier foundation models such as Kimi K3^[2], GLM-5^[3], and DeepSeek V4^[4]. According to Zhilin Yang, founder of Moonshot AI, compared with Adam, Muon delivers about a 2x improvement in token efficiency^[5], meaning the training tokens required to achieve the same performance are reduced by roughly half. Looking back at the history of optimizers, it is exceptionally rare for a new optimizer to gain attention and enter frontier foundation model training this rapidly.

Surprisingly, Muon's core operation is deceptively simple. For a momentum matrix \(M \in \mathbb{R}^{m \times n}\), let its Singular Value Decomposition (SVD) be

What Muon does can be summarized as orthogonalizing this matrix to obtain

which effectively erases the singular values from the SVD, or equivalently sets them all to one, and then updates parameters along this direction.

That's it.

Yet it is precisely this seemingly simple "orthogonalization" that enables Muon to exhibit astonishing capability, drastically improving training efficiency and final performance in Transformer models.

The Explanation of Orthogonalization: Elegant, But with a Gap

Muon's massive success sparks natural curiosity: where on earth does such an effective orthogonalization operation come from?

One classic explanation interprets Muon's orthogonalization as the Linear Minimization Oracle (LMO) within Frank–Wolfe (or Conditional Gradient) methods^[6].

Specifically, given a gradient matrix \(G_t \in \mathbb{R}^{m \times n}\), consider the linear minimization problem over the spectral norm ball:

where \(\langle \cdot, \cdot \rangle_F\) is the Frobenius inner product. If \(G_t = U\Sigma V^\top\), the solution to this problem is precisely

which is the orthogonalized direction of the gradient (up to a sign depending on convention).

In other words, Muon's orthogonalization operation corresponds exactly to the LMO over the matrix spectral norm ball.

Earlier work approached it from the perspective of steepest descent, interpreting Muon's orthogonalization as the steepest descent direction under the matrix spectral norm^[7]. Notably, the origin of this idea traces back to the Spectral Descent work of 2015, contemporaneous with Adam^[8]. Although these two explanations start from different premises, they converge on the same core: spectral norm constraints naturally give rise to Muon's orthogonalization operation.

While these works brilliantly answer "where orthogonalization comes from", they also leave a subtle gap:

Admittedly, related work has attempted to further extend the classic constrained optimization method of Frank–Wolfe into unconstrained optimization settings^[6]. However, when trying to understand Muon, this explanation still feels somewhat indirect: we must first artificially introduce a spectral norm constraint that does not exist in actual training, and then rely on the geometric properties of this constraint to explain why orthogonalization emerges. Furthermore, the matrix spectral norm ball itself is not a particularly intuitive geometric object, and the mathematical process of deriving orthogonalization from spectral norm constraints is not easy to parse. Thus, a more fundamental question arises:

The "Magic" of Mirror Descent

Continuing along the unconstrained perspective, we naturally recall Mirror Descent^[9]. Its ingenuity lies in the fact that many operations that appear to be specific transformations applied to gradients can be reinterpreted as advancing along the most natural direction within a non-Euclidean geometric space defined by a Bregman divergence. In Mirror Descent, this geometric structure defined by a Bregman divergence is commonly referred to as Proximal Geometry. Standard Gradient Descent itself is a special case of Mirror Descent: when the proximal geometry is characterized by the Bregman divergence corresponding to the Euclidean distance, Mirror Descent degenerates to Gradient Descent; when characterized by the KL divergence, it yields the classic Exponentiated Gradient^[10].

This reveals a profound shift in perspective: rather than asking what processing an optimizer "does to the gradient", we should instead ask—

What kind of proximal geometry does the optimizer choose?

So, does Muon's orthogonalization of gradients also correspond to a certain proximal geometry, such that the corresponding Mirror Descent update happens to be equivalent to orthogonalizing the gradient?

The answer is surprisingly concise.

The Matrix Online Mirror Descent Framework

To study this geometric perspective in matrix spaces, we first extend Mirror Descent to matrix parameters. To this end, we adopt the more modern Online Mirror Descent (OMD) framework. Traditional OMD focuses primarily on vector parameters; for matrix parameters, the more natural approach is not to vectorize matrices into high-dimensional vectors, but rather to define proximal geometry directly in the matrix space. Based on the OMD for Matrices framework proposed in Matrix AdaGrad^[11], we take matrices directly as decision variables and define proximal functions and their corresponding Bregman divergences in the matrix space. Specifically, the matrix OMD update takes the form:

where \(\psi\) is the proximal function defined on the matrix space that determines the optimization geometry, and \(B_\psi\) is the Bregman divergence induced by \(\psi\):

Row-wise and Column-wise Geometry Proximal Functions

Next, let us look at common proximal functions in matrix spaces. A typical and important choice is row-wise geometry. Given a symmetric positive definite metric matrix \(A \in \mathbb{R}^{m \times m}\), define the proximal function:

This is a quadratic function of \(X \in \mathbb{R}^{m \times n}\), whose gradient is \(\nabla\psi_A(X) = AX\). Substituting this into the definition of the Bregman divergence gives:

which is a matrix squared Mahalanobis distance weighted by \(A\). When \(A\) is a diagonal matrix, the above geometry degenerates to independent row-wise weighting, which is computationally efficient while preserving the matrix structure. Symmetrically, column-wise geometry can be defined via right multiplication by a metric matrix \(B \in \mathbb{R}^{n \times n}\), i.e., \(\psi_B(X) = \tfrac{1}{2}\langle X, XB \rangle_F\).

Now, the key question becomes:

Gradient-Induced Proximal Function

The answer is remarkably concise: Construct a proximal function induced by the Gram structure of the gradient itself.

Specifically, for the matrix gradient \(G_t \in \mathbb{R}^{m \times n}\), we consider its Gram matrix and take its principal square root (i.e., the unique symmetric positive semidefinite square root). When \(m \le n\), we use the row Gram matrix:

When \(m > n\), we can instead use the column Gram matrix \(G_t^\top G_t\) (the derivation is analogous and leads to the same conclusion). To focus on the core idea, we use the row Gram formulation throughout the following derivation and assume that \(C_t\) is non-singular. In the singular case arising from a rank-deficient gradient, the Moore–Penrose pseudoinverse can be used wherever the derivation requires \(C_t^{-1}\). The main conclusion remains valid, and we omit the associated technical details here. The row-wise proximal function defined by \(C_t\) is given by:

It is worth noting that this proximal function is not fixed, but rather time-varying, with its geometry induced by the gradient \(G_t\) and evolving dynamically throughout the iterations. With OMD constructed based on this proximal function, matrix orthogonalization can be naturally derived. Before giving the specific derivation, let us first understand what geometric information is encoded by the Gram structure.

Understanding \(G_t G_t^\top\)

The matrix \(G_t G_t^\top\) is the Gram matrix composed of the row vectors of \(G_t\):

Its diagonal entries measure the energy of individual rows, while its off-diagonal entries encode the correlations between different rows. In this way, the Gram matrix compresses information about the strength and mutual similarity of the row gradients into a single symmetric matrix, naturally describing the geometry induced by \(G_t\) in row space.

Let \(G_t = U_t \Sigma_t V_t^\top\). Using \(V_t^\top V_t = I\), we immediately obtain:

Taking its principal square root and expanding via spectral decomposition, we get:

where \(r\) is the rank of the matrix. This decomposition makes the geometry of \(C_t\) transparent: the left singular vectors \(u_{t,1}, \dots, u_{t,r}\) are mutually orthogonal and satisfy \(C_t u_{t,k} = \sigma_{t,k}\, u_{t,k}\). Thus, along each direction \(u_{t,k}\), the action of \(C_t\) decouples, scaling independently along that direction. Hence, the left singular vectors provide the orthogonal principal directions (principal coordinate system) of the metric defined by \(C_t\), and the corresponding singular values \(\sigma_{t,1}, \dots, \sigma_{t,r}\) determine the metric scales along each direction—larger singular values mean that a displacement of the same magnitude along that direction incurs a greater cost in the corresponding Bregman divergence.

The special role of \(C_t\) becomes even clearer when we compare it with \(G_t\): \(G_t\) generally rotates directions, whereas \(C_t\) is pure stretching without any rotation. In fact, the two differ precisely by a pure rotation matrix:

where \(U_t V_t^\top\) has orthonormal rows, i.e., is row-orthogonal, and becomes orthogonal when \(m = n\). That is, the action of \(G_t\) can be decomposed into two steps: first a rotation given by \(U_t V_t^\top\), followed by pure stretching along the principal directions of \(C_t\). Therefore, \(C_t\) precisely extracts the stretching component from \(G_t\) while completely stripping away the rotation.

From Proximal Geometry to Orthogonalization

The unconstrained Online Mirror Descent induced by the proximal function \(\psi_t(X)\) defined by \(C_t\) is:

where \(B_{\psi_t}(X; X_t) = \tfrac{1}{2} \langle X - X_t,\, C_t (X - X_t) \rangle_F\).

Taking the first-order derivative with respect to the objective function and setting it to zero yields the optimality condition:

Thus, the OMD update can be written as:

Using \(C_t = (G_t G_t^\top)^{1/2} = U_t \Sigma_t U_t^\top\), we have:

Substituting the SVD of the gradient \(G_t = U_t \Sigma_t V_t^\top\), we obtain:

At this point, we have naturally derived the orthogonalization operation from the unconstrained OMD update. Note that what we obtain here is the orthogonalization of the gradient \(G_t\); to exactly match Muon's orthogonalization, we simply replace \(G_t\) with the momentum matrix \(M_t\). Furthermore, a complete parallel derivation can be carried out based on the column Gram matrix \(G_t^\top G_t\), which we leave to interested readers.

Singular Value Scale Cancellation Mechanism

Let us now examine more closely how singular value scales are completely canceled out. This cancellation mechanism is particularly transparent in the principal coordinate system formed by the left singular vectors. \(C_t\) acts as the "metric" (Proximal Metric) in the Bregman divergence, determining the cost required to move along different principal directions (left singular vectors). Comparing it with the spectral scale of the gradient itself, we see that:

• The singular value scale of the gradient \(G_t\) is given by \(\Sigma_t\);
• The metric matrix \(C_t = U_t\Sigma_t U_t^\top\) has the exact same scale along the same principal directions;
• The inverse metric \(C_t^{-1}\) in the OMD update introduces \(\Sigma_t^{-1}\), which exactly cancels out the singular value scale in the gradient, i.e., \(\Sigma_t^{-1}\Sigma_t=I\).

Ultimately, only \(U_tV_t^\top\) remains in the update, which is the orthogonal factor in the gradient's polar decomposition. Thus, the key to orthogonalization lies in the scale matching between the gradient and the metric matrix of the proximal function: the inverse metric precisely cancels the singular value scales of the gradient, making the update independent of the magnitudes of individual singular values. If the scales of the two do not match, the corresponding singular value information will be retained in the update.

Beyond Muon: From Orthogonalization to General Spectral Transformations

The preceding derivations show that Muon's orthogonalization can be interpreted as the result of an OMD update under a specific proximal geometry. This perspective also inspires a more general design principle:

By choosing different matrix proximal geometries, different transformations of the gradient's singular values can be achieved.

Power Metric: A Continuously Adjustable Family of Spectral Transformations

First, consider generalizing the metric matrix to:

Since \(G_t G_t^\top = U_t \Sigma_t^2 U_t^\top\), we have:

The corresponding OMD update is:

By adjusting \(\alpha\), we can alter the scaling of singular values in the update. Several special values correspond to different spectral transformations:

When \(\alpha = 0\), the update preserves the original gradient; as \(\alpha\) increases from \(0\) to \(\tfrac{1}{2}\), the relative differences among singular values gradually narrow; when \(\alpha = \tfrac{1}{2}\), all non-zero singular values are normalized to \(1\), yielding the \(U_t V_t^\top\) update of Muon. When \(\alpha > \tfrac{1}{2}\), the transformation enters the negative power region, giving smaller singular values larger relative weights; in particular, \(\alpha = 1\) corresponds to inverting the singular values.

As the analysis above shows, varying the exponent \(\alpha\) yields a continuously tunable family of spectral transformations, of which Muon's orthogonalization is merely a special case. This naturally raises the question: Is the orthogonalization corresponding to \(\alpha = \frac{1}{2}\) truly optimal? Can other powers, or even more general spectral transformations, yield better optimization performance? This is a fundamental question that warrants further investigation.

Selective Transformations of Specific Singular Values

The power transformation above treats all singular values uniformly. We can further relax this restriction to design different scaling schemes for different singular values. For example, we can apply special treatment solely to the largest singular value while normalizing all remaining singular values to \(1\). Let:

We can specify the target transformation:

where \(s(\sigma_1)\) controls the magnitude retained by the largest singular value in the update. The corresponding update direction is:

When \(s(\sigma_1) = 1\), all singular values are normalized, degenerating to standard Muon orthogonalization; when \(s(\sigma_1) \neq 1\), specific magnitude information of the largest singular value is retained while the remaining singular values are normalized. Such a design allows targeted adjustments to the dominant spectral directions while preserving the overall orthogonalization structure.

Unified Form: Arbitrary Spectral Transformations

From a more general perspective, the above designs can be unified as:

where \(h\) acts on each singular value, with the requirement \(h(\sigma_i) > 0\). At this point, the gradient transformation in the OMD update is:

The power form \(h(\sigma) = \sigma^{2\alpha}\) is a simple and intuitive special case; whereas piecewise functions or rules defined for specific singular values allow us to construct more flexible spectral transformation methods.

Therefore, Muon's orthogonalization is not the sole outcome produced by this framework, but rather a special case with distinct geometric properties. From general power scaling to selective transformations for specific singular values, the proximal geometry under matrix Mirror Descent provides a unified perspective for designing a broader class of spectral-transformation-based optimization methods.

@misc{WpMirror2026,
  author = {Wenpeng Zhang},
  title  = {Deriving Muon From Mirror Descent: The Proximal Geometry Behind Orthogonalization},
  year   = {2026},
  url    = {https://your-blog-url.example/muon-mirror-descent}
}