Oct 30, 2026
Attention
QK Parameterization
Self-interaction
Although mathematically sound, the standard Transformer attention parameterization, based on $W_Q W_K^\top$,
may introduce structural friction during gradient-based optimization. Its skew-symmetric component contributes
identically zero in token self-interactions, while the symmetric component must simultaneously accommodate both
self- and cross-token interactions, potentially giving rise to implicit multi-objective gradient conflicts.
In this work, we analyze this structural bottleneck and propose a lightweight reparameterization that explicitly
decouples self- and cross-token interactions. By assigning these interactions to separate parameter sets,
our approach avoids such gradient interference by construction, leading to improved optimization efficiency.
Sep 26, 2026
Muon
Conflicting Spectral Directions
Cautious Weight Decay
Decoupled weight decay can conflict with the optimizer’s descent direction, while entry-wise
Cautious Weight Decay fails to resolve such conflicts for matrix-aware optimizers like Muon,
whose orthogonalized updates carry a spectral structure. We address this mismatch through a
geometric formulation that characterizes spectrally compatible decay components as a polar
factor cone and extracts the nearest compatible component via Euclidean projection to this cone with a
closed-form solution. The resulting Spectral-Aware Cautious Weight Decay is a drop-in modification
to Muon that structurally precludes spectral conflicts and enables full directional alignment
at the spectral level.
Aug 25, 2026
Optimization
Muon
Mirror Descent
Muon’s orthogonalization has traditionally been understood through constrained optimization, such as Frank-Wolfe under spectral norm constraints.
Yet Muon itself is an unconstrained optimizer. This post develops an alternative derivation from unconstrained Online Mirror Descent,
showing how orthogonalization emerges naturally from a gradient-induced proximal geometry defined by the gradient’s Gram matrix.
Under this geometry, the inverse metric cancels the gradient’s singular values, leaving only its orthogonal factor. This derivation
further reveals that orthogonalization is a special case \(\alpha = \frac{1}{2}\) within a continuous family of spectral transformations.