This note views first-order optimizers as gradient flows derived from the Onsager variational principle (OVP), and shows how different choices of the dissipation functional reproduce gradient descent, preconditioned/natural gradient, sign descent, spectral-norm steepest descent (Muon-style), and—after adding a kinetic term—Polyak’s Heavy-Ball method. Companion notes: Derive Nesterov Momentum Method and Designing the Inertia Estimator for Lookahead Methods.

Onsager variational principle in one slide

Let \(w\) be the state and \(f(w)\) the objective playing the role of free energy. Onsager’s principle prescribes the dynamics by minimizing the Rayleighian

\[\mathcal{R}(\dot w; w) \;=\; \dot f(w) \;+\; \Phi(\dot w; w),\]

over the velocity \(\dot w\), where \(\dot f(w)=\langle \nabla f(w),\dot w\rangle\) is the rate of energy release and \(\Phi(\dot w; w)\ge 0\) is the dissipation functional, assumed convex in \(\dot w\) with \(\Phi(0;w)=0\). The Euler-Lagrange condition \(\partial\mathcal{R}/\partial\dot w=0\) reads

\[\nabla f(w) \;+\; \partial_{\dot w}\Phi(\dot w; w) \;=\; 0,\]

so the velocity is the (sub)gradient of the convex conjugate of \(\Phi\) evaluated at \(-\nabla f(w)\):

\[\dot w \;\in\; \partial_{g}\Phi^{*}(-\nabla f(w); w).\]

Picking \(\Phi\) thus selects an optimizer; integrating the resulting ODE and discretizing in time gives a concrete algorithm.

Quadratic dissipation: gradient descent

The simplest choice is the isotropic quadratic dissipation

\[\Phi(\dot w) = \tfrac{1}{2}\|\dot w\|_2^{2}.\]

Onsager’s stationarity equation yields

\[\nabla f(w)+\dot w=0 \;\Longrightarrow\; \dot w=-\nabla f(w),\]

i.e. the standard gradient flow. Forward Euler with step \(\eta\) recovers gradient descent

\[w_{k+1}=w_k-\eta\,\nabla f(w_k).\]

Preconditioned dissipation: natural / Newton flow

Replace the Euclidean norm by a state-dependent metric. With

\[\Phi(\dot w; w) = \tfrac{1}{2}\,\dot w^\top M(w)\,\dot w,\qquad M(w)\succ 0,\]

stationarity gives

\[\nabla f(w)+M(w)\,\dot w=0 \;\Longrightarrow\; \dot w=-M(w)^{-1}\nabla f(w).\]

Choosing \(M(w)=\nabla^2 f(w)\) recovers continuous-time Newton flow; choosing \(M(w)\) as the Fisher information metric recovers natural gradient descent. Forward Euler with step \(\eta\) produces

\[w_{k+1}=w_k-\eta\,M(w_k)^{-1}\nabla f(w_k),\]

the preconditioned gradient method.

Non-quadratic dissipation: dual-norm steepest descent

When \(\Phi\) is not quadratic, the velocity is no longer linear in the gradient. Take a positively homogeneous-of-degree-two dissipation built from an arbitrary norm \(\|\cdot\|\),

\[\Phi(\dot w) = \tfrac{1}{2}\|\dot w\|^{2}.\]

Stationarity in \(\dot w\) gives

\[\dot w \;\in\; -\partial\Big(\tfrac{1}{2}\|\cdot\|^{2}\Big)^{*}\!\big(\nabla f(w)\big) = -\tfrac{1}{2}\,\partial\big(\|\nabla f(w)\|_{*}^{2}\big),\]

where \(\|\cdot\|_*\) is the dual norm. Equivalently, \(\dot w\) aligns with the unit ball direction that maximizes \(\langle \nabla f(w), -\dot w\rangle/\|\dot w\|\), scaled by \(\|\nabla f(w)\|_*\). Discretization yields steepest descent in the norm \(\|\cdot\|\):

\[w_{k+1}=w_k+\eta\,\Delta_k,\qquad \Delta_k=\arg\min_{\|\Delta\|\le 1}\langle \nabla f(w_k),\Delta\rangle.\]

Three concrete cases:

  • \(\|\cdot\|=\|\cdot\|_2\): \(\Delta_k\propto -\nabla f(w_k)\), recovers gradient descent.
  • \(\|\cdot\|=\|\cdot\|_\infty\) (so \(\|\cdot\|_*=\|\cdot\|_1\)): \(\Delta_k=-\mathrm{sign}(\nabla f(w_k))\), recovers sign descent / sign-SGD used in distributed training and signSGD-type compression.

  • Spectral / operator norm on matrices: for a matrix-valued parameter \(W\in\mathbb{R}^{m\times n}\) with \(\|\cdot\|=\|\cdot\|_{\sigma}\) (largest singular value), the steepest direction is \(\Delta_k=-U V^\top\) where \(\nabla f(W_k)=U\Sigma V^\top\) is the SVD. The update

    \[W_{k+1}=W_k-\eta\,U_k V_k^\top\]

    is exactly the spectral-norm steepest descent that underlies Muon. Different dissipation norms thus give qualitatively different optimizers, all sharing the OVP skeleton.

A unified summary: for \(\Phi=\tfrac12\|\dot w\|^2\) the velocity is

\[\dot w = -\arg\max_{\|\Delta\|\le 1}\langle \nabla f(w),\Delta\rangle\cdot \|\nabla f(w)\|_*,\]

which directly maps the choice of norm to the choice of optimizer.

Adding inertia: derivation of Heavy-Ball

OVP can also be extended to include a kinetic energy. Augmenting the Rayleighian by a kinetic term gives

\[\mathcal{L}(w,\dot w) = \tfrac{1}{2}\,m\|\dot w\|_2^{2} - f(w), \qquad \Phi(\dot w) = \tfrac{1}{2}\,\gamma\|\dot w\|_2^{2},\]

with mass \(m>0\) and damping \(\gamma>0\). The Euler-Lagrange equation with dissipation reads

\[\frac{d}{dt}\frac{\partial \mathcal{L}}{\partial \dot w}-\frac{\partial \mathcal{L}}{\partial w}+\frac{\partial\Phi}{\partial\dot w}=0,\]

which gives the dissipative Newton equation

\[m\,\ddot w + \gamma\,\dot w + \nabla f(w) = 0.\]

This is the standard inertial gradient flow, the prototype for all momentum methods. Setting \(m=1\) for simplicity, discretize with the symplectic-Euler / leapfrog scheme

\[\begin{aligned} \dot w_{k+1} &= \dot w_k - h\big(\gamma\,\dot w_k + \nabla f(w_k)\big),\\ w_{k+1} &= w_k + h\,\dot w_{k+1}. \end{aligned}\]

Introducing the discrete velocity \(v_k = h\,\dot w_k\) and grouping coefficients, the first equation becomes

\[v_{k+1} = (1-h\gamma)\,v_k - h^{2}\nabla f(w_k),\]

and the second reads \(w_{k+1}=w_k+v_{k+1}\). Defining

\[\mu = 1-h\gamma,\qquad \eta = h^{2},\]

we recover Polyak’s Heavy-Ball method exactly:

\[\begin{aligned} v_{k+1} &= \mu\,v_k - \eta\,\nabla f(w_k),\\ w_{k+1} &= w_k + v_{k+1}. \end{aligned}\]

Two observations:

  • The momentum coefficient \(\mu=1-h\gamma\) is set by the dissipation strength: weaker dissipation (smaller \(\gamma\)) yields more momentum, which matches the physical intuition that an under-damped particle keeps its velocity longer.
  • The step size \(\eta=h^{2}\) scales with the square of the discrete time step. This is why naive Heavy-Ball requires retuning \(\eta\) when the temporal resolution changes, and also why the Nesterov correction (which adds Hessian damping at the ODE level) is needed for stability under larger \(h\). The continuous-time picture behind that correction is the ISHD discussed in the Nesterov note.

Choosing a non-quadratic kinetic term—say \(\tfrac{1}{2}\|\dot w\|^2\) with a non-Euclidean norm—repeats the inertial derivation in that geometry, producing momentum methods adapted to sign descent or spectral steepest descent, in direct analogy with the optimizer family above.

Take-aways

  • OVP packages “what the optimizer is” into one object: the dissipation \(\Phi\).
  • Quadratic Euclidean \(\Phi\) gives gradient descent; preconditioned \(\Phi\) gives natural / Newton; non-Euclidean \(\Phi\) gives sign-SGD or Muon-style spectral steepest descent.
  • Adding a kinetic term turns first-order flows into second-order dynamics; symplectic discretization of the resulting damped Newton equation reproduces Heavy-Ball with \(\mu=1-h\gamma\) and \(\eta=h^{2}\).
  • The same recipe with state-dependent dissipation \(\beta_t\nabla^2 f(w)\dot w\) yields ISHD, whose discretization gives Nesterov-type lookahead momentum—covered in the companion note.