This note is a companion to Derive Nesterov Momentum Method. There we observed that the lookahead structure \(y_k = w_k + P_k\) does not require \(P_k\) to be the linear inertia \(\mu_k(w_k - w_{k-1})\); any predictor \(P_k(w_k, w_{k-1}, \dots)\) that approximates the next displacement preserves the lookahead character. This note answers the natural follow-up: how should one actually design \(P_k\)?

What “designing an inertia estimator” means

Formally, we are choosing a functional \(P_k\) that maps the recent trajectory \((w_k, w_{k-1}, \dots)\) to a vector approximating the next displacement \(w_{k+1} - w_k\). A good design balances four criteria, each of which controls a specific failure mode of the resulting algorithm.

Consistency

\(P_k\) should approximate the true displacement \(w_{k+1}-w_k\) to a controlled order in the step size \(h\). The simplest consistent choice is

\[P_k = w_k - w_{k-1},\]

which matches the true velocity to first order, \(P_k = h\,\dot{w}_{t_k} + \mathcal{O}(h^2)\). Higher consistency is obtained by polynomial extrapolation in time. Treating \(\{w_{k-j}\}_{j\ge 0}\) as samples of a smooth curve, the unique polynomial through the last \(m+1\) points predicts the next sample as

\[P_k^{(m)} = \sum_{j=0}^{m} c_j^{(m)}\,(w_{k-j} - w_{k-j-1}),\]

with coefficients \(c_j^{(m)}\) chosen to cancel error terms up to \(\mathcal{O}(h^{m+1})\). The case \(m=1\) recovers the second-order Adams-Bashforth velocity

\[P_k^{(1)} = \tfrac{3}{2}(w_k - w_{k-1}) - \tfrac{1}{2}(w_{k-1} - w_{k-2}).\]

Stability

Higher-order predictors amplify noise: their leading coefficients grow, so a small perturbation in any past iterate can be magnified by the predictor and create spurious oscillation. A good design therefore caps either the order \(m\) or the spectral radius of the recurrence matrix that \(P_k\) defines.

A common compromise is an exponentially weighted moving average,

\[P_k = (1-\beta)\sum_{j=0}^{\infty} \beta^{j}\,(w_{k-j}-w_{k-j-1}), \qquad \beta\in[0,1),\]

which stays first-order consistent (its expectation is the velocity) but heavily damps high-frequency noise. This is exactly the velocity buffer used by Adam, RMSProp-momentum and most deep-learning optimizers.

Adaptivity

The coefficient \(\mu_k\) in the linear predictor can itself depend on the trajectory. The classical Nesterov schedule \(\mu_k=\tfrac{k-1}{k+2}\) is asymptotically optimal for smooth convex \(f\), but it ignores local geometry. Two practical refinements:

  • Restart-based adaptation (O’Donoghue and Candès): reset \(\mu_k\to 0\) whenever the inner product \(\langle \nabla f(y_k),\,w_{k+1}-w_k\rangle > 0\), that is, whenever the momentum has begun to fight the gradient. This keeps the predictor consistent with the current descent direction at essentially zero extra cost.
  • Curvature-aware adaptation: choose \(\mu_k\) so that the predicted point matches a target proximal subproblem, for example \(\mu_k = (1-\sqrt{\eta\, m})/(1+\sqrt{\eta\, m})\) when \(f\) is \(m\)-strongly convex. The predictor adapts to the conditioning of \(f\) instead of using a one-size-fits-all schedule.

Predictability of the lookahead point

For the lookahead gradient to be useful, \(P_k\) must be both small enough that \(y_k=w_k+P_k\) stays in the region where the linear approximation

\[\nabla f(y_k)\approx \nabla f(w_k)+\nabla^2 f(w_k)\,P_k\]

is meaningful, and large enough that the difference \(\nabla f(y_k)-\nabla f(w_k)\) is informative rather than dominated by noise. This puts a soft scale constraint on \(\|P_k\|\), often enforced through trust-region clipping

\[P_k \leftarrow P_k\cdot \min\!\big(1,\,r_k/\|P_k\|\big),\]

for some adaptive radius \(r_k\) tied to \(\|\nabla f(w_k)\|\).

A unified recipe

Combining the four criteria gives a template that subsumes most lookahead methods used in practice:

\[\begin{aligned} \tilde v_k &= \beta\,\tilde v_{k-1} + (1-\beta)(w_k - w_{k-1}),\\ P_k &= \mu_k\,\tilde v_k\cdot \min\!\big(1,\,r_k/\|\tilde v_k\|\big),\\ y_k &= w_k + P_k,\\ w_{k+1} &= y_k - \eta_k\,\nabla f(y_k), \end{aligned}\]

where \(\beta\) controls smoothing, \(\mu_k\) controls the look-ahead distance, and \(r_k\) enforces trust-region scale. Different settings recover familiar schemes:

  • \(\beta=0\), \(\mu_k=\tfrac{k-1}{k+2}\), \(r_k=\infty\): classical Nesterov.
  • \(\beta>0\), \(\mu_k\) constant, \(r_k=\infty\): Adam-style momentum buffer.
  • Classical Nesterov with the restart rule on \(\mu_k\): O’Donoghue-Candès restarted Nesterov.
  • \(\mu_k\) replaced by a learned slow-update averaging step: Lookahead-SGD.

Why linear inertia still dominates

The reason the simplest linear form \(\mu_k(w_k - w_{k-1})\) remains the default in both theory and practice is that it sits at the most favorable point of these trade-offs at once: cheapest memory (one previous iterate), cleanest continuous-time limit (a second-order ODE, exactly the ISHD discussed in the companion note), and provably optimal worst-case rate on smooth convex problems. Nonlinear or higher-order predictors are useful precisely when at least one of these constraints can be relaxed, for instance when extra memory is acceptable, when the noise statistics demand stronger smoothing, or when local curvature information is available cheaply.