Blek NgossangaSoftware Engineering · AI Vision · Research

FR, Lire cette page en français

English interface, partly translated: navigation, lab, projects, series, about and contact are; the home page sections, the site search and the content itself (articles, tutorials, research) are still in French.

Interactive lab

Optimizers: gradient, momentum, Adam

Three optimizers start from the same point on a two-variable surface. Compare their trajectories over the level curves: zigzag, momentum, adaptive step.

Level
Intermediate
Topics
Momentum · Adam · Lignes de niveau
−4 −2 0 2 4 −2 −1 0 1 2
Figure 1 — Level curves of f and the trajectory of each optimizer. Crosses mark the minima. Step 0. Drag the starting point on the map, or use the x₀ and y₀ sliders.
  • Gradient descent
  • Momentum
  • Adam
10^-8 10^-6 10^-4 10^-2 10^0 10^2 0 20 40 60 80 100 120 step t
Figure 2 — f(θₜ) at each step, logarithmic scale: a descending straight line is geometric convergence.
Surface
Compared optimizers
0.09
0.02
0.3
0.9
Starting point
−4
1.5
120
20
Readout at the displayed step (t = 0)
optimizerηf(θ)distance to minimum
Gradient descent0.09 30.5 4.272
Momentum0.02 30.5 4.272
Adam0.3 30.5 4.272
f(x,y)=12x2+10 y2f(x, y) = \tfrac{1}{2}x^2 + 10\,y^2

Three ways to walk downhill

On a two-variable surface, the gradient ∇f(θ)\nabla f(\theta) is a vector pointing towards the steepest ascent. The three optimizers use it differently.

Gradient descent

θt+1=θt−η ∇f(θt)\theta_{t+1} = \theta_t - \eta\,\nabla f(\theta_t)

Each step follows the local slope, with no memory. In a narrow valley, the slope is steep across and gentle along the floor: the point zigzags from one wall to the other and crawls towards the minimum. A larger η\eta speeds up the floor but makes the cross direction diverge.

Momentum

vt+1=β vt+∇f(θt)θt+1=θt−η vt+1v_{t+1} = \beta\,v_t + \nabla f(\theta_t) \qquad \theta_{t+1} = \theta_t - \eta\,v_{t+1}

The velocity vv accumulates past gradients. Components that flip sign at every step (across the valley) cancel out; those that keep their sign (along the floor) add up. With β=0.9\beta = 0.9, a constant gradient is amplified up to ten times. The price: the trajectory may overshoot the minimum and spiral back.

Adam

mt=β1mt−1+(1−β1) gtvt=β2vt−1+(1−β2) gt2m_t = \beta_1 m_{t-1} + (1-\beta_1)\,g_t \qquad v_t = \beta_2 v_{t-1} + (1-\beta_2)\,g_t^2
θt+1=θt−η m^tv^t+ε\theta_{t+1} = \theta_t - \eta\,\frac{\hat m_t}{\sqrt{\hat v_t} + \varepsilon}

Adam divides each coordinate of the step by an estimate of the recent magnitude of its gradient (m^t\hat m_t and v^t\hat v_t are the averages corrected for their start-up bias). Every direction thus gets its own effective step, of order η\eta whatever the steepness: the narrow valley hardly bothers it.

What to look at

  • Narrow valley: the zigzag of plain descent, the drive of momentum, the nearly direct path of Adam.
  • Rosenbrock: all quickly reach the floor of the curved valley; the difference is made afterwards, on the almost flat floor.
  • Himmelblau: move the starting point. The optimizers do not always reach the same minimum: momentum can cross a ridge that plain descent cannot.

No optimizer is "the best": on figure 2, compare speed (slope of the curve) and stability (oscillations).

Other demos

  • K-Means step by step

    Artificial intelligence, Beginner

    Assign, update, assign… Follow the centroids until convergence, and discover where K-Means goes wrong: bad start, stretched clusters, rings.

  • Linear regression

    Artificial intelligence, Beginner

    Move the points and watch the least squares line follow. Then learn the same line by gradient descent and compare.

  • Convolution and filters

    Computer vision, Beginner

    Slide a 3 × 3 kernel over an image: blur, sharpen, edge detection. Edit the weights and follow the computation of one pixel, term by term.