Learn · Gradient Descent
Gradient descent, visualised
Optimisation is how models learn: nudge the parameters in the direction that lowers the loss, over and over. Here a point rolls across a 2-D loss surface using the real analytic gradient. Change the learning rate and watch it converge smoothly, oscillate, or diverge.
Brighter = lower loss. The dot starts at the marked spot and steps downhill along the gradient.
Small = slow but stable. Large = fast, then it overshoots.
The surface is f(x, y) = 0.5(x² + y²) − 0.8·e−((x−1)²+(y+1)²) over x, y ∈ [−3, 3] — a bowl with an extra dip near (1, −1). Steps use the exact gradient ∇f = (x + 1.6(x−1)E, y + 1.6(y+1)E).
What the learning rate does
Each step moves the point by −(learning rate) × gradient. Too small and it crawls toward the minimum; just right and it glides in; too large and it overshoots the valley, bouncing side to side (oscillation) or flying off entirely (divergence). Try 0.05, then 0.5, then 0.95 to feel the difference.
Caveat: this is a hand-picked 2-D surface for intuition. Real models optimise millions of parameters over noisy mini-batches, with momentum and adaptive rates (Adam, etc.) — but the core idea is exactly this. More on the Learn shelf.