Gradient descent

Picture the loss as a hilly landscape over all the weights. The gradient (the slope in every direction) points uphill, so take a small step the opposite way, downhill, and repeat. The step size is the learning rate: too big and you overshoot, too small and training crawls. This is how essentially every modern neural network learns.

Why it matters: This is the learning in machine learning.

Builds on: Slope and the derivative, Loss functions

Learn it, then prove it

Sign in (free) to take this step's mastery check and save your journey.