This is where you get to see how a neural network actually learns. We follow one training example all the way around the loop and watch the error travel backwards through the network, nudging every weight a little.
Pick a logic gate. The network has to learn to give the right answer for all four combinations of the inputs x₁ and x₂.
Watch the network learn, step by step – from a random guess to a fully trained model.
SvenskaAt every calculation step z, a, ŷ and L are concrete numbers – results worked out for a given input and given weights. What gets differentiated, though, is not those individual numbers but the relationship between them: the formula describing how the loss L would change if a weight w were altered slightly.
Hold every other value fixed and let w vary, and a curve appears. The derivative is the slope of that curve at exactly the point where the weight currently sits – not the height, but how steep it is there.
The network knows the equation of that curve, because it is built only from known, differentiable steps (weighted sums and sigmoid). So the slope can be worked out exactly, by plugging the current numbers into the finished formula – instead of guessing.
The chain rule multiplies these slopes together, step by step backwards through the network – which is exactly what Phase 3 · Backprop does.
The steeper the curve, the more sensitive the loss is to changes in the weight. Right at the bottom of the valley the slope is zero – there is no way to reduce the loss further by moving the weight in either direction. So it is at the bottom of the curve, where the derivative is zero, that the loss is at its smallest – and that is where gradient descent tries to take the network, one step at a time.