Backpropagation: calculate gradients, then update weights
“AI learns from mistakes” is an intuitive phrase, but the technical question is how a mistake becomes a calculation. Backpropagation computes gradients of the loss with respect to parameters; an optimizer performs the parameter update. These consecutive steps are often conflated.
➡️ 1. Start with a prediction
Consider a tiny model: prediction = w * x. Let the input be x = 2, the target y = 6, and the initial weight w = 1. The model predicts 2 instead of 6.
Choose the loss L = (prediction - y)^2. Initially, L = (2 - 6)^2 = 16. This is a one-parameter example you can calculate by hand, not a complete chatbot simulation.
📉 2. What does a gradient tell us?
We want to know how a small change in w affects the loss at the current point. The chain rule gives dL/dw = 2 * (w*x-y) * x, which equals -16.
The negative sign indicates that a small increase in w will reduce the loss locally. It does not assign “16 units of blame” to a neuron or guarantee that a large increase will help.
For a network of operations, backpropagation applies the chain rule across the computation graph. PyTorch implements this through autograd. Autograd documentation.
⚙️ 3. Updating is a separate step
With gradient descent and learning rate lr = 0.1, the new weight is w_new = w - lr * grad = 2.6. The new prediction is 5.2 and the loss becomes 0.64.
In PyTorch, loss.backward() calculates gradients and optimizer.step() uses them to adjust parameters. Gradients must also be managed or cleared appropriately between iterations. PyTorch's optimization loop.
🧪 4. Run the example in Python
This code uses basic arithmetic and requires no machine-learning library:
x, y, w, lr = 2.0, 6.0, 1.0, 0.1
prediction = w * x
loss = (prediction - y) ** 2
grad = 2 * (prediction - y) * x
w_new = w - lr * grad
new_prediction = w_new * x
new_loss = (new_prediction - y) ** 2
print("prediction:", prediction)
print("loss:", loss)
print("gradient:", grad)
print("new weight:", w_new)
print("new loss:", round(new_loss, 4))
The outputs are 2.0, 16.0, -16.0, 2.6 and 0.64. Change only lr to 1.0: the weight jumps to 17 and the loss rises to 784. A correct gradient can still accompany an excessively large update.
🔎 Check with finite differences
Evaluate the loss at w + epsilon and w - epsilon, subtract them and divide by 2 * epsilon. With epsilon = 0.000001, the result is close to -16. This checks the derivative in a small example; it is not a replacement for backpropagation in a large network.
💬 5. Editing a prompt is different from training weights
In this example, learning is represented by changing w. Adding instructions to a chat changes its input context; that alone does not establish that the model's weights were updated.
When a product says it “learns from you,” ask whether it stores memory, retrieves documents or runs a training process. These are distinct mechanisms that deserve separate explanations.
✅ 6. Low loss is not enough
Our example contains only one data point. Fitting it well does not establish performance elsewhere. To check your understanding, set x = 3 and target y = 9, then calculate the predictions before and after the update.
For a real model, reserve separate data for evaluation. Track both the update and performance on data not used for learning, rather than only watching training loss fall.