Backpropagation: Sending the Error Backward

Where We Are

We just learned Backpropagation: Sending the Error Backward.

But using it creates a new question.

What problem does this idea still leave us unable to solve?

That question leads naturally to Matrix Backpropagation: dW, db, dX by Hand.

Next → Matrix Backpropagation: dW, db, dX by Hand.

1. The Problem: The First Layer Cannot See the Final Error

Consider the smallest useful two-stage network:

with

Suppose the target is $y$ and the loss is

The second weight $v$ touches the prediction directly. The first weight $w$ does not.

So here is the training problem:

How can $w$ know whether increasing it would make the final loss better or worse?

The answer cannot simply be “look at the loss.” The loss is several operations away.

2. What Would a Solution Need?

We need a method that:

1. starts at the final loss;
2. moves backward one operation at a time;
3. measures the local effect of each operation;
4. combines those local effects into the total effect on the loss;
5. works for deeper networks without inventing a new rule for every architecture.

Notice the shape of the answer already: local pieces + a rule for connecting them.

3. First Attempt: Differentiate the Whole Expression at Once

Substitute everything into the loss:

Then expand it mentally and differentiate directly with respect to $w$.

For this tiny example, you can do it:

But this approach becomes ugly when the network contains many layers and nonlinear functions.

Worse, the long expression hides the structure we actually need: every operation only has to answer a tiny local question.

A Tempting Wrong Idea

“Just expand the whole neural network into one giant formula and differentiate that.”

You can sometimes do this on paper. But the expression grows with the network, repeated computation becomes wasteful, and it becomes difficult to see which local operation caused which part of the gradient.

We need to preserve the computation as a sequence of small operations.

4. The Discovery: Follow One Tiny Effect Backward

Start at the loss.

To understand how $w$ affects $L$, ask three smaller questions:

1. How does $L$ change when $\hat y$ changes?
2. How does $\hat y$ change when $z$ changes?
3. How does $z$ change when $w$ changes?

The chain rule says we can multiply those local answers:

This is the central discovery of the chapter.

Backpropagation is not a new kind of derivative. It is the systematic organization of the chain rule over a computation graph.

5. Three Levels of the Same Idea

The chain rule turns many small sensitivities into one global sensitivity.

6. Calculate Every Local Derivative

Our computation is

First:

so

Second:

so

Third:

so

Now multiply them:

The long derivative came from three tiny derivatives.

7. ✏️ Hand Calculation: The Error Travels Back

Choose

Forward pass

First operation:

Second operation:

Loss:

Backward pass

Start from the loss:

Move one step backward:

So

One more step:

Therefore

The negative sign is information: locally, increasing $w$ would decrease the loss.

8. Why “Backward”?

The forward computation is

input → layer 1 → layer 2 → prediction → loss

The derivative computation reverses direction:

loss → prediction → layer 2 → layer 1 → parameter

flowchart LR
    X[Input x] --> Z[ z = w x ]
    Z --> Y[ y-hat = v z ]
    Y --> L[ Loss L ]
    L -. dL/dy-hat .-> Y
    Y -. dL/dz .-> Z
    Z -. dL/dw .-> X

The backward arrows are gradient information, not data travelling backward through the network.

9. Computational Graphs: The Network as Small Operations

Write the same computation as a graph:

Every node performs a local operation. Every operation has a local derivative.

Backpropagation stores the intermediate values from the forward pass and then reuses them while moving backward.

That last part matters. A good algorithm should not repeatedly rebuild the same forward calculations just to differentiate them.

10. Local Derivatives Become a Reusable Vocabulary

Here are three tiny rules:

A complex network is then assembled from simple operations whose local derivatives can be chained together.

This is why the method scales conceptually even when the network becomes much larger.

11. The Subtle Part: A Variable Can Have Multiple Paths

So far $w$ had only one route to the loss.

Real computation graphs can branch.

Suppose

and then

The final quantity is

The variable $z$ influences $L$ through two paths.

The total derivative must add the contributions from both paths:

That gives us an important rule:

Along one path, multiply local derivatives. When several paths meet, add their contributions.

This is one reason computational graphs are a better mental model than one enormous formula.

12. Backpropagation Is Not the Optimizer

There are four separate jobs:

Forward pass       → calculate predictions
Loss               → measure wrongness
Backpropagation    → calculate gradients
Optimizer          → change parameters

The optimizer uses a gradient such as $dL/dw$:

Here $\eta$ is the learning rate.

With our gradient $-96$, a positive learning rate would increase $w$ because subtracting a negative number moves it upward.

Backpropagation does not choose the learning rate or perform the update. It supplies the direction and sensitivity information.

13. 🔬 The Experiment: Change One Variable

Keep

fixed. Change only $w$.

From

we can predict the gradient for several values of $w$.

At $w=3$, it is $-96$.

If $w$ increases toward the value that makes $vwx=30$, the error should shrink and the gradient should move toward zero.

The gradient is therefore not “the error.” It is a local sensitivity of the loss to the parameter.

14. Why This Idea Scales

Imagine a network with thousands of operations.

We do not want thousands of giant symbolic formulas.

Instead:

1. run the network forward once;
2. store the intermediate results we need;
3. start with the loss gradient;
4. move backward, applying local derivative rules;
5. reuse the same intermediate calculations where paths overlap.

The computation becomes systematic.

This is the conceptual foundation of automatic differentiation systems used by frameworks such as PyTorch.

15. 📜 History Lens — Backpropagation in the 1980s

Imagine working on multilayer networks when the major difficulty is not writing a neuron, but finding a practical way to train all the hidden weights.

David Rumelhart, Geoffrey Hinton and Ronald Williams published the influential 1986 paper Learning representations by back-propagating errors. Their work helped establish the practical use of backpropagation for multilayer neural networks.

The chain rule itself was much older. The contribution was showing how to organise repeated chain-rule calculations into a usable learning procedure for layered models.

That distinction matters: backpropagation is an algorithmic way of applying old calculus to a new computational structure.

🎯 Machine Learning Connection

At training time, a neural network repeatedly executes

The backward stage computes quantities such as

The optimizer then uses them to move parameters.

Once we have many weights and many examples, these scalar derivatives naturally become arrays of derivatives. That is the exact problem waiting in Chapter 085.

Distinctions That Matter

What We Discovered

1. A parameter can affect the loss through a chain of intermediate computations.
2. The chain rule turns local sensitivities into a total derivative.
3. Backpropagation is the systematic application of that rule backward through a computational graph.
4. Along a single path, derivatives multiply.
5. When paths merge, their gradient contributions add.
6. Backpropagation computes gradients; an optimizer uses them to change parameters.
7. The same idea can be packed into matrix operations when many values are involved.

Mathematics We Built

Network:

Chain rule:

Local derivatives:

Therefore:

Parameter update:

What Each Symbol Means

One-Minute Explanation

Imagine a chain of machines. The first machine uses a weight to turn an input into an intermediate value. The next machine turns that into a prediction. The loss tells us how wrong the final prediction is.

Backpropagation starts at the loss and walks backward. At each machine it asks: “If my input changed a little, how much would my output change?” It multiplies that local answer by the amount of error arriving from downstream.

If there are several paths, their contributions are added.

So backpropagation is simply the chain rule organised as an algorithm for a computation graph.

Exercises

Level 1 — Observe

Look at the graph $w\rightarrow z\rightarrow\hat y\rightarrow L$. Which quantity is furthest upstream from the loss?

Level 2 — Calculate

For $x=3$, $v=2$, $\hat y-y=-4$, compute $dL/dw=2(\hat y-y)vx$.

Level 3 — Derive

Starting from $L=(vwx-y)^2$, derive $dL/dw$ using the chain rule without expanding the square first.

Level 4 — Investigate

In the notebook, change only $w$ and plot the loss and gradient. Predict where the gradient becomes zero before running the code.

Level 5 — Design

Build a three-stage scalar computation with at least one branching path. Derive the gradient to the first parameter by multiplying along paths and adding where paths merge.

Common Mistakes

Socratic Questions

Why does a weight far from the output still receive a useful learning signal?

Why do local derivatives multiply along one path?

Why must gradients from two branches be added?

Why would differentiating one giant expanded expression become difficult to maintain?

What changes when one scalar parameter becomes an entire weight matrix?

🔭 Bridge to Chapter 085

Our scalar example is complete. But real layers do not have one input, one output and two weights.

A dense layer may have thousands of weights. Every training example contributes to those weights. Writing one scalar derivative after another would hide the repeated structure we just discovered.

Next question: Can we take the same chain-rule reasoning and pack all those repeated scalar derivatives into three matrix formulas — one for the weights, one for the bias, and one for the input?

Yes. The next chapter derives