Matrix Backpropagation: dW, db, dX by Hand

Where We Are

We just learned Matrix Backpropagation: dW, db, dX by Hand.

But using it creates a new question.

What problem does this idea still leave us unable to solve?

That question leads naturally to Tensors: Numbers in Many Dimensions.

Next → Tensors: Numbers in Many Dimensions.

1. The Problem: One Neuron Was Too Small

Our previous network had

with one input and one weight. Real layers look more like this:

Suppose two students each have three scores — maths, science and English — and we want two outputs: a science-strength score and an overall readiness score.

For a batch of two students:

and let the layer have two outputs:

The forward pass is easy. But now ask the training question:

If the final loss changes, how much should every entry of $W$, every entry of $b$, and every input value in $X$ change?

There are already $6+2+6=14$ numbers involved. We need a rule that works for all of them, not fourteen separate tricks.

2. What Would a Solution Need?

A useful backward calculation must:

1. give one gradient for every weight;
2. give one gradient for every bias;
3. give one gradient for every input value;
4. preserve matrix shapes so a wrong formula can be caught before arithmetic;
5. reuse the same intermediate values rather than recomputing every path separately.

The last requirement is the reason backpropagation becomes efficient rather than a giant collection of repeated algebra.

3. First Attempt: Different Formula for Every Weight

One tempting strategy is to differentiate $L$ with respect to $w_{11}$, then $w_{12}$, then $w_{21}$, and so on.

For six weights this is annoying. For six million it is absurd.

A Tempting Wrong Idea

“Because every weight is different, every derivative needs a different derivation.”

No. The local computation is the same for every connection. The only thing that changes is which input and which downstream error signal meet at that connection.

So the real task is to discover the pattern.

4. The Discovery: One Weight Receives Two Pieces of Evidence

Start with a single entry of the matrix.

For one output neuron,

Pick one weight, $w_{ij}$.

Changing $w_{ij}$ changes $z_j$ in proportion to the corresponding input $x_i$:

Backpropagation also tells us how much the loss cares about $z_j$:

Chain rule joins the two pieces:

That is the key pattern:

weight gradient = input × downstream gradient.

The formula is simple because the computation is simple: a weight only acts by multiplying its input.

5. Three Levels of the Same Idea

Now the scalar pattern is visible. The only remaining question is how to write all those pairings at once.

6. Two Students Instead of One: Why a Matrix Appears

Return to our batch.

The output is

so

Suppose the gradient arriving from the next part of the network is

Read $G$ in plain English:

entry $(r,j)$ says how sensitive the loss is to output $Z_{rj}$.

Now inspect one weight gradient by hand.

For $w_{11}$, the first input of each student contributes:

For $w_{12}$:

Notice what happened: we multiplied input values by output gradients and added over the batch.

That is exactly matrix multiplication.

7. The Discovery: $dW = X^T dZ$

The transpose appears because we want the dimensions to line up.

and

which is exactly the shape of $W$.

So the whole weight gradient is

For our numbers:

therefore

Check the first column by hand:

The matrix formula did not appear from nowhere. It is just the scalar chain rule repeated and packed efficiently.

8. What About the Bias?

The bias is easier because

and

So every example simply contributes its downstream gradient.

For our batch:

Hence

If $dZ$ has shape $(B,m)$, then $db$ has shape $(m,)$. The sum is across the batch axis because each example touches the same shared bias.

Think — Why do we sum the bias gradient but not the weight gradient in exactly the same notation?

The weight already has an input-specific multiplier. The bias has no input. Every example pushes directly on the same bias value, so their effects add.

9. Now Reverse the Question: What Is $dX$?

A weight can ask, “How much did I affect the loss?”

The input asks the opposite:

“How much did the downstream layer care about me?”

Take one input entry $x_i$.

Because

we have

There are several output neurons, so their effects add:

Pack the sums into a matrix product:

Check the shapes:

which matches the shape of $X$.

Using our values:

so

Again, the shape is a proof-check. If someone wrote $W dZ$ here, the dimensions would be $(3\times2)(2\times2)=(3\times2)$ — wrong shape, wrong answer.

10. One Layer, Three Gradients

We have reached the compact backward rule for a dense layer:

The forward pass was

The backward pass mirrors it:

flowchart LR
    X[Input X] --> M[Z = XW + b]
    W[Weights W] --> M
    b[Bias b] --> M
    M --> L[Loss downstream]
    L -. dZ .-> BW[dW = Xᵀ dZ]
    L -. dZ .-> BB[db = sum dZ]
    L -. dZ .-> BX[dX = dZ Wᵀ]

This is the matrix version of the scalar chain rule from Chapter 083.

11. Shapes Catch Mistakes Before Arithmetic

Write the shapes first:

Then verify:

and

This is more than notation. Shape algebra is a debugging tool.

12. 🔬 The Experiment: Change One Number

Take the same layer and change only the first student’s first feature.

The first column of $dW$ tells you how that feature influences the corresponding weights. Predict:

What happens to $dW$ when $x_{11}$ doubles while everything else stays fixed?

The answer should not require running code. From

the contribution from the first sample doubles. Other samples are unchanged.

This is exactly what a derivative should mean: a local sensitivity with respect to one quantity while the others are held fixed.

13. History Lens — Rumelhart, Hinton and Williams

📜 History Lens — the 1980s

Imagine having the chain rule but no convenient matrix notation for a layered neural network. David Rumelhart, Geoffrey Hinton and Ronald Williams published the influential 1986 paper Learning representations by back-propagating errors, showing a practical procedure for propagating error derivatives through multilayer networks.

The important historical lesson is not that the chain rule was new. It was not. The breakthrough was turning a familiar mathematical idea into a systematic training method for layered representations.

Our matrix formulas are the same philosophy made explicit: reuse local derivatives and arrange the repeated arithmetic as matrix operations.

14. 🎯 Machine Learning Connection

Every dense layer in a neural network performs a transformation like

During training, the framework repeatedly needs exactly these three quantities:

The optimizer usually updates parameters using

The input gradient $dX$ is not normally updated by the optimizer. It is passed backward to the previous layer so that the previous layer can compute its gradients.

That is why deep networks can have dozens or hundreds of layers without requiring a new kind of mathematics for every layer.

Distinctions That Matter

A transpose changes how a matrix is arranged; it is not an inverse.

What We Discovered

1. Every scalar weight gradient is input × downstream gradient.
2. A matrix is a compact way to compute all those scalar products at once.
3. The transpose in $X^TdZ$ is forced by which dimensions must pair.
4. Bias gradients add over examples because one shared bias affects every example.
5. $dX$ is the downstream gradient moved backward through the transpose of the weight matrix.
6. Shapes let us reject many wrong formulas before calculating a single number.

Mathematics We Built

Forward:

Backward:

These are not three unrelated memorized formulas. They are one chain-rule idea expressed for a matrix operation.

What Each Symbol Means

One-Minute Explanation

A dense layer takes a matrix of inputs, multiplies it by a weight matrix, and adds a bias. During backpropagation, the layer receives a matrix of “how much the loss cares” about each output.

To find how much the loss cares about each weight, pair the input that flowed through that weight with the downstream gradient, and sum across examples: $dW=X^TdZ$.

To find the bias gradient, sum the downstream gradients: $db=\sum dZ$.

To send the error toward the previous layer, multiply the downstream gradient by the transposed weights: $dX=dZW^T$.

The formulas are just the chain rule organized so a matrix library can compute many derivatives together.

Exercises

Level 1 — Observe

Look at

What shape must $Z$ have? What shape must $dW$ have?

Level 2 — Calculate

For one connection, $x_i=4$ and $dZ_j=-0.5$. Calculate $dW_{ij}$.

Level 3 — Derive

Starting from

derive $\partial L/\partial w_{ij}$ using the chain rule.

Level 4 — Investigate

In the notebook, double one row of $X$ while holding $W$, $b$ and $dZ$ fixed. Predict which rows or columns of $dW$ change, then test your prediction.

Level 5 — Design

Design a tiny two-layer network where you can compute every gradient by hand. Choose shapes that make matrix multiplication possible, then verify the gradients with a numerical finite-difference check.

Common Mistakes

Socratic Questions

Why does each weight gradient contain the corresponding input?

Why do examples add together in the bias gradient?

Why is the transpose forced in the formula for $dX$?

Why can a wrong gradient formula sometimes still produce numbers without immediately crashing?

What would happen if the layer had no bias?

🔭 Bridge to Chapter 086

We can now differentiate a matrix layer completely by hand.

But a real image is not naturally represented by one matrix. A batch of RGB images has four meaningful axes: batch, channel, height and width. Sequences, videos and feature maps need even more.

Next question: What single mathematical container can hold numbers in any number of dimensions while still giving us shapes and operations?

That container is the tensor.