025 — Matrix Calculus for Neural Networks

Where We Are

We already know how learning works for scalar parameters:

\theta \leftarrow \theta-\eta\frac{\partial L}{\partial\theta}

But a neural-network layer looks like

Y=XW+b

where

$X$ is a batch of inputs,
$W$ is a matrix of weights,
$b$ is a bias vector,
$Y$ is a matrix of outputs.

So the real object we need is not one derivative. It is

\frac{\partial L}{\partial W}

— a whole matrix telling us how every weight affects the loss.

1. Start With One Neuron

Suppose

y=x_1w_1+x_2w_2+b

and the loss is $L(y)$ .

By the chain rule,

\frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial y}\frac{\partial y}{\partial w_1}=\frac{\partial L}{\partial y}x_1

Similarly,

\frac{\partial L}{\partial w_2}=\frac{\partial L}{\partial y}x_2

Collect both derivatives into a vector:

\frac{\partial L}{\partial \mathbf{w}}= \begin{bmatrix} \frac{\partial L}{\partial w_1}\\ \frac{\partial L}{\partial w_2} \end{bmatrix} = \begin{bmatrix} x_1\\x_2 \end{bmatrix} \frac{\partial L}{\partial y}

Already we can see the pattern:

input × upstream gradient

That pattern becomes matrix multiplication when we process batches.

2. From One Example to a Batch

Let

X\in\mathbb{R}^{B\times D_{in}}

and

W\in\mathbb{R}^{D_{in}\times D_{out}}.

Then

Y=XW

has shape

Y\in\mathbb{R}^{B\times D_{out}}.

Now imagine the next part of the network has already told us

G=\frac{\partial L}{\partial Y}

with the same shape as $Y$ :

G\in\mathbb{R}^{B\times D_{out}}.

We want the gradient of the loss with respect to $W$ .

The answer is

\boxed{\frac{\partial L}{\partial W}=X^T G}

Check the shapes:

X^T:[D_{in},B]

G:[B,D_{out}]

Therefore

X^TG:[D_{in},D_{out}]

which matches the shape of $W$ exactly.

This is not coincidence. A gradient must match the shape of the object it updates.

3. Why $X^TG$ ?

Take one weight $W_{ij}$ .

The output entry for batch item $b$ and output unit $j$ is

Y_{bj}=\sum_i X_{bi}W_{ij}.

\frac{\partial Y_{bj}}{\partial W_{ij}}=X_{bi}.

The loss depends on $W_{ij}$ through every example in the batch:

\frac{\partial L}{\partial W_{ij}} = \sum_b \frac{\partial L}{\partial Y_{bj}} \frac{\partial Y_{bj}}{\partial W_{ij}}

Substitute:

\frac{\partial L}{\partial W_{ij}} = \sum_b G_{bj}X_{bi}

But that sum is exactly the $(i,j)$ entry of

X^TG.

So matrix calculus is not magic. It is many scalar chain-rule calculations compressed into one matrix operation.

4. Gradient With Respect to the Input

Training also needs to pass information backward into earlier layers.

For

Y=XW

we get

\boxed{\frac{\partial L}{\partial X}=GW^T}

Shape check:

G:[B,D_{out}]

W^T:[D_{out},D_{in}]

therefore

GW^T:[B,D_{in}]

which matches $X$ .

This quantity is often called dX in neural-network derivations.

5. Gradient With Respect to Bias

Now include the bias:

Y=XW+b

The same bias is added to every example in the batch.

So each example contributes to the bias gradient.

Therefore

\boxed{\frac{\partial L}{\partial b}=\sum_{b=1}^{B} G_b}

In code, this is usually a sum across the batch dimension:

db = dY.sum(axis=0)

If $b$ has shape [D_out], then db also has shape [D_out].

6. The Three Equations to Remember

For a linear layer

Y=XW+b

and upstream gradient

dY=\frac{\partial L}{\partial Y},

we get

\boxed{dW=X^TdY}

\boxed{dX=dYW^T}

\boxed{db=\sum_{batch} dY}

Do not memorize them as unrelated formulas.

Read them as information flow:

dW asks: which inputs were present when the output error arrived?
dX asks: how should that output error be projected back through the weights?
db asks: how much total output error accumulated across the batch?

7. Tiny Numerical Example

Let

X= \begin{bmatrix} 1&2\\ 3&4 \end{bmatrix}

and

W= \begin{bmatrix} 5&6\\ 7&8 \end{bmatrix}.

Then

Y=XW= \begin{bmatrix} 19&22\\ 43&50 \end{bmatrix}.

Suppose the next layer sends back

dY= \begin{bmatrix} 1&2\\ 3&4 \end{bmatrix}.

Then

dW=X^TdY

with

X^T= \begin{bmatrix} 1&3\\ 2&4 \end{bmatrix}.

Therefore

dW= \begin{bmatrix} 1&3\\ 2&4 \end{bmatrix} \begin{bmatrix} 1&2\\ 3&4 \end{bmatrix} = \begin{bmatrix} 10&14\\ 14&20 \end{bmatrix}.

Every number in this matrix tells us how changing one weight would change the final loss.

8. NumPy From Scratch

import numpy as np

X = np.array([[1., 2.],
              [3., 4.]])
W = np.array([[5., 6.],
              [7., 8.]])
b = np.array([1., 1.])

Y = X @ W + b

dY = np.array([[1., 2.],
               [3., 4.]])

dW = X.T @ dY
dX = dY @ W.T
db = dY.sum(axis=0)

print("Y =\n", Y)
print("dW =\n", dW)
print("dX =\n", dX)
print("db =", db)

Before running it, predict every shape.

9. Finite-Difference Gradient Check

How do we know the matrix formula is correct?

We can perturb one weight slightly.

For a small $\epsilon$ :

\frac{\partial L}{\partial W_{ij}} \approx \frac{L(W_{ij}+\epsilon)-L(W_{ij}-\epsilon)}{2\epsilon}

This is called a numerical gradient check.

It is slow, so we do not train this way.

But it is extremely useful for verifying a hand-derived backward pass.

eps = 1e-5

Change one entry of W, recompute the loss, and compare the numerical slope with the matching entry in dW.

This is how you turn calculus into testable software.

10. PyTorch Confirms the Derivation

import torch

X = torch.tensor([[1., 2.],
                  [3., 4.]])
W = torch.tensor([[5., 6.],
                  [7., 8.]], requires_grad=True)
b = torch.tensor([1., 1.], requires_grad=True)

Y = X @ W + b
loss = (Y ** 2).sum()
loss.backward()

print(W.grad)
print(b.grad)

PyTorch’s autograd does not replace the mathematics.

It automates exactly the chain-rule bookkeeping we just derived.

When debugging a neural network, understanding the underlying shapes and formulas is what lets you decide whether an autograd result makes sense.

11. Shape Check

Suppose

X  : [32, 128]
W  : [128, 64]
b  : [64]

Then

Y  : [32, 64]
dY : [32, 64]

Predict:

dW = X.T @ dY   → [128, 64]
dX = dY @ W.T   → [32, 128]
db = sum(dY, 0) → [64]

This habit catches a huge fraction of backpropagation mistakes before code runs.

12. Why Transposes Keep Appearing

A transpose is often needed because forward and backward information travel in opposite structural directions.

Forward:

XW

Backward into $X$ :

dYW^T

Backward into $W$ :

X^TdY

The transpose is what makes the inner dimensions align while preserving the correct output shape.

A useful debugging question is:

What shape must the gradient have? What multiplication produces exactly that shape?

Shape reasoning does not prove the derivative, but it eliminates many impossible answers.

13. From Matrix Calculus to Backpropagation

A neural network is a composition:

X \rightarrow XW_1+b_1 \rightarrow \text{ReLU} \rightarrow HW_2+b_2 \rightarrow L

Backpropagation repeatedly applies the same principle:

receive an upstream gradient,
compute local gradients,
produce parameter gradients,
pass a new upstream gradient to the previous operation.

Matrix calculus is the language that makes those steps compact.

Without it, backpropagation looks like mysterious symbol pushing.

With it, backpropagation becomes structured bookkeeping.

14. Break It

Deliberately create these bugs:

use X @ dY instead of X.T @ dY,
sum dY across the wrong axis for db,
swap the order in dY @ W.T,
create a batch of size 1 and accidentally squeeze a dimension,
compare your manual gradient against PyTorch autograd.

For each failure, explain whether the problem is:

mathematically wrong,
shape-incompatible,
accidentally broadcastable,
numerically plausible but conceptually wrong.

The last category is the dangerous one.

15. What You Should Now Understand

You should be able to derive and explain:

dW=X^TdY

dX=dYW^T

db=\sum dY

You should also understand that:

matrix gradients are collections of scalar partial derivatives,
the chain rule is still doing the real work,
transposes arise from the structure of matrix multiplication,
gradient shapes must match the quantities they update,
numerical gradient checks can verify a backward pass,
autograd automates these exact operations.

Level 2 Milestone

We began this level with a simple question: how does a value change?

That forced us to discover:

functions
  ↓
slopes
  ↓
derivatives
  ↓
partial derivatives
  ↓
gradients
  ↓
chain rule
  ↓
computational graphs
  ↓
gradient descent
  ↓
learning-rate geometry
  ↓
matrix calculus

We now have the mathematical machinery required to train neural networks.

But learning from data also means reasoning when outcomes are uncertain.

Next → Level 3: Probability and Uncertainty.

Level 1 — Explain It

Why must dW have the same shape as W?
Explain dW = X.T @ dY without using the words “because that is the formula”.
What does dX represent conceptually?
Why does the bias gradient sum across the batch?

Level 2 — Calculate It

Given

X=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad W=\begin{bmatrix}2&0\\1&3\end{bmatrix},\quad b=\begin{bmatrix}1&-1\end{bmatrix}

Compute $Y=XW+b$ .
Let

dY=\begin{bmatrix}1&2\\-1&3\end{bmatrix}.

Compute by hand:

dW=X^TdY

dX=dYW^T

db=\sum_{batch}dY

Verify every output shape.

Level 3 — Predict It

Without running code, fill in the shapes.

X  : [64, 256]
W  : [256, 128]
b  : [128]
Y  : [?]
dY : [?]
dW : [?]
dX : [?]
db : [?]

Then repeat for:

X : [1, 10]
W : [10, 3]

and

X : [512, 768]
W : [768, 3072]

Level 4 — Break It

Start from a working manual backward pass and introduce these bugs one at a time:

dW = X @ dY
dX = W @ dY
db = dY.sum(axis=1)
remove the batch dimension when batch size is 1
rely on broadcasting that produces an output but changes the intended meaning

For each bug, classify it as:

immediate shape error,
mathematically wrong but executable,
silent broadcasting bug,
wrong reduction.

Level 5 — Build It

Implement a minimal Linear layer from scratch using NumPy.

It should support:

class Linear:
    def forward(self, X):
        ...

    def backward(self, dY):
        ...

Store:

X from the forward pass,
W, b,
dW, db,
and return dX.

Then train a tiny two-layer network without autograd.

Level 6 — Investigate It

Gradient-check experiment

Choose a tiny linear layer with random values.

Compute dW analytically.
Compute every entry of dW numerically using central differences:

\frac{L(W_{ij}+\epsilon)-L(W_{ij}-\epsilon)}{2\epsilon}

Measure the maximum absolute difference.
Repeat for several values of $\epsilon$ such as:

1e-1, 1e-3, 1e-5, 1e-7, 1e-9

Explain why an extremely tiny $\epsilon$ can make the numerical estimate worse because of floating-point precision.

Write your conclusion as:

Hypothesis → Method → Result → Explanation → Limitation

Challenge — Derive One Entry

For

Y=XW

show from scalar notation that

Y_{bj}=\sum_i X_{bi}W_{ij}

and therefore

\frac{\partial L}{\partial W_{ij}} = \sum_b X_{bi}\frac{\partial L}{\partial Y_{bj}}.

Then show that this entire collection of scalar equations is exactly

\boxed{\frac{\partial L}{\partial W}=X^T\frac{\partial L}{\partial Y}}.

If you can move comfortably between the scalar equation and the matrix equation, you understand the core idea of matrix calculus for neural networks.

Where We Are

1. Start With One Neuron

2. From One Example to a Batch

3. Why XTGX^TGXTG?

4. Gradient With Respect to the Input

5. Gradient With Respect to Bias

6. The Three Equations to Remember

7. Tiny Numerical Example

8. NumPy From Scratch

9. Finite-Difference Gradient Check

10. PyTorch Confirms the Derivation

11. Shape Check

12. Why Transposes Keep Appearing

13. From Matrix Calculus to Backpropagation

14. Break It

15. What You Should Now Understand

Level 2 Milestone

Level 1 — Explain It

Level 2 — Calculate It

Level 3 — Predict It

Level 4 — Break It

Level 5 — Build It

Level 6 — Investigate It

Gradient-check experiment

Challenge — Derive One Entry

3. Why $X^TG$ ?