084 — dW, db and dX by Hand

1. Forward pass

Use batch-first notation:

X [B,D]
W [D,H]
b [H]
Z [B,H]

$Z=XW+b.$

Suppose the next part of the network sends us

$G=\frac{\partial L}{\partial Z},\qquad G:[B,H].$

We need gradients for $W$ , $b$ , and $X$ .

Each weight $W_{dh}$ connects input feature $d$ to output feature $h$ . Its contribution accumulates over all examples:

$\frac{\partial L}{\partial W}=X^TG.$

Shape Check:

[D,B] @ [B,H] → [D,H]

Exactly the shape of W.

The same bias is added to every example, so its gradient sums across the batch:

$db=\sum_{i=1}^{B}G_i.$

Shape:

[B,H] sum over B → [H]

Earlier layers need to know how the loss changes with $X$ :

$dX=GW^T.$

[B,H] @ [H,D] → [B,D]

Again, exactly the shape of X.

For one example with $z=xw+b$ and upstream gradient $g=dL/dz$ :

$dW=xg,\qquad db=g,\qquad dx=wg.$

The matrix rules are just all scalar paths organized efficiently.

If $A=f(Z)$ and upstream gradient is $dA$ ,

$dZ=dA\odot f'(Z).$

Then feed dZ into the three equations above.

dW = Xᵀ dZ
db = sum_batch(dZ)
dX = dZ Wᵀ

Do not memorize blindly: derive them from dependency paths and verify every shape.

Next → Matrix Backpropagation: how do these local rules compose across an entire multilayer network?