Partial Derivatives, Gradients and the Chain Rule

Where We Are

A derivative tells us how the loss changes when we move one parameter.

But a real model has many parameters. Changing one at a time is not enough.

Can we describe all those directions of change with one object — and find the direction that rises fastest?

Next → Gradients.

1. The Problem: What Is the Slope of a Bowl?

Chapter 018 closed on a question, and it is worth restating in full before we touch any mathematics.

Our house-price office now records two measurements per house — the data from Chapter 005:

Chapter 005 found the rule hiding in this table:

Check house C: $2(3) + 0.005(900) + 1 = 6 + 4.5 + 1 = 11.5$ ✓. Our model has the same shape with the numbers left blank:

Three dials. $w_1$ is ₹ lakh per room, $w_2$ is ₹ lakh per square foot, $b$ is the price of the bare plot. And the loss is Chapter 1’s mean squared error:

Now stand somewhere on that loss and ask Chapter 018’s question: how steep is it here?

The question has no answer. Not “we don’t know the answer yet” — the question is incomplete. Turning up $w_1$ might raise the loss. Turning up $b$ at the same moment might lower it. Turning both up together does something else again. There is no single number called “the slope”, because the loss does not have one slope; it has a different slope for every direction you could move in.

Think — With one dial, “left” and “right” were the only choices, and one number covered both (its sign chose the side). With two dials you can move in a full circle of directions, and with a billion dials in a space of directions so large it has no picture. A single number cannot possibly describe all of them.

Pinning a dial so we can draw

To keep pictures possible for the next several sections, we do exactly what Chapter 018 did: pin one dial at its true value and watch the other two move. Chapter 018 pinned $b = 1$; we pin

and study the surface $L(w_1, b)$. This is a convenience, not a cheat — and it is temporary. Everything we build will be stated for $n$ dials, and §22 puts $w_2$ back.

With $w_2$ pinned at its true value, the remaining two dials still have a perfect answer waiting at $(w_1, b) = (2, 1)$, where the loss is exactly zero.

The surface, in closed form

We can write $L(w_1, b)$ down exactly, and it is worth doing because every number in this chapter will be checkable by hand afterwards.

Our prediction for house $i$ is $\hat{y}_i = w_1 r_i + 0.005\,a_i + b$. The truth is $y_i = 2 r_i + 0.005 a_i + 1$. Subtract:

The area term cancels completely — both lines price area identically, because we pinned $w_2$ correctly. Write $u = w_1 - 2$ and $\beta = b - 1$ for “how wrong each dial is”. Then

Split the sum into three pieces and pull out everything that does not depend on $i$:

Both leftovers are properties of the four houses alone:

Sanity-check it at the origin, $w_1 = 0$, $b = 0$. Predictions are $\hat{y} = 0\cdot r + 0.005a + 0 = [4,\,6,\,4.5,\,8]$, errors are $[-5,\,-5,\,-7,\,-9]$, so

And the formula gives $8.25(4) + 5.5(-2)(-1) + 1 = 33 + 11 + 1 = 45$ ✓. Two completely different routes, same number.

Notice the $8.25$. That is exactly the $\frac{1}{n}\sum r_i^2$ that Chapter 005 §10 used to explain why raw features are badly scaled, and the same quantity Chapter 1 §10 called $7.5$ for its own data. The steepness of a loss valley along a feature is always that feature’s mean square. It keeps returning because it is the same fact.

2. What Would a Solution Need?

Before inventing anything, list what an honest answer must do.

1. Give one number per dial, not one number overall. We have to know what to do with $w_1$ and what to do with $b$, separately, because we have to set them separately.
2. Combine into an answer for any direction. Once we know the per-dial numbers, we must be able to say how steep the surface is along any direction, including diagonals nobody asked about in advance.
3. Name the steepest direction. Of the infinitely many ways to walk, one is downhill fastest. We want it, not just a direction that happens to work.
4. Scale to a billion dials. Anything requiring a picture, or requiring us to try each direction, is dead on arrival.
5. Survive composition. Our loss is not a formula in $w_1$ — it is a formula in the predictions, which are a formula in $w_1$. Whatever we build must handle a quantity that reaches the parameters only through other quantities.

Requirements 1–3 occupy the first half of this chapter. Requirement 5 is the second half, and it is the one that will still be doing work in Chapter 083.

3. First Attempt: Nudge Both Dials Together

The obvious move is to copy Chapter 018 exactly. It measured a slope by nudging the input and dividing:

We have two inputs, so nudge them both:

This is perfectly well defined. It produces a number. Let us see what the number is worth.

At $(w_1, b) = (0, 0)$ this comes out to $-51.5$ (the notebook checks it numerically in Step 3). Negative — so the loss falls when we turn both dials up together. Useful? A little. But now ask the question we actually need answered: by how much should we turn up $w_1$, and by how much $b$? The number $-51.5$ has nothing to say. It would be $-51.5$ whether the truth were ”$w_1$ is doing all the work” or ”$b$ is doing all the work” or anything in between. One number cannot instruct two dials. Requirement 1, violated on the first line.

It is worse than uninformative. It actively lies.

A Tempting Wrong Idea

“Nudge everything together and read one slope. If it comes out zero, we are at the bottom.”

Stand at $(w_1, b) = (3.5,\, -3.4)$. Nudge both dials together and the measurement reads exactly zero. By this test we have arrived: the ground is flat, training is finished, go home.

The loss there is $L = 1.6225$, and the true minimum — where the loss is $0$ — sits at $(2, 1)$, a distance of $4.65$ away. We are nowhere near the bottom and the instrument says we are standing on it.

Why? Because at that point $w_1$ is too large and $b$ is too small by amounts that happen to cancel. The individual slopes are $+0.55$ and $-0.55$. Moving both dials up in lockstep really does leave the loss unchanged — we have measured one direction out of infinitely many, and picked one that happens to run along the contour. The instrument is not broken; it answered a different question from the one we asked.

The failure is precise, and it tells us what to fix: we must stop mixing the dials. If we want to know what $w_1$ is doing, we must move $w_1$ and nothing else.

4. The Discovery: Nudge One Dial, Hold the Others

Here is the whole idea, and it is almost embarrassingly simple once the failure above has made it necessary.

To find out what one dial is doing, freeze every other dial and vary only that one.

The instant you freeze $b$, the function $L(w_1, b)$ stops being a surface and becomes a curve — a function of the single variable $w_1$ — and Chapter 018 knows exactly what to do with those. So:

Read it aloud, slowly:

“Partial dee L by dee $w_1$: nudge $w_1$ by a tiny amount, leave $b$ exactly where it is, see how much $L$ moves, and divide by the size of the nudge.”

This is a partial derivative. The word “partial” is doing exactly one job: reminding you that this measurement is about part of the input, with the rest held still.

Likewise, freezing $w_1$ and varying $b$:

The curly $\partial$

The symbol $\partial$ is a stylised letter d, and it is not the same as Chapter 018’s $d$. The difference is a promise about context:

That is the entire distinction. $\partial$ is a warning label: other variables exist; I am ignoring them on purpose.

Never read $\partial$ as a number being divided. As with $\frac{dy}{dx}$ in Chapter 018 §6, $\frac{\partial L}{\partial w_1}$ is one indivisible symbol shaped like a fraction.

The three levels

That last line says the general thing in one breath: move along axis $j$ only. Chapter 005’s basis vectors have quietly come back to work.

How to actually compute one

The mechanical rule falls straight out of the definition, and it is the only rule you need:

Treat every other variable as if it were a constant, then differentiate with Chapter 018’s rules.

Because that is literally what “hold them fixed” means. If $b$ is not moving, then $b$, $b^2$, $5b$ and $\sin b$ are all just numbers as far as $w_1$ is concerned — and Chapter 018 §7 told us the derivative of a constant is zero.

Try it on something small, to build the reflex. Let $f(x,y) = x^2 y + 3y$.

Differentiating with respect to $x$, treat $y$ as a number, say $7$: $f = 7x^2 + 21$, whose derivative is $14x$. Put $y$ back:

Differentiating with respect to $y$, treat $x$ as a number: $f = (x^2)y + 3y = (x^2+3)y$, whose derivative is the coefficient $x^2+3$:

Two different functions, from one $f$. That is normal and it is the point.

5. Deriving Our Two Partials

Now do it for the loss that matters. We will get the same two answers by three different routes, because agreement between independent derivations is the only real proof that we have not fooled ourselves.

Route 1 — from the closed form

Hold $b$ fixed, so $(b-1)$ is a constant. Differentiate term by term with the sum, constant-multiple and power rules from Chapter 018 §7:

• $\dfrac{\partial}{\partial w_1}\Big[8.25(w_1-2)^2\Big] = 8.25 \cdot 2(w_1-2) = 16.5\,(w_1-2)$
• $\dfrac{\partial}{\partial w_1}\Big[5.5(w_1-2)(b-1)\Big] = 5.5(b-1)$ — the bracket $(b-1)$ is a constant multiplier, and $w_1 - 2$ has derivative $1$
• $\dfrac{\partial}{\partial w_1}\Big[(b-1)^2\Big] = 0$ — a pure constant

Now hold $w_1$ fixed and do the same in $b$:

• $\dfrac{\partial}{\partial b}\Big[8.25(w_1-2)^2\Big] = 0$
• $\dfrac{\partial}{\partial b}\Big[5.5(w_1-2)(b-1)\Big] = 5.5(w_1-2)$
• $\dfrac{\partial}{\partial b}\Big[(b-1)^2\Big] = 2(b-1)$

Look at what the middle term did. $\frac{\partial L}{\partial w_1}$ contains a $(b-1)$, and $\frac{\partial L}{\partial b}$ contains a $(w_1-2)$. The dials are coupled. How steeply the loss climbs in the $w_1$ direction depends on where $b$ currently is. This is not an artefact of our algebra — it is why tuning parameters one at a time is such a miserable way to train a model, and it is why the cross term $5.5(w_1-2)(b-1)$ exists at all. It exists because rooms and the bias are correlated across our four houses: every house has some rooms, so raising $b$ and raising $w_1$ do partly overlapping jobs.

Route 2 — from the sum, without the closed form

The closed form was a luxury of this tiny problem. In general we have only

Differentiate inside the sum — legal, because the sum rule from Chapter 018 §7 says the derivative of a sum is the sum of the derivatives, and $\frac{1}{n}$ is a constant multiple. For one term, with $e_i = \hat{y}_i - y_i$:

That step used the chain rule, which we have not derived yet — §16 does, properly, and §19 returns to redo this line honestly. For now note only that $e_i = w_1 r_i + w_2 a_i + b - y_i$, so nudging $w_1$ by $h$ changes $e_i$ by exactly $h r_i$, giving $\frac{\partial e_i}{\partial w_1} = r_i$. Therefore

These are exactly the two formulas Chapter 1 §11 handed over on trust, promising a derivation later. This is later.

Read them in English:

• $\frac{\partial L}{\partial w_1}$ is the average error, weighted by how many rooms each house has. A house with 4 rooms pushes on the rooms-weight four times as hard as a house with 1 room, because changing the rooms-weight moves that house’s prediction four times as much.
• $\frac{\partial L}{\partial b}$ is the average error, plain. Every house pushes on the bias equally hard, because the bias moves every prediction by the same amount.

The structure is the same in both: error $\times$ how much this parameter moves that prediction. Hold onto that sentence. In Chapter 083 it becomes backpropagation, and in Chapter 085 it becomes $dW = \delta^\top X$.

Route 3 — numerically

The notebook (Step 5) computes both partials with the finite-difference recipe from Chapter 018 §11, $h = 10^{-6}$, and asserts agreement with Routes 1 and 2 at four different points. That is a gradient check, and it is the single most useful debugging tool in this entire subject: when you derive a gradient by hand and implement it, you compare it against finite differences before you trust one line of it.

6. ✏️ Hand Calculation — Both Partials at the Origin

Start ignorant: $w_1 = 0$, $b = 0$ (with $w_2$ pinned at $0.005$).

Predict.

Errors. $\hat{y}_i - y_i = [4-9,\; 6-11,\; 4.5-11.5,\; 8-17] = [-5,\; -5,\; -7,\; -9]$.

Loss.

Partial with respect to $w_1$ — error times rooms, averaged, doubled:

Partial with respect to $b$ — error, averaged, doubled:

Cross-check with the closed form. $16.5(0-2) + 5.5(0-1) = -33 - 5.5 = -38.5$ ✓ and $5.5(0-2) + 2(0-1) = -11 - 2 = -13$ ✓.

Read the answer. Both partials are negative, so both dials are too small — raising either one lowers the loss. And $w_1$‘s slope is nearly three times steeper than $b$‘s, which makes sense: a one-unit change in the rooms-weight moves every prediction by 2 to 4 lakh, while a one-unit change in the bias moves every prediction by exactly 1.

The notebook asserts every number in this box: $45$, $-38.5$, $-13$, and the four predictions.

7. From Many Slopes to One Object: The Gradient

We have two numbers where we used to have one. With three dials we would have three; with a billion, a billion. Carrying them around as loose scalars is exactly the mess Chapter 005 solved for house measurements, and the solution is the same: put them in a vector.

This is the gradient. The symbol $\nabla$ is an upside-down Greek delta, pronounced “nabla” or just spoken as “grad”: $\nabla L$ is read “grad L” or “the gradient of L”.

At our origin:

Three things are true about this object, and each one is doing work:

It is a vector, so it has a length. From Chapter 005 §6:

It is a vector, so it has a direction. And §10 will prove that direction is the single most important fact about the point we are standing on.

It lives in the same space as the parameters. $\nabla L \in \mathbb{R}^2$ because $\theta = (w_1, b) \in \mathbb{R}^2$. This is not a coincidence, it is a structural guarantee, and §22 makes it a safety check.

Intuition — The gradient is not “the slope.” It is the complete slope report: one entry per dial, saying how sensitive the loss is to that dial right now. Chapter 1 wrote $\nabla_\theta L$ in its update rule and told you to take it on faith. Here it is.

General definition

For any function $f$ of $n$ variables $\theta = (\theta_1, \ldots, \theta_n)$:

Nothing changes when $n$ is a billion except the amount of storage. That is requirement 4, satisfied — the gradient is a list, and lists scale.

Common Mistake — saying “the gradient of $L$ at $w_1$.” The gradient is evaluated at a point in parameter space, $(w_1, b)$, not at a single coordinate. Change $b$ and both entries of $\nabla L$ change, as §5 showed.

8. The Linear Approximation: What the Gradient Predicts

We have per-dial slopes. Requirement 2 asks for a slope along any direction. Getting there needs one idea, and it is the idea that makes the gradient more than bookkeeping.

Suppose we move from $(w_1, b)$ to $(w_1 + \Delta w_1,\; b + \Delta b)$, with both changes small. How much does $L$ change?

Split the move into two legs — first east, then north:

This is exact — the two middle terms cancel, nothing has been approximated yet. Now handle each leg.

Leg 1 varies only $w_1$, which is precisely what $\frac{\partial L}{\partial w_1}$ measures, so for small $\Delta w_1$,

Leg 2 varies only $b$, so

That partial is evaluated at a slightly shifted point. If the partials are continuous — which for our polynomial loss they certainly are — the shift changes them by an amount that vanishes as $\Delta w_1 \to 0$, and the error it introduces is second-order small. Replacing it with the partial at our own point:

And the right-hand side is a dot product — Chapter 005 §7, the operation we invented to multiply a list of weights against a list of measurements:

This is the multivariable version of "$f(x+h) \approx f(x) + f'(x)h$". With one dial, the derivative told you what a small step does. With many dials, the gradient dotted with your step tells you what that step does. The gradient is the machine that converts a proposed move into a predicted change in loss.

Check it numerically

At $(0,0)$, propose the step $\Delta\theta = (0.01,\, 0.01)$. Prediction:

Truth: $L(0.01, 0.01) = 45 - 0.5148...$, so the actual change is $-0.51486$. The prediction is wrong by about $1.4 \times 10^{-4}$ — and the error shrinks like the square of the step size, which the notebook checks in Step 8 by halving the step and watching the error fall by four.

The tangent plane

Geometrically, $L(\theta + \Delta\theta) \approx L(\theta) + \nabla L\cdot\Delta\theta$ is the equation of a plane touching the bowl at our point — the exact analogue of Chapter 018 §9’s tangent line. One dial gave a tangent line; two dials give a tangent plane; $n$ dials give a tangent hyperplane nobody can draw and everybody can compute with.

9. Directional Derivatives: Steepness in Any Direction You Choose

Now requirement 2 is one line away.

Pick a direction and walk in it. A direction is a unit vector $\mathbf{u}$ — Chapter 005 §6 — meaning $\|\mathbf{u}\| = 1$, so that “distance travelled” means what it says. Step a distance $t$ along it: $\Delta\theta = t\,\mathbf{u}$. The directional derivative is the rate of change of $L$ per unit of distance travelled in that direction:

Substitute the linear approximation from §8, with $\Delta\theta = t\mathbf{u}$:

Divide by $t$ — and the $t$ cancels exactly, as it did in Chapter 018 §4 — leaving

One dot product answers every direction at once. We computed two numbers, and from them we can now report the steepness along any of the infinitely many directions in the plane, without measuring any of them.

The full report at $(0,0)$

$\nabla L = (-38.5,\, -13)$. Dot it with various unit directions:

Every one of these comes from the same two numbers and one dot product. The notebook verifies each row against a direct finite difference in Step 5.

Two rows deserve attention immediately.

The diagonal $\tfrac{1}{\sqrt2}(1,1)$ — “turn both dials up equally”, the intuitive move — gives $-36.416$. The last row gives $-40.636$. The intuitive move is 10% worse than the best available. With two dials that is a mild waste; with a billion dials, systematically moving in a direction that is not the steepest is the difference between training finishing this week and never.

And the row that reads exactly $0$: there is a direction in which the loss does not change at all, to first order. You can walk along it and get nowhere. That direction is not a curiosity — §11 shows it is the contour line through our point, and §12 shows what happens to people who ignore it.

10. Why the Gradient Is the Direction of Steepest Ascent

Now requirement 3, and this is the theorem the whole chapter is built to prove.

We want the unit vector $\mathbf{u}$ that makes $D_{\mathbf{u}}L = \nabla L\cdot\mathbf{u}$ as large as possible. Chapter 005 §8 derived, from the law of cosines, the identity that settles it:

Apply it with $\mathbf{p} = \nabla L$ and $\mathbf{q} = \mathbf{u}$, remembering $\|\mathbf{u}\| = 1$:

where $\theta$ is the angle between our chosen direction and the gradient. Now stare at that for a second, because it contains the entire answer.

$\|\nabla L\|$ is a fixed positive number — it does not depend on which direction we pick. The only thing we control is $\cos\theta$, and cosine lives in $[-1, 1]$. So:

That is not a definition, a convention or an analogy. It is a consequence of the dot product formula, which was itself a consequence of the law of cosines in Chapter 005. We derived it.

Check the numbers

At $(0,0)$, $\|\nabla L\| = 40.6356$. The theorem says no direction can beat $-40.6356$ for descent. Look back at §9’s table: east gave $-38.5$, the diagonal gave $-36.416$, and the direction $-\nabla L/\|\nabla L\| = (0.94745,\, 0.31992)$ gave exactly $-40.6356$ ✓. Nothing beat it, and the steepest-descent direction hit the bound exactly, as it must.

Precision matters here. “Steepest” is measured with respect to the ordinary Euclidean length of a step — the $\|\cdot\|$ we have been using since Chapter 005. Measure step size differently and a different direction wins. This is not pedantry: it is exactly the door that Chapter 32’s optimizers (Adam rescales each coordinate) and Chapter 56’s metrized deep learning walk through. For now, Euclidean, and we will say so every time it matters.

And “steepest” means locally, at this point, for an infinitesimal step. Nothing here says the gradient direction is a good direction to travel any appreciable distance. §12 is about people who forget this.

11. The Geometry: Contour Lines and Perpendicularity

There is a picture underneath §10, and it is the picture to carry for the rest of the course.

A contour (or level set) of $L$ is the set of parameter pairs that produce the same loss — the loop you would walk along a hillside without going up or down. On a map they are the concentric rings around a peak; on our loss they are concentric rings around $(2, 1)$.

For our quadratic loss they are ellipses. The set $L(w_1,b) = c$ is

and the cross term $5.5(w_1-2)(b-1)$ is exactly what tilts them off the axes.

   b
   │        ╱‾‾‾‾‾‾‾‾╲               each ring is one loss value
 3 │      ╱   ╱‾‾‾╲   ╲
   │     │   │  ⭐  │   │            ⭐ = (2, 1), L = 0
 1 │     │   ╲ ___ ╱   │
   │      ╲           ╱              rings are long and thin:
-1 │        ╲_______╱                one direction is ~122× steeper
   └──────────────────────── w₁
      0     1    2    3    4

Now the key fact, and it follows immediately from §9 and §10:

The gradient at a point is perpendicular to the contour through that point.

Why: walking along the contour leaves $L$ unchanged, so the directional derivative along the contour’s tangent $\mathbf{t}$ is zero. But $D_{\mathbf{t}}L = \nabla L \cdot \mathbf{t}$, so $\nabla L\cdot\mathbf{t} = 0$ — and Chapter 005 §8 told us a zero dot product means orthogonal, a $90°$ angle. Nothing more is needed.

Verify it at $(0,0)$. The gradient is $(-38.5, -13)$. A perpendicular direction is $(13, -38.5)$ normalised, i.e. $(0.31992,\, -0.94745)$ — flip the components and negate one, the standard trick in two dimensions. Dot them:

And the notebook walks a distance $0.001$ along that tangent and finds the loss moved from $45$ to $45.00000007$ — flat to seven decimal places, exactly as first-order theory demands.

Intuition — Water poured on a hillside runs perpendicular to the contour lines, because that is the fastest way down. The gradient is the same statement written in algebra.

12. ⚠️ A Tempting Wrong Idea: The Gradient Points At the Minimum

Here is a belief that is almost universal among people meeting gradients for the first time, is very nearly true in easy cases, and is flatly false.

“The gradient points uphill, so minus the gradient points towards the bottom. Follow it and you go straight to the answer.”

The first clause is proven. The second does not follow from it, and the difference is the reason Chapter 32 exists.

Stand at $(w_1, b) = (2, 3)$. Read that point carefully: $w_1$ is already exactly correct. The rooms-weight is 2, the truth. Only the bias is wrong — it is 3 when it should be 1. The minimum is straight below us: the direction to it is $(0, -1)$.

Compute the gradient:

Steepest descent says: decrease $w_1$. Decrease the one dial that is already perfect, and decrease it hard — $94\%$ of the step is spent on $w_1$ and only $34\%$ on the bias, the parameter that is actually wrong.

The angle between the steepest-descent direction and the direction to the minimum is

Seventy degrees off target. And yet it is not a mistake — the loss really does fall faster that way. Step a distance $0.1$ downhill and the loss drops from $4$ to $2.9212$. Step the same distance straight at the minimum and it only drops to $3.61$. The gradient is telling the truth about the next instant and saying nothing whatever about the destination.

Why this happens. The contours are tilted ellipses, not circles. On a circular bowl, perpendicular-to-the-contour really does aim at the centre, and gradient descent goes straight home. On a stretched, tilted bowl it does not, and the more stretched the bowl, the worse the aim. The stretch factor has a name we already met: Chapter 005 §10 called it the condition number, and Chapter 023 will compute it for this exact loss.

What to take from this: the gradient is a local instrument. It is a compass, not a map. Chapter 023 builds an algorithm that uses a compass repeatedly and shows exactly what that costs.

13. When Partials Are Not Enough

Everything from §8 onward rested on one approximation: $\Delta L \approx \nabla L\cdot\Delta\theta$. It is worth knowing that this can fail, because the failure explains why textbooks are careful about a word we have been using loosely.

Having partial derivatives is weaker than being well approximated by a plane. Partials only probe two directions — along the axes. A function can be perfectly well behaved along both axes and insane everywhere else.

The standard example:

Along the $x$-axis, $y = 0$, so $f(x, 0) = 0$ for every $x$. Flat. So $\frac{\partial f}{\partial x}(0,0) = 0$. By symmetry $\frac{\partial f}{\partial y}(0,0) = 0$. Both partials exist and are zero, so $\nabla f(0,0) = (0,0)$, and the linear approximation predicts $f$ is flat near the origin.

Now walk the diagonal, $y = x$:

for every non-zero $t$, however small. Approach the origin along the diagonal and $f$ sits stubbornly at $0.5$; approach along an axis and it is $0$. The function is not even continuous at the origin, let alone flat. The directional derivative along the diagonal is $\lim_{t\to0}\frac{0.5 - 0}{t} = \infty$ — it does not exist.

So $\nabla f\cdot\mathbf{u}$ gave $0$ and the truth was $\infty$. Partials existing is not enough.

The condition that is enough, and that is what “differentiable” means for several variables, is the existence of a genuine linear approximation:

The useful practical theorem: if all the partial derivatives exist and are continuous near a point, the function is differentiable there. Our loss is a polynomial, so its partials are polynomials, so they are continuous everywhere, so we are safe everywhere. Every loss in this course built from sums, products and smooth activations is safe for the same reason — and the exceptions (ReLU’s corner, Chapter 018 §13) are known, named and handled deliberately rather than accidentally.

14. The Problem Again: Our Loss Is Built By Composition

We have the gradient and we know what it means. But look back at §5, Route 2, where an unexplained step sat in the middle of the derivation:

We got away with it because our model is linear and the algebra was easy to check another way. That luck is about to run out, and it is worth seeing why now rather than in Chapter 083.

The loss does not depend on $w_1$ directly. It depends on it through a chain:

flowchart LR
    A["w₁, w₂, b<br/>parameters"] --> B["ŷ = w₁r + w₂a + b<br/>prediction"]
    B --> C["e = ŷ − y<br/>error"]
    C --> D["ℓ = e²<br/>squared error"]
    D --> E["L = mean of ℓ<br/>loss"]

Change $w_1$, and the prediction moves; because the prediction moved, the error moves; because the error moved, its square moves; because the squares moved, their average moves. Four links, and we want the rate of change across the whole chain.

This is not a quirk of our example. Every model in this course is a composition, and the deeper the model the longer the chain. A 50-layer network has a chain of 50 links between a first-layer weight and the loss. If we cannot differentiate through composition, we cannot train anything.

That is requirement 5, and it needs its own discovery cycle.

15. First Attempt: Expand Everything

The direct approach: substitute every link into the next until only $w_1$ remains, then differentiate the result.

Take house A alone: $r = 2$, $a = 800$, $y = 9$, with $w_2$ pinned at $0.005$ and the bias written as $b$.

Expand:

Differentiate with §4’s rule, holding $b$ fixed:

At $w_1 = 0, b = 0$ this is $-20$. Correct — the notebook confirms it.

So it works. Why is it not the answer?

Because the expansion is a dead end. Three reasons, in increasing order of seriousness:

1. It does not survive size. That was one house with a linear model. Four houses triples the algebra. A two-layer network with a nonlinearity in between produces an expression you cannot write on a page, and a 50-layer network produces one you cannot write in this universe.
2. It throws away structure. The expansion mixes $8w_1 + 4b - 20$ into one lump, destroying the information that this number is twice the error, times the number of rooms. That structure is what makes the formula computable at scale.
3. It cannot be automated as written. A framework cannot symbolically expand a 50-layer network. It needs a rule that works link by link, locally, using only what is at hand.

A Tempting Wrong Idea

”$\ell$ depends on $e$ and $e$ depends on $w_1$, so $\frac{\partial\ell}{\partial w_1}$ is just $\frac{\partial \ell}{\partial e}$ — the rate at which the square grows.”

That is $2e = -10$ at our point, and the true answer is $-20$. Off by a factor of exactly $2$ — which is $r$, the number of rooms.

The missing factor is the whole content of the next section. The error does not move at the same speed as $w_1$; it moves twice as fast, because one extra ₹ lakh per room raises a two-room house’s prediction by ₹2 lakh. Ignoring the exchange rate between links is the most common gradient bug there is.

What we need is a rule that composes rates of change. Something that says: if $\ell$ changes $p$ times as fast as $e$, and $e$ changes $q$ times as fast as $w_1$, then $\ell$ changes $\underline{\hspace{1.2em}}$ times as fast as $w_1$.

Everyone’s instinct fills that blank with $pq$. Let us find out whether the instinct is right.

16. The Discovery: The Chain Rule

The child’s version

Bicycle gears. Turn the pedals once and the chainring turns once. The chainring drives the rear sprocket at 3 turns per turn. The wheel is bolted to the sprocket, so it turns 1-for-1 with it, and the bike moves 2 metres per wheel turn.

How far does the bike go per pedal turn?

Nobody hesitates over that. Rates along a chain multiply. The chain rule says calculus behaves exactly like gears, and the honest question is why that needs proving at all.

It needs proving because a derivative is an instantaneous rate, and each gear ratio in a real function changes as you move. The chainring ratio at $e = -5$ is not the ratio at $e = -1$. So the multiplication has to be justified, and the justification has to say where each factor is evaluated.

The derivation

Let $y = g(x)$ and $z = f(y)$, so $z = f(g(x))$ — read “f of g of x”, the composition. We want $\frac{dz}{dx}$.

Start from Chapter 018’s definition. Nudge $x$ by $h$ and let $k$ be the resulting change in $y$:

Then $z$ changes from $f(g(x))$ to $f(g(x) + k)$, and

Now the trick, and it is a single algebraic move: multiply and divide by $k$.

Two difference quotients, exactly the shape Chapter 018 defined. Now let $h \to 0$. Because $g$ is differentiable it is continuous, so $k \to 0$ too — the inner nudge shrinks along with the outer one. The first factor heads to $f'(y)$ and the second to $g'(x)$:

or, in Leibniz’s notation, in the form everybody remembers:

This is why Chapter 018 §6 insisted that Leibniz’s notation was worth learning even though it is not a fraction. Written this way the chain rule looks like cancelling $dy$ — which is not what is happening, but the notation was deliberately designed so that the true rule looks like the false operation. That is good design, and it is why $\frac{dy}{dx}$ won.

Where each factor is evaluated matters, and the boxed formula says so: $f'$ is evaluated at $g(x)$ — the value the inner function actually produced — not at $x$. Getting this wrong is the second most common gradient bug, after forgetting a factor entirely.

A gap worth naming. The step “multiply and divide by $k$” needs $k \neq 0$, and $k$ can be zero for arbitrarily small $h$ (if $g$ is locally constant). The repair is standard: define $\phi(k) = \frac{f(y+k)-f(y)}{k}$ for $k \neq 0$ and $\phi(0) = f'(y)$, which makes $\phi$ continuous at $0$; then $\frac{z(x+h)-z(x)}{h} = \phi(k)\cdot\frac{k}{h}$ holds for every small $h$, including those where $k = 0$, and the limit goes through. The idea is unchanged; the bookkeeping is just made airtight. Berkeley (Chapter 018 §5) would have asked, and he would have deserved an answer.

Use it on our chain

Back to house A, the chain $w_1 \to \hat{y} \to e \to \ell$, at $w_1 = 0$, $b = 0$.

Compute each link’s local rate — each one is a one-line derivative of a simple function:

Multiply along the chain:

Exactly the $-20$ that §15’s brute-force expansion produced — and obtained without expanding anything. Each factor came from differentiating a formula with at most two symbols in it.

And the same three factors, with the last one swapped, give the bias:

This is the shape of backpropagation. You never differentiate the whole network. You differentiate each link — one tiny, local, obvious derivative — and multiply along the path. Chapter 083 gives this an algorithm and a direction of travel; the mathematics is already here.

17. The Multivariable Chain Rule: Adding Up the Paths

One gap remains. Our real loss is an average over four houses, so $w_1$ reaches $L$ through four separate predictions, not one. Chains that fork need a rule for recombining.

The statement

Suppose $z = f(y_1, y_2, \ldots, y_m)$ and each $y_j$ depends on $t$. Then

In English: multiply along each path, then add over the paths. Every route from $t$ to $z$ contributes, and their contributions are independent, so they sum.

The derivation

It falls straight out of §8’s linear approximation. Nudge $t$ by $h$. Each intermediate moves by approximately $\Delta y_j \approx \frac{dy_j}{dt}h$. The linear approximation for $f$ says

Divide by $h$ and let $h\to 0$; the approximations become exact in the limit because their errors are second-order in $h$:

Why a sum, intuitively — if a change in $t$ raises your electricity bill by ₹30 and your water bill by ₹20, your total bill rises by ₹50. Separate channels of influence add.

A test we can check completely

Let $p = uv$, with $u = t^2$ and $v = t^3$. Two paths from $t$ to $p$.

Local rates: $\frac{\partial p}{\partial u} = v$, $\frac{\partial p}{\partial v} = u$, $\frac{du}{dt} = 2t$, $\frac{dv}{dt} = 3t^2$.

Multiply along paths and add:

Now check directly: $p = t^2\cdot t^3 = t^5$, and the power rule gives $\frac{dp}{dt} = 5t^4$ ✓. At $t=2$: $80$ both ways. The rule survives a case where we know the answer independently.

18. The Product Rule, For Free

Chapter 018 §7 derived four differentiation rules and postponed two. One of them just fell out of §17 without any extra work — notice that the calculation above never used the specific forms $u = t^2$, $v = t^3$ until the last step.

Redo it in general. Let $p = uv$ where $u$ and $v$ are both functions of $t$. The multivariable chain rule with $m = 2$ gives