019 — Partial Derivatives: One Dial at a Time

Where We Are

Chapter 018 gave us the derivative.

For a one-variable function

f(x),

the derivative tells us the local rate of change:

\frac{df}{dx}.

That is enough when there is only one dial.

But real machine-learning models have many dials.

Even the tiny house-price model

\hat y=w_1r+w_2a+b

already has three parameters:

w_1,\quad w_2,\quad b.

Its loss is not a curve.

It is a function of several variables:

L(w_1,w_2,b).

Now the phrase “the slope” becomes incomplete.

Slope in which direction?

Change $w_1$ ?

Change $w_2$ ?

Change $b$ ?

Change all three together?

To learn what each parameter is doing, we need to isolate them.

That is exactly what a partial derivative does.

1. The Problem: One Surface, Many Possible Slopes

Start with a simple two-variable function:

f(x,y)=x^2+y^2.

Its graph is a bowl.

             z
             ↑
            / \
          /     \
        /         \
      /             \
    -------------------→ x,y plane

Stand at the point

(x,y)=(2,1).

Ask:

How steep is the surface here?

That question is incomplete.

Walk in the x-direction and you experience one slope.

Walk in the y-direction and you experience another.

Walk diagonally and you experience something else.

With one variable there were only two local directions: left and right.

With two variables there are infinitely many directions around you.

With a billion parameters, the space is impossible to visualize—but the same issue remains.

2. A Tempting Wrong Idea: Move Everything Together

Suppose we try to imitate the one-variable derivative by nudging both inputs:

\frac{f(x+h,y+h)-f(x,y)}{h}.

This measures something real.

It tells us how $f$ changes when we move along the diagonal direction

(1,1).

But it does not tell us what x is doing separately from y.

At one point, x may be pushing the function upward while y pushes it downward.

Those effects could cancel.

A Tempting Wrong Idea

“Just nudge every parameter by the same small amount and read one slope.”

That measures one chosen direction through parameter space. It cannot tell us which parameter caused what, and a zero result does not imply every individual direction is flat.

We need one measurement per dial.

3. The Discovery: Freeze Everything Except One Variable

Take

f(x,y)=x^2+y^2.

To understand how $f$ changes with x, freeze y.

Suppose

y=1.

Then the two-variable function becomes a one-variable curve:

f(x,1)=x^2+1.

Now ordinary calculus works.

Differentiate with respect to x:

\frac{d}{dx}(x^2+1)=2x.

Because the original function has multiple inputs, we write this as

\boxed{ \frac{\partial f}{\partial x}=2x }

The symbol $\partial$ reminds us:

other variables exist, but they are being held fixed.

Likewise, freeze x and vary y:

\boxed{ \frac{\partial f}{\partial y}=2y }

These are partial derivatives.

4. What the Symbol Means

Compare:

\frac{df}{dx}

and

\frac{\partial f}{\partial x}.

Symbol	Typical context	Meaning
$df/dx$	one-variable function	local change in $f$ as x changes
$\partial f/\partial x$	multi-variable function	local change in $f$ as x changes while other inputs stay fixed

The curly $\partial$ is not a mysterious new kind of arithmetic.

It is a context marker.

“There are other variables here. I am deliberately moving only x.”

5. Tiny Numerical Example

Let

f(x,y)=x^2+3xy+y^2.

We want the partial derivative with respect to x.

Treat y as a constant.

Differentiate term by term:

\frac{\partial}{\partial x}(x^2)=2x,

\frac{\partial}{\partial x}(3xy)=3y,

because y behaves like a fixed number,

and

\frac{\partial}{\partial x}(y^2)=0.

Therefore

\boxed{ \frac{\partial f}{\partial x}=2x+3y }

Now with respect to y:

\frac{\partial}{\partial y}(x^2)=0,

\frac{\partial}{\partial y}(3xy)=3x,

\frac{\partial}{\partial y}(y^2)=2y.

\boxed{ \frac{\partial f}{\partial y}=3x+2y }

(x,y)=(2,1),

we get

\frac{\partial f}{\partial x}=2(2)+3(1)=7,

\frac{\partial f}{\partial y}=3(2)+2(1)=8.

Interpretation:

nudging x upward changes $f$ locally at about 7 output units per x-unit,
nudging y upward changes $f$ locally at about 8 output units per y-unit.

One point. Two different sensitivities.

6. Partial Derivatives Are Slices Through a Surface

A two-variable function creates a surface.

To take

\frac{\partial f}{\partial x},

freeze y.

That cuts the surface with a vertical plane parallel to the x-axis.

The intersection is a curve.

Then take the ordinary slope of that curve.

To take

\frac{\partial f}{\partial y},

freeze x and slice the other way.

So a partial derivative is not an abstract trick.

It is the slope of a cross-section of the surface.

7. The Axis-Direction Interpretation

Chapter 006 taught us about basis vectors.

In 2D:

\mathbf e_1=\begin{bmatrix}1\\0\end{bmatrix}, \qquad \mathbf e_2=\begin{bmatrix}0\\1\end{bmatrix}.

Changing x while holding y fixed means moving along

\mathbf e_1.

Changing y while holding x fixed means moving along

\mathbf e_2.

So partial derivatives are local rates of change along the coordinate axes.

In n dimensions:

\frac{\partial f}{\partial x_j}

measures change along basis direction

\mathbf e_j.

Linear algebra has returned inside calculus.

8. Definition From the One-Variable Derivative

For

f(x_1,\ldots,x_n),

the partial derivative with respect to $x_j$ is

\boxed{ \frac{\partial f}{\partial x_j}(\mathbf x) = \lim_{h\to0} \frac{ f(\mathbf x+h\mathbf e_j)-f(\mathbf x) }{h} }

Read it slowly:

start at the current input vector,
move only in coordinate direction j,
measure output change,
divide by the tiny step,
shrink the step toward zero.

This is exactly the derivative from Chapter 018, restricted to one coordinate direction.

9. Return to the House Model

Model:

\hat y=w_1r+w_2a+b.

For one house, squared loss:

L=(\hat y-y)^2.

The loss depends on three parameters:

L(w_1,w_2,b).

A partial derivative answers questions like:

\frac{\partial L}{\partial w_1}

If I change only the room weight, how does loss change?

\frac{\partial L}{\partial w_2}

If I change only the area weight, how does loss change?

\frac{\partial L}{\partial b}

If I change only the bias, how does loss change?

That is precisely the information training needs.

10. A Fully Worked ML Example

Take one training example:

r=2, \qquad a=3, \qquad y=10.

Use a simplified model

\hat y=w_1r+w_2a+b.

Let current parameters be

w_1=1, \quad w_2=1, \quad b=0.

Prediction:

\hat y=1(2)+1(3)+0=5.

Error:

e=\hat y-y=5-10=-5.

Loss:

L=e^2=25.

Now ask how the loss reacts to each dial.

We can reason numerically first.

Increase $w_1$ slightly.

Because r=2, prediction rises twice as fast as $w_1$ .

Increase $w_2$ slightly.

Because a=3, prediction rises three times as fast as $w_2$ .

Increase b slightly.

Prediction rises one-for-one.

Since prediction is currently too low, all three increases reduce loss—but with different sensitivities.

This is why one scalar “slope” cannot train all parameters correctly.

11. Partial Derivatives Through Composition Preview

The loss is not directly written as a simple polynomial in every parameter unless we substitute everything.

Instead we have stages:

w_1,w_2,b \rightarrow \hat y \rightarrow e \rightarrow L.

Changing $w_1$ changes prediction.

Prediction changes error.

Error changes loss.

So to compute

\frac{\partial L}{\partial w_1},

we will need to combine local rates of change across stages.

That is the chain rule—Chapter 021.

For now, the key point is that partial derivatives define which input variable’s influence we are tracking.

12. Partial Derivative Rules

Once other variables are treated as constants, ordinary derivative rules apply.

Let

f(x,y)=4x^3y+2xy^2+7y.

With respect to x:

\frac{\partial f}{\partial x} = 12x^2y+2y^2.

Why does $7y$ disappear?

Because y is frozen, so $7y$ is constant with respect to x.

With respect to y:

\frac{\partial f}{\partial y} = 4x^3+4xy+7.

Now x is frozen instead.

Same expression. Different variable being allowed to move.

13. What “Hold Constant” Really Means

Suppose

f(x,y)=xy.

(x,y)=(3,10),

with respect to x:

\frac{\partial f}{\partial x}=y=10.

That means:

if y remains exactly 10 and x changes a tiny amount, output changes about 10 times that amount.

With respect to y:

\frac{\partial f}{\partial y}=x=3.

Meaning:

if x remains exactly 3 and y changes a tiny amount, output changes about 3 times that amount.

A partial derivative describes a controlled experiment.

Move one cause. Freeze the others.

14. Finite-Difference Check

We can estimate a partial derivative numerically.

For

f(x,y)=x^2+3xy+y^2,

(2,1),

analytically:

\frac{\partial f}{\partial x}=7.

Use a small

h=10^{-5}.

Then estimate:

\frac{f(2+h,1)-f(2,1)}{h}.

It should be very close to 7.

This is called a finite-difference gradient check when applied across many parameters.

It becomes a powerful debugging tool for backpropagation.

15. Why Not Use an Extremely Tiny h?

It seems that smaller h should always be better.

But computers use finite-precision floating-point numbers.

If h is too tiny, subtracting

f(x+h)-f(x)

can lose meaningful digits because the two numbers are nearly identical.

This is catastrophic cancellation.

So numerical derivative checks balance two errors:

h too large → approximation error,
h too small → floating-point error.

A value around

10^{-5}\text{ or }10^{-6}

is often a reasonable starting point for double precision, depending on the problem.

16. Partial Derivatives Are Not Yet the Whole Direction Story

Suppose at one point:

\frac{\partial f}{\partial x}=3, \qquad \frac{\partial f}{\partial y}=4.

We know slopes along x and y axes.

But what is the slope if we move diagonally?

For example, along direction

\mathbf u = \frac1{\sqrt2} \begin{bmatrix}1\\1\end{bmatrix}?

Partial derivatives alone look like separate facts.

We need to organize them into one object that can answer any direction question.

That object will be the gradient in Chapter 020.

17. Mixed Partial Derivatives

We can differentiate a partial derivative again.

For

f(x,y)=x^2y+3xy^2,

first:

\frac{\partial f}{\partial x}=2xy+3y^2.

Then differentiate that with respect to y:

\frac{\partial^2f}{\partial y\partial x} =2x+6y.

Reverse order:

\frac{\partial f}{\partial y}=x^2+6xy,

then

\frac{\partial^2f}{\partial x\partial y} =2x+6y.

They match here.

For sufficiently smooth functions, mixed partials are equal under standard regularity conditions.

Later this structure forms the Hessian matrix.

18. Shape Check

f:\mathbb R^3\to\mathbb R,

then each partial derivative at a point is a scalar:

∂f/∂x1 → scalar
∂f/∂x2 → scalar
∂f/∂x3 → scalar

Three inputs create three coordinate sensitivities.

If a neural network has one million scalar parameters, the loss has one million partial derivatives.

We clearly need a compact way to organize them.

19. Code From Scratch

def f(x, y):
    return x*x + 3*x*y + y*y


def partial_x(f, x, y, h=1e-6):
    return (f(x + h, y) - f(x, y)) / h


def partial_y(f, x, y, h=1e-6):
    return (f(x, y + h) - f(x, y)) / h

print(partial_x(f, 2.0, 1.0))  # near 7
print(partial_y(f, 2.0, 1.0))  # near 8

Notice the implementation mirrors the definition exactly.

For partial_x, y never moves.

For partial_y, x never moves.

20. Break It: Move More Than One Variable

If you compute

(f(x+h, y+h) - f(x,y)) / h

you are not estimating either partial derivative.

You are measuring change along a diagonal direction.

That can be useful, but it is a different quantity.

This is a common conceptual mistake in multivariable calculus.

21. Break It: Forget Which Variable Is Constant

Take

f(x,y)=xy.

A learner may say:

derivative with respect to x is x.

But that confuses the variables.

With respect to x, y is the fixed multiplier:

\frac{\partial f}{\partial x}=y.

With respect to y:

\frac{\partial f}{\partial y}=x.

Always ask:

Which dial am I moving?

Everything else becomes constant for this measurement.

22. Break It: Confuse Parameter Derivative With Input Derivative

A model

\hat y=wx+b

has different derivatives depending on what we differentiate with respect to.

With respect to input x:

\frac{\partial \hat y}{\partial x}=w.

With respect to weight w:

\frac{\partial \hat y}{\partial w}=x.

With respect to bias b:

\frac{\partial \hat y}{\partial b}=1.

Same formula. Different question. Different answer.

This distinction is central to backpropagation.

23. 🎯 Machine-Learning Connection — Parameter Sensitivity

A partial derivative such as

\frac{\partial L}{\partial w_j}

means:

if only parameter $w_j$ changes slightly, how sensitive is the loss to that change?

If it is strongly positive, increasing $w_j$ raises loss quickly.

If strongly negative, increasing $w_j$ lowers loss quickly.

If near zero, loss is locally insensitive to that parameter.

Training needs all these sensitivities.

24. 🎯 Machine-Learning Connection — Input Sensitivity

We can also differentiate output with respect to inputs:

\frac{\partial \hat y}{\partial x_j}.

This can reveal how sensitive a prediction is to a feature.

In image models, derivatives with respect to pixels are related to saliency methods and adversarial perturbations.

In optimization, derivatives with respect to parameters guide learning.

The mathematics is the same; the interpretation changes with the variable.

25. History Lens — Extending Calculus Beyond One Variable

Once calculus proved useful for curves, surfaces and physical systems demanded a broader language.

Quantities in mechanics, geometry and later thermodynamics depend on many variables at once.

Partial differentiation emerged as the natural way to isolate one variable’s local influence while keeping others fixed.

Modern machine learning pushes this idea to extraordinary scale: a loss may depend on billions of parameters, yet conceptually each partial derivative still asks the same simple question:

What happens if I nudge this one dial?

26. Distinctions That Matter

Pair	Difference
ordinary derivative vs partial derivative	one input variable vs one variable among many
partial derivative vs directional derivative	coordinate-axis direction vs arbitrary direction
parameter derivative vs input derivative	sensitivity to learned dial vs sensitivity to data
analytical derivative vs finite difference	symbolic/exact formula vs numerical approximation
first partial vs mixed second partial	first-order sensitivity vs how one sensitivity changes with another variable

27. What We Discovered

Multi-variable functions do not have one scalar slope.
A partial derivative changes one variable while freezing all others.
$\partial$ signals that other independent variables exist.
Partial derivatives are slopes of axis-aligned slices through a surface.
They are directional rates along coordinate basis vectors.
Ordinary derivative rules apply once other variables are treated as constants.
Finite differences can numerically verify partial derivatives.
Too-small finite-difference steps can suffer floating-point cancellation.
Different differentiation variables produce different meanings and formulas.
Partial derivatives give one sensitivity per parameter.
We still need one object that organizes all these sensitivities and predicts change in arbitrary directions.

28. Mathematics We Built

For a multivariable function

f(x_1,\ldots,x_n),

the j-th partial derivative is

\frac{\partial f}{\partial x_j} = \lim_{h\to0} \frac{ f(\mathbf x+h\mathbf e_j)-f(\mathbf x) }{h}.

For

f(x,y)=x^2+3xy+y^2,

we found

\frac{\partial f}{\partial x}=2x+3y,

\frac{\partial f}{\partial y}=3x+2y.

At $(2,1)$ :

\frac{\partial f}{\partial x}=7, \qquad \frac{\partial f}{\partial y}=8.

29. One-Minute Explanation

A partial derivative is what you use when a function has many inputs but you want to measure the local effect of just one. Freeze every other input, nudge the chosen one, and compute the ordinary derivative of that one-variable slice. For a loss function with millions of parameters, there is one partial derivative per parameter. Each tells how sensitive the loss is to that dial. The next problem is that a million separate numbers are awkward; we need to organize them into one geometric object. That object is the gradient.

30. Mastery Check

Why is “the slope of a surface” incomplete?
What does “hold other variables constant” actually mean?
Why does $\partial$ differ conceptually from $d$ ?
Compute both partial derivatives of $x^2+3xy+y^2$ .
What is the axis-slice interpretation of a partial derivative?
How do basis vectors appear in the formal definition?
Why is moving x and y together not a partial derivative?
Why can finite-difference checks fail when h is too tiny?
For $\hat y=wx+b$ , compute derivatives with respect to x, w and b and explain why they differ.
What information does $\partial L/\partial w_j$ provide during training?
Why are partial derivatives still not enough to answer “which arbitrary direction rises fastest?”

🔭 Bridge to Chapter 020

At one point we may have

\frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}, \ldots, \frac{\partial f}{\partial x_n}.

A billion-parameter model has a billion such sensitivities.

Listing them separately is not enough.

We want one object that:

stores all partial derivatives,
predicts change in any direction,
points toward steepest increase.

What vector contains the complete first-order local change information of a scalar function?

That question forces the gradient.

Level 1 — Explain It

Explain a partial derivative using a mixing-desk analogy.
Why is “the slope of a surface” incomplete?
What does it mean to hold every other variable fixed?
Explain why $\partial f/\partial x$ is not ordinary division.
What is the difference between a partial derivative and a directional derivative?

Level 2 — Calculate It

For

f(x,y)=x^2+3xy+y^2,

calculate by hand:

\frac{\partial f}{\partial x}

and

\frac{\partial f}{\partial y}.

Evaluate both at:

$(0,0)$
$(1,1)$
$(2,1)$
$(-1,2)$

Interpret the sign of each result.

Level 3 — Derive It

Use the limit definition to derive

\frac{\partial}{\partial x}(x^2+3xy+y^2)=2x+3y.

Start with

\frac{f(x+h,y)-f(x,y)}{h}

and simplify before taking $h\to0$ .

Repeat for the y partial derivative.

Level 4 — Predict It

For

f(x,y,z)=x^2y+yz^2+3x,

predict before calculating:

Which terms disappear in $\partial f/\partial x$ ?
Which terms disappear in $\partial f/\partial y$ ?
Which terms disappear in $\partial f/\partial z$ ?

Then derive all three.

Level 5 — Break It

A. Move two variables together

Compare

(f(x+h, y) - f(x,y)) / h

with

(f(x+h, y+h) - f(x,y)) / h

for the same function and point.

Explain why only the first estimates $\partial f/\partial x$ .

B. Too-small finite-difference step

Estimate a partial derivative using

h = 1e-2, 1e-4, 1e-6, 1e-9, 1e-12, 1e-15

Compare with the analytical answer.

Explain why accuracy eventually gets worse.

C. Different differentiation targets

For

\hat y=wx+b,

compute

\partial\hat y/\partial x, \quad \partial\hat y/\partial w, \quad \partial\hat y/\partial b.

Explain why the same equation produces three different sensitivities.

Level 6 — Build It

Implement a finite-difference partial derivative helper:

def partial(f, point, axis, h=1e-6):
    pass

Requirements:

point can contain any number of variables
only one coordinate may move
compare against analytical derivatives on at least five functions
include an explicit experiment showing the danger of extremely small h

ML Investigation — One Training Example

Use

\hat y=w_1r+w_2a+b

with

r = 2
a = 3
y = 10
w1 = 1
w2 = 1
b = 0

and squared loss

L=(\hat y-y)^2.

Compute the current prediction and loss.
Estimate $\partial L/\partial w_1$ numerically.
Estimate $\partial L/\partial w_2$ numerically.
Estimate $\partial L/\partial b$ numerically.
Which parameter has the largest local sensitivity magnitude?
Why are the sensitivities different even though all three parameters affect the same prediction?

Do not use autograd.

Shape Check

For

f:\mathbb{R}^4\to\mathbb{R},

predict:

f(x)                 → scalar
∂f/∂x1               → ______
∂f/∂x2               → ______
∂f/∂x3               → ______
∂f/∂x4               → ______
number of first partials → ______

Then answer: what object would naturally hold all four values together?

Mastery Gate

Do not continue until you can answer:

Why does a multivariable function have many coordinate slopes?
What precisely is frozen in a partial derivative?
How is the partial-derivative limit definition connected to a basis vector?
Why can moving all parameters together hide individual effects?
Why is finite difference useful for debugging but not usually used for training large networks?
Why does the variable you differentiate with respect to change the interpretation?
What problem remains after computing every partial derivative separately?