020 — Gradients: The Direction of Steepest Change

Where We Are

Chapter 019 gave us one partial derivative per input coordinate.

For

f(x,y),

we can compute

\frac{\partial f}{\partial x}

and

\frac{\partial f}{\partial y}.

These tell us what happens when we move along the x-axis or y-axis.

But optimization does not have to move only along coordinate axes.

From one point on a loss surface, we can move diagonally, sideways, or in any mixture of parameter directions.

So a new question appears:

If I know all the partial derivatives, can I predict the slope in an arbitrary direction?

And an even more important one:

Which direction makes the function increase fastest?

Both answers come from one vector: the gradient.

1. The Problem: A Million Separate Numbers Are Not Yet Geometry

Suppose at one point we find

\frac{\partial f}{\partial x}=3

and

\frac{\partial f}{\partial y}=4.

What do we know?

If we nudge x while freezing y, the function rises about 3 units per x-unit.

If we nudge y while freezing x, it rises about 4 units per y-unit.

But what if we move along

\mathbf u= \frac{1}{\sqrt2} \begin{bmatrix}1\\1\end{bmatrix}?

What if we move along

\begin{bmatrix}-1\\2\end{bmatrix}?

What if our model has a billion parameters?

We need to convert the list of partial sensitivities into geometry.

2. Put the Partial Derivatives Into a Vector

For

f(x,y),

define

\boxed{ \nabla f = \begin{bmatrix} \frac{\partial f}{\partial x}\\[4pt] \frac{\partial f}{\partial y} \end{bmatrix} }

The upside-down triangle

\nabla

is called nabla or del.

The vector

\nabla f

is the gradient.

For n variables,

\boxed{ \nabla f(\mathbf x) = \begin{bmatrix} \partial f/\partial x_1\\ \partial f/\partial x_2\\ \vdots\\ \partial f/\partial x_n \end{bmatrix} }

One coordinate sensitivity per coordinate direction.

3. Tiny Example

Let

f(x,y)=x^2+y^2.

Then

\frac{\partial f}{\partial x}=2x,

\frac{\partial f}{\partial y}=2y.

Therefore

\boxed{ \nabla f(x,y)= \begin{bmatrix}2x\\2y\end{bmatrix} }

(3,4),

we get

\nabla f(3,4) = \begin{bmatrix}6\\8\end{bmatrix}.

This vector points directly away from the origin.

That makes geometric sense: for the bowl

x^2+y^2,

moving directly away from the center increases height fastest.

4. The Gradient Is Not the Function Value

At $(3,4)$ :

f(3,4)=9+16=25.

But

\nabla f(3,4)= \begin{bmatrix}6\\8\end{bmatrix}.

These objects answer different questions.

Object	Meaning
$f(3,4)=25$	height of the surface
$\nabla f(3,4)=[6,8]^T$	local sensitivity vector

A scalar value says where you are vertically.

A gradient says how the surface changes around you.

5. From Axis Slopes to Any Direction

Suppose we take a tiny step

\Delta\mathbf x = \begin{bmatrix}\Delta x\\\Delta y\end{bmatrix}.

For a smooth function, the first-order local change is approximately

\Delta f \approx \frac{\partial f}{\partial x}\Delta x + \frac{\partial f}{\partial y}\Delta y.

But that expression is exactly a dot product:

\boxed{ \Delta f \approx \nabla f\cdot\Delta\mathbf x }

This is one of the most important equations in the course.

The gradient packages all coordinate sensitivities so a dot product can predict local change in any small direction.

6. Derive the Directional Derivative

Let

\mathbf u

be a unit vector describing a direction.

Take a tiny step of size h:

\Delta\mathbf x=h\mathbf u.

Then

\Delta f \approx \nabla f\cdot(h\mathbf u) = h(\nabla f\cdot\mathbf u).

Divide by h:

\frac{\Delta f}{h} \approx \nabla f\cdot\mathbf u.

As h shrinks, this becomes the directional derivative:

\boxed{ D_{\mathbf u}f = \nabla f\cdot\mathbf u }

So once we know the gradient, we know the local slope in every unit direction.

A billion directional questions collapse into one vector plus a dot product.

7. Why the Gradient Points Uphill Fastest

From Chapter 007:

\mathbf a\cdot\mathbf b = \|\mathbf a\|\|\mathbf b\|\cos\theta.

Apply that to the directional derivative:

D_{\mathbf u}f = \nabla f\cdot\mathbf u.

Since $\mathbf u$ is a unit vector,

\|\mathbf u\|=1.

Therefore

D_{\mathbf u}f = \|\nabla f\|\cos\theta,

where $\theta$ is the angle between the gradient and the chosen direction.

Now ask: when is this largest?

Cosine reaches its maximum value 1 when

\theta=0.

That means

\mathbf u

points in exactly the same direction as the gradient.

Therefore:

\boxed{ \nabla f\text{ points in the direction of steepest local increase} }

And its magnitude

\|\nabla f\|

is the maximum local rate of increase per unit step.

This is not a memorized slogan.

It falls directly out of the dot product.

8. The Opposite Direction Goes Downhill Fastest

If the gradient points uphill fastest, then

-\nabla f

points downhill fastest.

Because the angle between $\nabla f$ and $-\nabla f$ is $180^\circ$ ,

\cos180^\circ=-1.

So the directional derivative becomes

-\|\nabla f\|.

This is the steepest possible local decrease per unit movement.

That is the geometric reason gradient descent moves in the negative gradient direction.

9. Contour Lines

A contour line contains points with the same function value.

For

f(x,y)=x^2+y^2,

contours satisfy

x^2+y^2=c.

These are circles.

        ↑ gradient
      ↗
   ( contour )
  (    •      )
   (         )

Moving along a contour does not change $f$ locally.

So the directional derivative along the contour tangent is zero.

That means

\nabla f\cdot\mathbf t=0.

Therefore the gradient is perpendicular to contour lines.

Intuition — A contour map on a mountain connects equal-height locations. Walking along a contour keeps height unchanged. The steepest uphill direction crosses contours at right angles.

10. Worked Example With a Non-Circular Surface

Let

f(x,y)=x^2+4y^2.

Then

\nabla f= \begin{bmatrix}2x\\8y\end{bmatrix}.

(1,1),

\nabla f(1,1)= \begin{bmatrix}2\\8\end{bmatrix}.

This does not point diagonally at 45°.

Why?

Because the function changes four times more strongly with y than x near comparable coordinates.

The contour ellipses are narrow in y.

The gradient points across them in the direction where height increases fastest.

11. Directional Derivative Example

At $(1,1)$ for

f(x,y)=x^2+4y^2,

we have

\nabla f= \begin{bmatrix}2\\8\end{bmatrix}.

Take unit direction

\mathbf u= \frac{1}{\sqrt2} \begin{bmatrix}1\\1\end{bmatrix}.

Then

D_{\mathbf u}f = \begin{bmatrix}2\\8\end{bmatrix} \cdot \frac1{\sqrt2} \begin{bmatrix}1\\1\end{bmatrix}.

D_{\mathbf u}f = \frac{10}{\sqrt2} =5\sqrt2 \approx7.07.

But gradient magnitude is

\sqrt{2^2+8^2} = \sqrt{68} \approx8.25.

So the diagonal direction rises quickly—but not as quickly as the actual gradient direction.

12. A Tempting Wrong Idea: “Choose the Largest Partial Derivative Axis”

Suppose

\nabla f= \begin{bmatrix}3\\4\end{bmatrix}.

The largest component is 4, so maybe we should move only along y.

Slope along y is 4.

But gradient magnitude is

5.

Moving along normalized gradient

\frac15 \begin{bmatrix}3\\4\end{bmatrix}

gives directional derivative 5.

So combining coordinates can produce a steeper direction than any single axis.

A Tempting Wrong Idea

The steepest direction is not necessarily “the axis with the biggest partial.” It is the vector formed by all partial derivatives together.

13. Gradient of a Loss Function

Suppose our model has parameters

\boldsymbol\theta= \begin{bmatrix} w_1\\w_2\\b \end{bmatrix}.

Loss is

L(\boldsymbol\theta).

The gradient is

\boxed{ \nabla_{\boldsymbol\theta}L = \begin{bmatrix} \partial L/\partial w_1\\ \partial L/\partial w_2\\ \partial L/\partial b \end{bmatrix} }

Each component tells one coordinate sensitivity.

The whole vector tells the locally steepest uphill direction in parameter space.

Therefore

-\nabla_{\boldsymbol\theta}L

is the natural local downhill direction.

14. The Gradient Has the Same Shape as the Parameter Vector

\boldsymbol\theta\in\mathbb R^n

and

L:\mathbb R^n\to\mathbb R,

then

\nabla_{\boldsymbol\theta}L\in\mathbb R^n.

One parameter coordinate → one gradient coordinate.

For 100 million parameters, the gradient has 100 million scalar entries.

That shape correspondence is fundamental to training.

15. Local Linear Approximation

The gradient gives more than a direction.

For a small parameter change

\Delta\boldsymbol\theta,

we can approximate

\boxed{ L(\boldsymbol\theta+\Delta\boldsymbol\theta) \approx L(\boldsymbol\theta) + \nabla L(\boldsymbol\theta)^T \Delta\boldsymbol\theta }

This is the multivariable version of “zoom in and a smooth curve looks like a line.”

A smooth surface locally looks like a plane.

The gradient stores that plane’s slopes.

16. Why This Approximation Matters for Optimization

Suppose we choose

\Delta\boldsymbol\theta=-\eta\nabla L,

where

\eta>0

is small.

Then local approximation gives

\Delta L \approx \nabla L^T(-\eta\nabla L).

\Delta L \approx -\eta\|\nabla L\|^2.

Since

\|\nabla L\|^2\ge0,

this is non-positive.

Therefore, for a sufficiently small step in a smooth region, moving along negative gradient should reduce loss.

This is the mathematical seed of gradient descent.

17. But the Gradient Is Local

A crucial warning:

The gradient tells us what happens near the current point.

It does not guarantee what happens after a huge step.

Imagine standing on a curved mountain.

The steepest downhill direction at your feet may not remain downhill for 10 kilometres.

Likewise, a large learning rate can overshoot a valley even when the gradient direction is correct.

Direction and step size are different problems.

Chapter 024 will study that deeply.

18. Gradient Zero Does Not Always Mean Minimum

\nabla f=0,

all first-order coordinate slopes vanish.

This can occur at:

Minimum

f(x,y)=x^2+y^2

at $(0,0)$ .

Maximum

f(x,y)=-(x^2+y^2)

at $(0,0)$ .

Saddle

f(x,y)=x^2-y^2

at $(0,0)$ .

The first-order gradient alone cannot distinguish them.

Curvature—second derivatives—becomes relevant.

19. Saddle Point Geometry

For

f(x,y)=x^2-y^2,

\nabla f= \begin{bmatrix}2x\\-2y\end{bmatrix}.

At the origin:

\nabla f(0,0)=0.

But along x:

f(x,0)=x^2

rises away from zero.

Along y:

f(0,y)=-y^2

falls away from zero.

So the origin is neither minimum nor maximum.

This matters in deep learning because high-dimensional loss surfaces contain many saddle-like regions.

20. Gradients and Feature Scaling

Suppose one parameter direction naturally produces gradient component

0.001

and another produces

1000.

Then the gradient vector is extremely imbalanced.

This can happen because input features or curvature have very different scales.

Optimization may zig-zag or progress slowly.

This connects back to feature scaling from Chapter 006 and forward to:

normalization,
adaptive optimizers,
conditioning,
learning-rate schedules.

21. Gradient of a Simple Linear Model

Take

\hat y=wx+b

and one-example squared loss

L=(\hat y-y)^2.

Suppose

x=3, \quad y=10, \quad w=2, \quad b=1.

Prediction:

\hat y=2(3)+1=7.

Error:

e=7-10=-3.

Soon the chain rule will give

\frac{\partial L}{\partial w}=2ex,

\frac{\partial L}{\partial b}=2e.

Numerically:

\frac{\partial L}{\partial w}=2(-3)(3)=-18,

\frac{\partial L}{\partial b}=2(-3)=-6.

\nabla L= \begin{bmatrix}-18\\-6\end{bmatrix}.

The negative components say increasing both w and b locally lowers loss.

The w direction is three times more sensitive here because x=3.

22. Finite-Difference Gradient Check

We can estimate every gradient component numerically.

For parameter vector

\boldsymbol\theta=[\theta_1,\ldots,\theta_n],

estimate component j by

\frac{ L(\boldsymbol\theta+h\mathbf e_j) - L(\boldsymbol\theta) }{h}.

Repeat for every j.

This is expensive: n parameters require roughly n additional loss evaluations.

But it is useful for checking analytical/backprop gradients on small models.

Backpropagation will compute all needed derivatives far more efficiently.

23. Why We Usually Use Central Differences for Checking

Forward difference:

\frac{f(x+h)-f(x)}{h}

has first-order approximation error.

A better numerical check often uses central difference:

\boxed{ \frac{f(x+h)-f(x-h)}{2h} }

It samples symmetrically around the point and usually has smaller truncation error for smooth functions.

This is a numerical-analysis tool, not the training algorithm itself.

24. Code: Build a Gradient Numerically

import numpy as np


def numerical_gradient(f, x, h=1e-6):
    x = np.asarray(x, dtype=float)
    grad = np.zeros_like(x)

    for i in range(len(x)):
        xp = x.copy()
        xm = x.copy()
        xp[i] += h
        xm[i] -= h
        grad[i] = (f(xp) - f(xm)) / (2*h)

    return grad


def bowl(v):
    x, y = v
    return x*x + 4*y*y

print(numerical_gradient(bowl, [1., 1.]))
# approximately [2., 8.]

The loop nudges one coordinate at a time.

That is Chapter 019 turned into a vector.

25. Shape Check

For

L:\mathbb R^{10}\to\mathbb R,

parameter vector θ     shape (10,)
loss L(θ)              scalar
gradient ∇L            shape (10,)
small step Δθ           shape (10,)
∇L · Δθ                scalar

The dot product produces predicted scalar loss change.

That shape story is worth memorizing because it generalizes.

26. Gradients of Matrix Parameters

Neural networks often have a weight matrix

W\in\mathbb R^{m\times n}

rather than one flat vector.

The gradient with respect to W is arranged with the same shape:

\frac{\partial L}{\partial W} \in\mathbb R^{m\times n}.

Each entry

\left(\frac{\partial L}{\partial W}\right)_{ij} = \frac{\partial L}{\partial W_{ij}}.

So “gradient has the same shape as the thing being differentiated” becomes a powerful practical rule for scalar losses.

Later Chapter 025 will develop matrix calculus carefully.

27. Gradient vs Jacobian

The word gradient is most naturally used when a scalar-valued function has vector input:

f:\mathbb R^n\to\mathbb R.

If the function itself outputs a vector,

\mathbf f:\mathbb R^n\to\mathbb R^m,

then each output has its own gradient.

Stacking those derivatives produces a Jacobian matrix.

We do not need full Jacobian machinery yet, but this distinction will matter in matrix calculus and autodiff.

28. A Tempting Wrong Idea: “Negative Gradient Is Always Globally Best”

The negative gradient is the steepest local descent direction under ordinary Euclidean geometry and unit-length steps.

It does not claim:

that the global minimum lies directly that way,
that a huge step is safe,
that Euclidean geometry is always the best optimization geometry,
that gradient descent is the fastest possible optimizer.

A Tempting Wrong Idea

“The gradient points to the minimum.”

No. The gradient points uphill locally. Negative gradient points downhill locally. The minimum may require a curved path and many updates.

29. History Lens — From Surfaces to Optimization

Multivariable calculus converted geometric surfaces into local algebraic information.

The gradient became central because it links three ideas at once:

coordinate partial derivatives,
directional rates of change,
steepest ascent.

Optimization then turns the sign around and uses negative gradients to search for minima.

Modern deep learning scales this classical geometry to parameter spaces with millions or billions of dimensions.

The picture is impossible to draw, but the mathematics is unchanged.

30. Distinctions That Matter

Pair	Difference
partial derivative vs gradient	one coordinate sensitivity vs vector of all sensitivities
gradient vs directional derivative	vector of local information vs scalar slope along one chosen direction
gradient vs function value	local change information vs current scalar height
gradient vs Jacobian	scalar-output derivative vector vs vector-output derivative matrix
gradient direction vs step size	where to move vs how far to move
stationary point vs minimum	zero first-order gradient vs actual local/global optimum

31. What We Discovered

The gradient collects all partial derivatives into one vector.
It has one component per input coordinate.
Local change satisfies

\Delta f\approx\nabla f\cdot\Delta\mathbf x.

Directional derivative is

D_{\mathbf u}f=\nabla f\cdot\mathbf u.

The dot-product formula proves the gradient points toward steepest local increase.
Negative gradient points toward steepest local decrease.
Gradient magnitude is the maximum local directional slope for a unit step.
Gradients are perpendicular to level sets/contours.
Zero gradient does not prove minimum; maxima and saddles can also be stationary.
Numerical gradient checks are useful but expensive.
Gradient shape mirrors parameter shape for scalar losses.
We still need a way to compute gradients efficiently through deeply composed functions.

32. Mathematics We Built

Gradient

\nabla f= \begin{bmatrix} \partial f/\partial x_1\\ \vdots\\ \partial f/\partial x_n \end{bmatrix}.

Directional derivative

D_{\mathbf u}f = \nabla f\cdot\mathbf u.

Dot-product form

D_{\mathbf u}f = \|\nabla f\|\cos\theta.

Local linear approximation

f(\mathbf x+\Delta\mathbf x) \approx f(\mathbf x) + \nabla f(\mathbf x)^T\Delta\mathbf x.

Negative-gradient local loss change

\Delta\mathbf x=-\eta\nabla f,

then

\Delta f \approx -\eta\|\nabla f\|^2.

33. One-Minute Explanation

A multivariable function has one partial derivative per input coordinate. Put those partial derivatives into a vector and you get the gradient. The gradient is powerful because its dot product with any unit direction gives the slope in that direction. The dot-product geometry then proves that the gradient itself points in the direction of steepest local increase, while the negative gradient points in the direction of steepest local decrease. For optimization, that gives us the direction to move—but not yet how to compute these sensitivities efficiently through many nested functions. That is where the chain rule enters.

34. Mastery Check

Write the gradient of $x^2+4y^2$ .
What is the difference between a partial derivative and a gradient?
Why does $\nabla f\cdot\Delta x$ approximate local change?
Derive the directional derivative formula.
Use the dot product to prove the gradient is steepest ascent.
Why is negative gradient steepest local descent?
Why is the gradient perpendicular to contour lines?
Why does zero gradient not guarantee a minimum?
What is the gradient shape for a scalar loss with a (5,3) weight matrix?
Why can a correct gradient still produce a bad update if the learning rate is huge?
What is the distinction between a gradient and a Jacobian?
Why is numerical gradient estimation impractical for training a billion-parameter network?

🔭 Bridge to Chapter 021

We now know what gradient information we want.

But a neural-network loss is not usually one simple formula.

It is a composition:

\text{parameter} \rightarrow \text{linear operation} \rightarrow \text{activation} \rightarrow \cdots \rightarrow \text{prediction} \rightarrow \text{loss}.

A parameter affects the loss only through everything downstream of it.

How do we multiply local rates of change through a chain of computations?

That question forces the chain rule.

Level 1 — Explain It

Explain the gradient in plain English.
Why is a gradient a vector rather than a scalar?
What is the difference between a partial derivative and a directional derivative?
Why does the gradient point uphill fastest?
Why is the negative gradient used for descent?
Why does a zero gradient not automatically mean “minimum”?

Level 2 — Calculate It

For

f(x,y)=x^2+4y^2,

derive $\nabla f$ ,
evaluate it at $(1,1)$ ,
calculate its magnitude,
normalize it,
calculate the directional derivative along

\mathbf u=\frac1{\sqrt2}[1,1]^T.

Compare the directional slope with $\|\nabla f\|$ .

Level 3 — Derive It

Starting from

D_{\mathbf u}f=\nabla f\cdot\mathbf u,

use

\mathbf a\cdot\mathbf b=\|\mathbf a\|\|\mathbf b\|\cos\theta

to derive why the maximum directional derivative occurs when $\mathbf u$ points along the gradient.

Then derive why the minimum occurs in the opposite direction.

Level 4 — Predict It

Without plotting, predict the gradient direction for:

$f(x,y)=x^2+y^2$ at $(3,4)$
$f(x,y)=x^2+10y^2$ at $(1,1)$
$f(x,y)=-(x^2+y^2)$ at $(2,0)$
$f(x,y)=x^2-y^2$ at $(1,1)$

Sketch contour lines and gradient arrows afterward.

Level 5 — Break It

A. Gradient zero but not minimum

Investigate

f(x,y)=x^2-y^2.

Show that the gradient is zero at the origin, then move along x and y separately.

Explain why this is a saddle.

B. Too-large negative-gradient step

Use

f(x)=x^2

starting at $x=2$ .

Try update sizes corresponding to several learning rates. Show that the negative-gradient direction can be right while the step is still too large.

C. Biggest partial is not steepest direction

Use a gradient [3,4]. Compare slope along x-axis, y-axis and normalized gradient direction.

Level 6 — Build It

Implement central-difference numerical gradient:

def numerical_gradient(f, x, h=1e-6):
    pass

Requirements:

support arbitrary-length 1D parameter vectors
move only one coordinate at a time
use central differences
verify on at least five analytical functions
compare errors for several h values

Investigation — Contours and Gradients

For

f(x,y)=x^2+4y^2,

draw contour lines,
choose at least 10 points,
calculate gradients,
draw gradient arrows,
verify visually that arrows cross contours at right angles.

Then repeat for

f(x,y)=x^2-y^2.

Explain how the saddle changes the picture.

ML Investigation — Local Loss Prediction

Choose a simple loss function $L(\theta)$ with a 2D parameter vector.

At a chosen point:

compute the exact gradient,
choose several small parameter steps $\Delta\theta$ ,
predict loss change with

\Delta L\approx\nabla L^T\Delta\theta,

compare with the true loss change,
repeat with larger steps.

Explain when and why the linear approximation breaks down.

Shape Check

For

L:\mathbb R^{20}\to\mathbb R,

fill in:

θ                    → shape ______
L(θ)                 → shape ______
∇L                   → shape ______
Δθ                   → shape ______
∇L · Δθ              → shape ______

For

W\in\mathbb R^{4\times7},

predict the shape of

\partial L/\partial W.

Mastery Gate

Do not continue until you can answer:

How are partial derivatives assembled into a gradient?
How does a gradient predict change in arbitrary directions?
Why does the dot product prove steepest ascent?
What does gradient magnitude mean?
Why is negative gradient only a local descent direction?
Why can stationary points be minima, maxima or saddles?
Why does gradient shape mirror parameter shape for scalar losses?
Why is an efficient chain-rule computation still needed for neural networks?