Why Squared Error Is Smooth - and Why Optimization Cares

Where We Are

Chapter 003 gave us two reasonable ways to punish prediction error:

and

Both are zero when the prediction is perfect.

Both increase as the prediction moves away from the truth.

Both prevent positive and negative mistakes from cancelling.

So why do introductory optimization examples so often reach for squared error?

The answer is not “because absolute error is bad.”

Absolute error is useful and widely used.

The deeper answer is about shape.

One curve has a corner.

One curve bends continuously.

And when learning depends on local slopes, that geometric difference matters.

Today: we build the meaning of “smooth” from pictures, tiny numbers and slopes. We will not assume calculus first.

Next: once we understand slopes, we can formalize them as functions and derivatives.

1. Start With One Prediction Dial

Forget the full model for a moment.

Suppose the true house price is

Imagine a dial that directly controls our prediction $\hat{y}$.

We want to turn the dial until

Define error

Now compare two possible losses:

and

This is the simplest possible setting in which “smooth versus sharp” becomes visible.

2. See the Two Shapes Before Talking About Slopes

Absolute error:

loss
 ↑
 6|\          /
 5| \        /
 4|  \      /
 3|   \    /
 2|    \  /
 1|     \/
 0|------●----------→ prediction
         7

Squared error:

loss
 ↑
36|\              /
25| \            /
16|  \          /
 9|   \        /
 4|    \      /
 1|      \__/
 0|-------●----------→ prediction
          7

The drawings are crude, but one feature is unmistakable.

Absolute error forms a V.

Squared error forms a U.

The V has a corner at the minimum.

The U has a rounded bottom.

That rounded bottom is what we mean informally by smooth here.

3. What Does “Sharp” Actually Mean?

The word “sharp” can sound vague, so make it numerical.

For absolute error, look just to the left of the correct prediction.

Take

Then

Move right by $0.01$:

Loss changes:

So moving right makes loss fall.

Now stand just to the right:

Move right again:

Loss changes:

Now moving right makes loss rise.

So immediately around 7:

left side:  move right → loss decreases
right side: move right → loss increases

At exactly 7, the preferred direction flips instantly.

That instant flip is the corner.

4. The Idea of Slope Without Calculus

You already know slope from school geometry:

For a loss curve:

Take absolute error to the left of the minimum.

From prediction 5 to 6:

So

From 6 to 7:

On the left side, slope is always

Now on the right:

From 7 to 8:

From 8 to 9:

So absolute loss looks like:

left of 7   slope = -1
right of 7  slope = +1

But what is the slope at 7?

That is the problem.

5. Why the Slope at the Corner Is Undefined

Approach the point from the left.

The slope tends to

Approach from the right.

The slope tends to

A single point cannot have both slopes as one ordinary derivative.

So the derivative at the corner does not exist in the usual sense.

Intuition — Imagine driving on a road. On a smooth bend, your steering wheel turns continuously. At a perfect V-shaped corner, the road direction jumps instantly. There is no single tangent direction exactly at the corner.

This is what “undefined slope at zero” means for absolute error.

It does not mean the loss is useless.

It means ordinary smooth calculus needs special handling there.

6. Squared Error Changes Direction Gradually

Now repeat with

Calculate a few values:

Notice how the curve flattens as we approach the minimum.

Far away, the loss changes rapidly.

Near the truth, it changes gently.

At the exact minimum, the slope becomes zero.

No jump is required.

7. Approximate the Squared-Loss Slope Numerically

Use a tiny step $h=0.01$.

Approximate slope with

At $\hat{y}=5$:

and

So

Roughly

At $\hat{y}=6$:

so slope is approximately

At $\hat{y}=6.9$:

so

The slope is approaching zero smoothly.

8. Discover the Pattern

Look at error and slope:

The pattern is

Since

we get

We have just discovered the derivative pattern from numbers before formally deriving it.

9. Why a Slope Helps Learning

Suppose the prediction is

while target is

Squared-loss slope:

A negative slope means:

moving the prediction slightly right will reduce the loss.

Why?

Because if the curve slopes downward as x increases, going right goes downhill.

Now suppose

Slope:

Positive slope means:

moving the prediction left reduces the loss.

So the slope encodes direction.

This is why derivatives become a compass for learning.

10. The Opposite-Slope Rule

If slope is positive, move left.

If slope is negative, move right.

In symbols:

The minus sign is not magic.

It simply says:

move opposite the uphill direction.

For prediction 5, slope is -4.

With learning rate

we get

Loss before:

Loss after:

It fell.

11. Why Smoothness Is Useful

A smooth loss gives us a slope that changes continuously as the prediction moves.

That provides two useful signals:

1. direction — which way is downhill,
2. strength — how steeply loss changes.

For squared loss:

far from target → large slope magnitude
near target     → small slope magnitude
at target       → slope 0

This naturally produces large corrections when far away and gentler corrections near the minimum, assuming a sensible learning rate.

That is a convenient optimization geometry.

12. But Absolute Error Still Works

Important precision:

A non-smooth point does not make a loss unusable.

Absolute error is used in real machine learning.

For

the slope away from zero is simple:

and

At zero, we can use ideas such as a subgradient.

A valid subgradient there can be any value between -1 and +1; choosing 0 is common.

You do not need subgradient theory yet.

The key correction is:

absolute error has a corner that ordinary differentiation cannot assign one unique derivative to, but optimization has mathematical tools for handling such cases.

This is much more accurate than saying “absolute error cannot be optimized.”

13. A More Important Difference: Gradient Magnitude

There is another practical difference between MAE and MSE.

For absolute loss:

So an error of 100 and an error of 0.01 produce the same gradient magnitude:

For squared loss:

So:

error 100  → gradient magnitude 200
error 1    → gradient magnitude 2
error 0.01 → gradient magnitude 0.02

Squared loss reacts much more strongly to large errors.

Again: choice, not universal superiority.

14. Outliers Reveal the Tradeoff

Suppose five errors are

MAE:

MSE:

The single error 20 dominates MSE.

That can be useful if catastrophic mistakes must be fixed aggressively.

It can be harmful if the value 20 is merely a corrupted measurement.

So smoothness is not the only consideration.

Robustness to outliers matters too.

15. History Lens — Fermat, Newton and Leibniz

History Lens

Long before machine learning, mathematicians wanted to solve problems like: where is a curve highest, where is it lowest, and how fast is it changing?

Pierre de Fermat developed methods for maxima and minima in the 1600s. Later, Isaac Newton and Gottfried Wilhelm Leibniz independently developed systematic calculus notation and methods for rates of change.

The machine-learning use is a modern continuation of the same question:

Given a function, how do we find where it becomes smaller?

Deep learning applies this idea not to one or two variables, but sometimes to billions of parameters.

The scale changed dramatically.

The core mathematical question did not.

16. From Prediction Dial to Weight Dial

So far we changed $\hat{y}$ directly.

But real models do not usually store the prediction as a parameter.

They store weights.

Recall

Take

Then

Squared loss becomes

Now loss is a function of the weight.

Try values:

The same bowl-shaped idea reappears, but now the horizontal axis is parameter space.

That is the loss landscape.

17. Chain of Influence

Notice the dependency:

w
↓
prediction ŷ = wx + b
↓
error e = ŷ - y
↓
loss L = e²

If we change $w$, prediction changes.

If prediction changes, error changes.

If error changes, loss changes.

This chain is exactly why the chain rule will later become essential.

For now, the important mental model is causality:

parameters affect loss through intermediate computations.

Backpropagation will eventually walk backward through these dependencies.

18. A Tempting Wrong Idea: “Smooth Means Flat”

Smooth does not mean flat.

A steep curve can be perfectly smooth.

For example:

at error 100 has slope 200 — extremely steep.

Smooth means, roughly:

the curve has no abrupt break in its local direction at the point we are considering.

Another tempting confusion:

“If the derivative is zero, learning always succeeded.”

Not necessarily.

A derivative can be zero at:

• a minimum,
• a maximum,
• a saddle point,
• or a flat plateau.

One-dimensional squared error is friendly, but deep-network landscapes are far more complicated.

19. Visualizing Tangents

A derivative is the slope of the tangent line.

At a smooth point on squared loss, imagine zooming in farther and farther.

The curve starts to look locally like a straight line.

That straight line has one slope.

At the absolute-value corner, zooming in never removes the V shape.

From the left, the local line points one way.

From the right, it points another.

That is the geometric heart of differentiability.

20. Numerical Derivative From First Principles

Before learning symbolic differentiation, we can estimate slope using a tiny change:

Take

with

Then

and

So

Close to

And

This numerical experiment previews the exact derivative.

21. Why Tiny $h$ Is Not Infinitely Tiny in Code

A subtle practical point:

Mathematics studies the limit as

Computers use finite-precision floating-point numbers.

If $h$ becomes absurdly tiny, subtracting two nearly equal values can lose numerical precision.

So finite-difference gradient checks often use something like

not an unimaginably small number.

This distinction between exact mathematics and numerical computation will matter repeatedly in deep learning.

22. One-Minute Explanation

Absolute error makes a V-shaped loss curve.

Its slope is -1 on the left and +1 on the right, so exactly at the bottom there is no single ordinary derivative.

Squared error makes a smooth U-shaped curve.

Its slope changes gradually, becoming zero at the minimum.

A slope tells us which direction increases the loss, so moving in the opposite direction can reduce it.

Smoothness therefore makes gradient-based optimization especially convenient — but non-smooth losses can still be optimized with appropriate methods.

23. Distinctions That Matter

24. Common Mistakes

25. What We Discovered

1. A sharp corner means the local direction changes abruptly.
2. Absolute error has slope -1 on one side and +1 on the other.
3. Therefore ordinary differentiation has no unique derivative at the minimum of $|e|$.
4. Squared error has a continuously changing slope.
5. Its slope is proportional to error: $2e$.
6. The slope sign tells us the local downhill direction.
7. The slope magnitude tells us local sensitivity.
8. Smoothness makes gradient-based optimization convenient, not mandatory.
9. MSE’s growing gradient makes it sensitive to large errors and outliers.
10. Parameter changes affect loss through a chain of intermediate computations.

26. Mathematics We Built

Absolute loss:

Squared loss:

Finite-difference slope:

Squared-loss derivative pattern:

Prediction error:

Squared-loss slope with respect to prediction:

Basic downhill update idea:

27. Socratic Questions

1. Why is the left-hand slope of $|e|$ equal to -1?
2. Why is the right-hand slope +1?
3. Why does that make the ordinary derivative at zero undefined?
4. Why does $e^2$ flatten as $e$ approaches zero?
5. If squared-loss slope is negative, which direction should we move the variable?
6. Why does a large error create a larger MSE gradient?
7. Why might that be useful?
8. Why might it be harmful?
9. What is the difference between a derivative and a finite-difference estimate?
10. Why will the chain rule become necessary once parameters influence loss through several operations?

Exercises

See exercises.md for calculations, graph-reading, finite-difference experiments, failure cases and design questions.

🔭 Bridge to Chapter 005 — Numbers Become Vectors

We now understand one adjustable quantity and one local slope.

But real data rarely has one feature.

A house has rooms, area, age, location and many other measurements.

An image has thousands or millions of pixel values.

A word representation may contain hundreds or thousands of coordinates.

How can many numbers travel together as one mathematical object without losing their individual meaning?

That problem forces us into vectors.