Why Does a Machine Need a Loss Function

Where We Are

In Chapter 002, we separated the pieces of a learning system:

input → model parameters → prediction
                       ↓
                  compare with target
                       ↓
                      loss

That diagram hid the most important design decision in the entire loop.

We wrote a loss as though it were obvious.

It is not.

Suppose a model predicts ₹3 lakh for a house that sold for ₹7 lakh.

We can say the model is wrong by ₹4 lakh.

But that leaves several unanswered questions:

• Is being ₹4 lakh too low as bad as being ₹4 lakh too high?
• Is one ₹8 lakh mistake worse than two ₹4 lakh mistakes?
• Should a few extreme mistakes dominate training?
• How do we combine mistakes from many houses into one score?
• What properties must this score have if we want an optimizer to improve it?

A loss function is where we answer those questions.

Today: we derive why learning needs an objective, show why several obvious choices fail, and build the idea of a loss function from requirements rather than memorization.

Next: once we know why a loss is needed, we can compare absolute error and squared error geometrically and understand why “smoothness” matters to optimization.

1. The Problem: “Make the Model Better” Is Not an Instruction

Return to the house-price model:

Use the familiar dataset:

Suppose our current parameters are

Then predictions are

The model is clearly bad.

A human can look at the table and say:

“Increase the predictions.”

But a machine cannot optimize a sentence.

It needs a number.

Not four numbers.

Not “pretty good.”

Not “closer than yesterday.”

One number that can answer:

Is model A better than model B?

That number is the role of a loss or objective.

2. What Would a Useful Loss Need?

Before choosing a formula, reason from requirements.

For our house problem, a useful loss should ideally:

1. be small when predictions are close to targets,
2. be zero when predictions are perfect,
3. not allow positive and negative mistakes to cancel,
4. give a single score for a dataset,
5. react sensibly when mistakes become larger,
6. be useful for deciding how parameters should move.

Requirement 6 is easy to overlook.

A metric can be excellent for reporting and terrible for optimization.

We will see why later.

3. First Attempt: Count Correct Predictions

A natural idea is accuracy.

For each house, call the prediction correct if it exactly equals the price.

With current predictions

and targets

accuracy is

Now improve the model dramatically:

Every prediction is only ₹0.1 lakh away.

Exact accuracy is still

The score says nothing improved.

That violates requirement 5.

A Tempting Wrong Idea — “Just optimize the final evaluation metric.”

Some metrics throw away too much information to guide learning. Exact accuracy does not tell us whether a wrong prediction is barely wrong or catastrophically wrong.

This is a recurring pattern in machine learning:

The best metric for reporting performance is not always the best function for training.

4. Second Attempt: Use the Signed Error

For one example define

For four houses, suppose predictions are all ₹6 lakh:

Average error:

The model is wrong on every example.

Yet the average says perfect.

Why?

Because signed errors describe direction as well as size.

The +3 and -3 cancel.

The +1 and -1 cancel.

So signed error is useful diagnostically, but not as our dataset objective.

5. The Core Discovery: A Loss Is a Rule for Valuing Mistakes

We need a transformation that converts each mistake into a cost.

Symbolically:

This sentence matters more than any particular formula.

A loss is not merely “error with decoration.”

It encodes what kinds of mistakes we care about.

For house prices we might choose:

• absolute error,
• squared error,
• Huber loss,
• asymmetric losses if overpricing and underpricing have different consequences.

For classification we may use cross-entropy.

For ranking we may use pairwise losses.

For language modeling we often use negative log-likelihood / cross-entropy.

Different tasks, different notion of “bad.”

6. Absolute Error: The Simplest Honest Cost

The easiest way to stop sign cancellation is to remove the sign:

For a prediction of 3 when truth is 7:

For a prediction of 11 when truth is 7:

Equal distance, equal cost.

This has a clean interpretation:

How far away was the prediction?

For the constant ₹6 lakh model:

are the four absolute errors.

Mean absolute error:

Now the model can no longer hide behind cancellation.

7. Squared Error: A Different Value System

Another way to remove the sign is to square:

For errors

squared errors are

Mean squared error:

Notice what changed.

Absolute error says:

error 1 → cost 1
error 2 → cost 2
error 3 → cost 3

Squared error says:

error 1 → cost 1
error 2 → cost 4
error 3 → cost 9

Large mistakes become disproportionately expensive.

That is not automatically good or bad.

It is a modeling choice.

8. A Thought Experiment: One Big Mistake or Four Small Ones?

Consider two models.

Model A

Four errors:

Model B

Four errors:

Both have total absolute error

So both have the same MAE:

But squared error differs.

Model A:

Model B:

Squared error strongly prefers spreading mistakes rather than allowing one catastrophic miss.

Intuition — MAE charges linearly. MSE adds a “large-error penalty.”

This is why choosing a loss is partly choosing what failure pattern you consider unacceptable.

9. Dataset Loss: Many Complaints Become One Verdict

Training uses many examples.

So after defining a per-example loss

we need to aggregate them.

The most common choice is the mean:

Here $J$ is the dataset objective.

For squared error:

This equation is a milestone.

Now loss is no longer merely a score attached to predictions.

It is a function of the parameters.

That is exactly what optimization needs.

10. Parameters Turn Loss Into a Landscape

Take the house data and hold $b=1$ fixed.

Vary only $w$.

For each $w$:

1. compute predictions,
2. compute errors,
3. compute squared losses,
4. average them.

You get something like:

Plot $w$ horizontally and loss vertically:

loss
 ↑
30| *
  |   *
  |      *
  |         \       /
  |          \     /
  |           \   /
  |            \_/
  +--------------------→ w
               2

The exact picture is a parabola.

Now “learning” has become geometric:

move through parameter space toward lower loss.

11. Why One Number Matters

Imagine comparing two parameter settings.

Model A:

Model B:

Without an objective, “better” is vague.

With loss:

Model A → loss 0.675
Model B → loss 0.075

Now comparison is mechanical.

This does not mean the chosen loss perfectly captures all real-world value.

It means we created an optimization target.

That distinction matters:

Loss makes learning mathematically possible; it does not guarantee that the learned behavior matches every human goal.

12. Loss Versus Metric

These are often confused.

A loss is usually chosen to guide optimization.

A metric is usually chosen to evaluate performance in a human-relevant way.

They can be the same, but need not be.

Examples:

Why not always optimize the metric directly?

Because some metrics are flat, discontinuous, non-differentiable, or otherwise difficult for gradient-based optimization.

This is one reason surrogate losses exist.

13. History Lens — Least Squares and Gauss

History Lens

In the late 1700s and early 1800s, astronomers faced a practical problem: repeated observations of the same object disagreed because measurement was noisy.

They needed a principled way to choose the “best” orbit or position from imperfect measurements.

The method of least squares emerged from this world. Adrien-Marie Legendre published the method in 1805; Carl Friedrich Gauss later argued that he had used it earlier in astronomical calculations and developed probabilistic justifications.

The central idea was exactly our modern optimization pattern: choose parameters that minimize the sum of squared residuals.

Deep learning did not invent the notion of a loss landscape. It inherited a centuries-old optimization idea and scaled it dramatically.

The interesting historical lesson is not a date.

It is the problem:

When observations disagree, define what “best fit” means numerically.

That is still what a loss function does.

14. A Tempting Wrong Idea: “Lower Training Loss Means a Better Model”

Suppose model A memorizes every training example.

Training loss:

Model B has small but nonzero training loss.

Which is better?

We cannot know yet.

Why?

Because learning is not about reproducing the past perfectly.

It is about performing well on unseen examples.

This is the seed of generalization.

So loss is necessary, but not sufficient.

Later we will ask:

• training loss versus validation loss,
• overfitting,
• regularization,
• bias-variance tradeoff,
• generalization bounds.

For now remember:

Loss tells us how well the model satisfies the chosen objective on the examples we measure. It does not automatically tell us whether the model learned the right thing.

15. Loss Can Encode Asymmetry

So far, predicting ₹4 lakh too high and ₹4 lakh too low had equal cost.

But suppose a medical application has asymmetric consequences.

Missing a dangerous condition may be much worse than triggering an unnecessary follow-up test.

Then we might want a loss that penalizes the two mistakes differently.

Or in inventory:

• understocking may lose customers,
• overstocking may create waste.

A loss can encode that tradeoff.

This is another reason not to think of loss as “the obvious formula.”

It is an explicit statement of priorities.

16. Shape of the Loss Matters

Consider one error value $e$.

Absolute loss:

Squared loss:

Their graphs look different.

Absolute:

loss
 ↑
 |      /
 |     /
 |    /
 |   /
 |  /
 | /\
 |/  \
 +--------→ error

Squared:

loss
 ↑
 |       /
 |     /
 |   _/
 | _/
 |/
 +--------→ error

The first has a corner at zero.

The second has a smooth bottom.

At this point, do not worry about the formal definition of smoothness.

Just notice:

• absolute loss changes direction abruptly,
• squared loss bends continuously.

Why should an optimizer care?

That is exactly the next chapter.

17. Small Numerical Comparison

Take target

Evaluate several predictions.

Three observations:

1. both are minimized at the truth,
2. both are symmetric,
3. squared loss grows much faster far from the truth.

Those three facts are visible before calculus.

18. Loss Functions as Surrogates

Many real goals are hard to optimize directly.

Suppose our real goal is:

“Make as few wrong classifications as possible.”

That sounds like accuracy.

But accuracy changes in jumps.

A probability can move from 0.51 to 0.99 and accuracy remains 1 either way.

Cross-entropy, by contrast, rewards increasing confidence in the correct answer and strongly penalizes confident wrong answers.

So we optimize a smooth surrogate that correlates with the real goal.

This idea appears everywhere in ML:

real-world goal
      ↓
choose measurable proxy
      ↓
construct optimization loss
      ↓
train parameters
      ↓
evaluate with real-world metrics

That separation is one of the most important engineering habits in machine learning.

19. One-Minute Explanation

A model cannot improve from the instruction “be less wrong.”

It needs a numerical objective.

A loss function converts a prediction and a target into a cost.

Different loss functions value mistakes differently.

Absolute error treats error size linearly.

Squared error punishes large mistakes more strongly.

Across a dataset we aggregate per-example losses into one objective, which becomes a function of the model parameters.

Training then becomes the problem of finding parameters that reduce that objective.

20. Distinctions That Matter

21. What Each Symbol Means

22. Common Mistakes

23. What We Discovered

1. “Make the model better” must be turned into a numerical objective.
2. Signed errors can cancel and therefore fail as a dataset objective.
3. A loss function assigns a cost to prediction mistakes.
4. Different losses express different preferences about mistakes.
5. Aggregating per-example losses creates a dataset objective.
6. Once the objective is written in terms of parameters, learning becomes optimization.
7. Training loss and evaluation metrics need not be the same.
8. Low training loss does not guarantee generalization.
9. Loss functions are often surrogates for harder real-world goals.

24. Mathematics We Built

Raw error:

Absolute loss:

Squared loss:

Mean absolute error:

Mean squared error:

Parameter-dependent objective:

25. Socratic Questions

1. Can two models have the same MAE but different MSE?
2. Why might that happen?
3. Why can accuracy fail to distinguish a barely wrong model from a wildly wrong model?
4. Why is an optimization objective a design decision?
5. When might large errors deserve extra punishment?
6. When might we not want outliers to dominate?
7. Why is zero training loss not sufficient evidence of learning?
8. Why might a metric be difficult to optimize directly?
9. What does it mean to call a loss a surrogate?
10. What would happen if every loss value were constant regardless of the parameters?

🔭 Bridge to Chapter 004 — Why Squared Error Is Smooth, and Why Optimization Cares

We have two reasonable losses:

and

Both are zero at the correct answer.

Both grow as the prediction moves away.

But their shapes differ at exactly the point we care about most: the minimum.

Absolute error has a sharp corner.

Squared error has a smooth bowl.

Why should the shape of the loss curve affect whether a machine can learn efficiently?

That question takes us directly to slopes, local direction and optimization.