Cross-Entropy

1. The Classification Problem

A 3-class model outputs probabilities

$Q=[0.7,0.2,0.1].$

If the true class is class 1, target distribution is

$P=[1,0,0].$

We need a loss that strongly rewards assigning probability to what actually happened and strongly punishes assigning it tiny probability.

Cross-entropy is

For one-hot $P$:

The entire loss becomes the negative log probability assigned to the correct class.

2. Why the Log?

If the model gives the true class probability:

0.9   → small loss
0.5   → moderate loss
0.01  → huge loss

Because

$-\log q$

rises sharply as $q\to0$.

The logarithm also turns products of probabilities into sums, which connects classification loss to maximum likelihood.

3. Cross-Entropy = Negative Log-Likelihood

For labeled examples $(x_i,y_i)$, a classifier models

$P_\theta(y_i\mid x_i).$

Likelihood of independent observations:

Log likelihood:

Maximizing it equals minimizing

which is categorical cross-entropy.

So cross-entropy is not an arbitrary neural-network convention. It is maximum likelihood for categorical outcomes.

4. Softmax + Cross-Entropy

Logits $z_i$ are unrestricted real numbers. Softmax converts them to probabilities:

For true class $k$:

A beautiful derivative appears:

The gradient is simply predicted probability minus target probability.

This clean gradient is one reason softmax and cross-entropy fit together so naturally.

5. Numerical Stability

Never implement large-logit softmax naively. $e^{1000}$ overflows.

Subtract the maximum logit:

Libraries combine log_softmax and negative log-likelihood using the log-sum-exp trick.

In PyTorch:

import torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, 1.0, -1.0]])
target = torch.tensor([0])
loss = F.cross_entropy(logits, target)

Pass logits, not already-softmaxed probabilities, to F.cross_entropy.

6. Binary Cross-Entropy

For binary target $y\in\{0,1\}$ and predicted probability $p$:

With a sigmoid logit, PyTorch’s numerically stable choice is binary_cross_entropy_with_logits.

7. Cross-Entropy vs Squared Error

Squared error can be used on probabilities, but it corresponds to a different statistical story and often gives less convenient classification gradients.

Cross-entropy directly asks:

How much probability did you assign to the observed class?

and follows from categorical maximum likelihood.

Choose losses from the probabilistic problem, not from habit.

8. Shape Check

Multiclass classification:

logits: [B, C]
target: [B]      # integer class indices
loss:   []       # default mean reduction

For $B=32$, $C=10$:

[32,10] → cross_entropy(target [32]) → scalar

9. Break It

Softmax twice

Applying softmax before F.cross_entropy changes the computation and harms numerical behavior.

Wrong axis

Softmax must normalize across classes for each example, not across the batch.

log(0)

Naive probability implementations can produce infinities. Work in logits/log-space when possible.

Confidence ≠ calibration

Low cross-entropy encourages probability on correct labels, but a trained model can still be miscalibrated.

10. Real-World Connection

Cross-entropy trains image classifiers, token predictors, language models and many discriminative systems. Next-token language-model loss is simply cross-entropy repeated across sequence positions.

For logits [B,T,V] and token targets [B,T], the model is solving a $V$-class classification problem at every token position.

Mastery Check

1. Derive one-hot cross-entropy from the general formula.
2. Explain its connection to maximum likelihood.
3. Why does tiny true-class probability receive huge loss?
4. What is the gradient of softmax-cross-entropy with respect to logits?
5. Why should PyTorch cross-entropy receive logits?
6. How is next-token prediction a classification problem?

Level 3 Complete

We started with uncertainty and ended with the loss used to train modern classifiers and language models:

probability
→ conditional probability
→ Bayes
→ random variables
→ expectation
→ covariance
→ Gaussian
→ likelihood
→ MAP
→ entropy
→ KL divergence
→ cross-entropy

Next → Level 4: Classical Machine Learning. We now have enough mathematics to derive learning algorithms rather than merely call them.