Regularization

1. Fit plus a penalty

Ordinary training minimizes data loss:

$L_{data}(\theta).$

Regularization adds a preference:

$L(\theta)=L_{data}(\theta)+\lambda R(\theta).$

The model may accept a slightly worse training fit to gain a simpler or more stable solution.

2. L2 regularization

$R(w)=\sum_j w_j^2.$

Large weights are expensive, so optimization prefers distributed, smaller coefficients when multiple solutions fit similarly.

For gradient descent,

$\frac{\partial}{\partial w_j}\lambda w_j^2=2\lambda w_j.$

The update therefore nudges weights toward zero. In neural networks, closely related weight decay is widely used.

3. L1 regularization

$R(w)=\sum_j |w_j|.$

L1 often drives some weights exactly to zero, producing sparse models. Geometrically, the shape of the constraint makes axis-aligned zero solutions easier to hit.

4. Lambda controls the bargain

λ = 0       fit data freely
small λ     mild simplicity preference
huge λ      penalty dominates → underfit

Regularization strength is a hyperparameter chosen using validation data—not the test set.

5. Regularization is broader than penalties

Early stopping, dropout, data augmentation and architectural constraints can all regularize learning by limiting how easily the model adapts to accidental detail.

What to Remember

Regularization changes the learning objective so that not all equally fitting solutions are equally desirable.

Next → Data Leakage: regularization cannot save an experiment if information from the future secretly enters training.