045 — Gradient Boosting

Where We Are

Random Forests reduce variance by training many varied trees and combining their votes. The trees mostly work in parallel.

Boosting asks a different question:

What if tree 2 studies tree 1’s mistakes, tree 3 studies the remaining mistakes, and the ensemble improves one small correction at a time?

1. Learn the residual

Suppose the true house prices are

y       = [100, 150, 200]
pred_0  = [150, 150, 150]
residual= [-50,   0,  50]

The first model is simply the mean, 150. A small tree is then trained to predict the residuals.

If it predicts [-40, 0, 40] and our learning rate is $\eta=0.5$ ,

$\hat y_1 = \hat y_0 + \eta h_1(x)$

pred_1 = [150,150,150] + 0.5*[-40,0,40]
       = [130,150,170]

The errors got smaller.

2. Why “gradient” is in the name

For squared error

$L=(y-\hat y)^2,$

$\frac{\partial L}{\partial \hat y}=2(\hat y-y).$

The negative gradient points toward a better prediction:

$-\frac{\partial L}{\partial \hat y}=2(y-\hat y).$

Up to a constant factor, that is exactly the residual. So fitting a tree to residuals is a special case of fitting the direction that reduces loss.

This turns boosting into a beautiful idea:

gradient descent, but the thing being updated is a function, not one scalar weight.

3. The additive model

After $M$ rounds,

$F_M(x)=F_0(x)+\eta\sum_{m=1}^{M}h_m(x).$

Each $h_m$ is usually a shallow tree. A single tree is weak; the sequence can be extremely strong.

4. Learning rate trades speed for caution

A large learning rate lets each tree make a big correction. Training improves quickly, but the ensemble can overfit.

A smaller learning rate usually needs more trees but makes gentler corrections.

large eta:   correction ━━━━━━━━━▶
small eta:   correction ━━━▶ ━━━▶ ━━━▶

5. From scratch mental model

pred = np.full_like(y, y.mean(), dtype=float)
for _ in range(M):
    residual = y - pred
    tree.fit(X, residual)
    pred += learning_rate * tree.predict(X)

Real implementations generalize this from residuals to gradients of arbitrary differentiable loss functions.

6. XGBoost, LightGBM and CatBoost

These systems are engineered descendants of gradient boosting. They add regularization, efficient split finding, missing-value handling, histogram methods and other optimizations. The central story remains: add a learner that improves the current ensemble.

Break It

Train too many deep trees with a large learning rate on a small dataset. Watch training loss keep falling while validation loss eventually rises. That is Chapter 56 waiting for us.

What to Remember

Random Forests mostly average independent trees.
Gradient boosting builds trees sequentially.
Each new learner follows a loss-reducing direction.
Learning rate controls the size of each correction.
Many weak corrections can form a very strong model.

Next → Support Vector Machines: boosting combines many flexible rules; can geometry instead find one boundary with the safest possible margin?

1 — Explain it

Contrast bagging and boosting without using either word “parallel” or “sequential” until your final sentence.

2 — Calculate it

Given y=[2,6], initial prediction [4,4], residual learner [-1.5,1.0], and $\eta=0.2$ , compute the new predictions.

3 — Predict it

What happens if learning rate is reduced from 0.3 to 0.03 while number of trees stays fixed?

4 — Break it

Fit overly deep boosted trees to a tiny noisy dataset. Plot train and validation loss.

5 — Build it

Implement squared-error gradient boosting using shallow regression trees as residual learners.

6 — Investigate it

Compare several (learning_rate, n_estimators) pairs over repeated train/validation splits. Explain the trade-off rather than reporting only the best score.