079 — ReLU and Geometry

1. The formula is simple

$\operatorname{ReLU}(z)=\max(0,z).$

For positive input the slope is 1. For negative input the output is 0.

output
  ↑      /
  |     /
  |____/______→ z
      0

But its importance is geometric.

2. A neuron creates a boundary

Before ReLU, a neuron computes

$z=w^Tx+b.$

The equation $w^Tx+b=0$ is a line in 2-D, a plane in 3-D, and a hyperplane in higher dimensions. ReLU treats the two sides differently:

wᵀx+b < 0  → 0
wᵀx+b > 0  → wᵀx+b

So a ReLU neuron folds/clips space along a learned boundary.

3. Many ReLUs create regions

Two ReLU neurons can introduce two boundaries. A network then behaves like different linear functions in different regions.

That is why a ReLU network is called piecewise linear: locally linear, globally capable of complex bends.

4. Why gradients work well

For $z>0$ ,

$\frac{d\operatorname{ReLU}}{dz}=1.$

For $z<0$ , the derivative is 0. At exactly zero the classical derivative is undefined; frameworks choose a convenient subgradient convention, commonly 0.

Positive activations therefore pass gradients without the strong saturation of sigmoid/tanh.

5. Dead ReLUs

If a neuron stays negative for every example, its ReLU output and local gradient remain zero. It may stop learning: a dead ReLU. Initialization, learning rate and variants such as Leaky ReLU can reduce this problem.

Shape Check

ReLU changes values, not shape:

[B,H] → ReLU → [B,H]

What to Remember

ReLU is not merely a threshold formula. It lets learned hyperplanes divide space into regions and gives deep networks a way to bend an otherwise linear transformation.

Next → XOR: can these bends solve the smallest famous problem that one straight boundary cannot?