XOR: Why Linear Models Fail

XOR has four inputs:

x1 x2 | y
0  0  | 0
0  1  | 1
1  0  | 1
1  1  | 0

Plot them:

x2 ↑
1   ●(0,1)      ○(1,1)
0   ○(0,0)      ●(1,0)
    0-----------1 → x1

No single straight line can put both black points on one side and both white points on the other.

1. Why stacking linear layers does not help

Suppose

$h=W_1x+b_1,$

$y=W_2h+b_2.$

Substitute:

$y=W_2(W_1x+b_1)+b_2=(W_2W_1)x+(W_2b_1+b_2).$

That is still just one affine transformation.

Ten linear layers without nonlinear activations collapse into one linear layer.

2. ReLU changes the algebra

Now insert

$h=\operatorname{ReLU}(W_1x+b_1).$

We can no longer combine $W_2$ and $W_1$ into one matrix because ReLU changes behaviour depending on which region the input occupies.

A hidden layer can create two boundaries, isolate the middle XOR region, and let the output combine those hidden features.

3. The conceptual breakthrough

A hidden neuron need not mean “dog” or “cat.” It can learn an intermediate feature such as

“the point lies beyond this boundary.”

Another hidden neuron learns a different boundary. The output neuron recombines these new coordinates.

This is representation learning in miniature.

What to Remember

• XOR is not linearly separable.
• Repeating linear transformations without nonlinearity remains linear.
• Hidden nonlinear units create features that make the final task easier.

Next → Hidden Layers: how should we think about the intermediate representations that appear between raw input and final prediction?