081 — Hidden Layers

A hidden layer computes a new representation:

$h=f(Wx+b).$

The vector $h$ becomes the next layer’s input.

1. Think “new coordinate system”

Raw coordinates may make a problem difficult. A learned layer transforms them so useful distinctions become easier.

raw input x
   ↓ W₁x+b₁
learned features h₁
   ↓ W₂h₁+b₂
better features h₂
   ↓
prediction

In XOR, hidden neurons can represent regions separated by different lines. In images, early CNN features may respond to edges; later representations can combine them into textures, parts and objects.

2. Why “hidden”?

We directly observe input and target, but nobody supplies the correct values of intermediate neurons. Training discovers them indirectly because they help reduce the final loss.

3. Width and depth

Width = number of neurons in a layer. Depth = number of successive learned transformations.

Width gives a layer more features at one stage. Depth lets the model compose features repeatedly.

4. Shape Check

For

X  [32,100]
W1 [100,64]
W2 [64,10]

we get

H  [32,64]
Y  [32,10]

The inner dimensions explain the wiring.

5. Hidden representations are learned, not named

Do not assume neuron 17 “is ears.” Representations are often distributed: many neurons jointly encode a concept, and one neuron may participate in several concepts.

What to Remember

Hidden layers learn intermediate features that reshape the problem. Deep learning’s power comes from composition of learned representations.

Next → Neural Networks as Function Approximators: how far can repeated linear transformations plus nonlinearities take us?