TensorTau PathToAGI Training Deep Networks Level 07

Level 07 · Training Deep Networks · Chapter 101

Dropout

Can randomly hiding parts of a network during training make the full network more robust at inference?

During training, dropout independently masks activations. With keep probability q=1−pq=1-p:

mj∼Bernoulli(q).m_j\sim Bernoulli(q).

In inverted dropout,

h~j=mjqhj.\tilde h_j=\frac{m_j}{q}h_j.

The 1/q1/q factor keeps the expected activation approximately unchanged.

1. Why it can regularize

A feature cannot rely on the same neighbouring features always being present. The network is pushed toward more redundant, robust representations.

Another intuition is that training samples many thinned subnetworks while inference uses the combined full network.

2. Train versus eval

Dropout is active during training and disabled during evaluation:

model.train()  # dropout on
model.eval()   # dropout off

Forgetting this can make validation predictions randomly fluctuate.

3. More is not always better

Excessive dropout can destroy useful signal and cause underfitting. Its usefulness depends on architecture, data scale and other regularization.

What to Remember

Dropout injects multiplicative Bernoulli noise during training to discourage fragile feature dependence.

Next → Residual Connections: instead of randomly removing paths, can we create direct identity paths that make very deep networks easier to optimize?

Six ways to work the chapter, climbing from explaining it in your own words to investigating it like a researcher. Reading is not enough — this is where the chapter lands.

  1. Explain: Why does inverted dropout divide by keep probability?
  2. Calculate: Apply a supplied dropout mask with p=0.5.
  3. Predict: What happens if dropout remains active during inference?
  4. Break: Sweep dropout from 0 to 0.9.
  5. Build: Implement inverted dropout forward pass.
  6. Investigate: Compare train/validation gaps across dropout strengths and dataset sizes.