077 — Weights and Biases

Where We Are

The smallest neural network introduced the neuron. Now we open it.

A neuron first computes

$z=w_1x_1+w_2x_2+\cdots+w_dx_d+b=w^Tx+b.$

The weights decide how strongly each input influences the neuron. The bias lets the neuron’s response shift even when every input is zero.

1. A concrete example

Suppose a toy animal detector receives

x = [ear_score, nose_score, fur_score]
  = [1.0, 0.5, 2.0]

and

w = [2.0, -1.0, 0.5]
b = -0.25

Then

$z=2(1)-1(0.5)+0.5(2)-0.25=2.25.$

The positive first weight rewards ear evidence. The negative second weight pushes the score down. The bias moves the decision threshold.

2. Why bias is not optional decoration

Without bias, the boundary $w^Tx=0$ must pass through the origin. With bias,

$w^Tx+b=0$

can slide through space. In one dimension, $wx+b=0$ gives threshold $x=-b/w$ .

3. Layers are many neurons at once

For a batch,

$Z=XW+b.$

With X:[B,D], W:[D,H], b:[H]:

[B,D] @ [D,H] + [H] → [B,H]

Bias broadcasts across the batch.

Shape Check

If 32 examples each have 100 features and the layer has 64 neurons:

X [32,100]
W [100,64]
b [64]
Z [32,64]

4. Learning means changing these dials

Backpropagation will eventually compute how each weight and bias affects loss. An optimizer then changes them. The architecture defines the wiring; the parameters store what the network learned.

What to Remember

Weight = learned strength/direction of a connection.
Bias = learned offset/threshold shift.
A dense layer is matrix multiplication plus broadcasting.
Learning changes parameters to reduce loss.

Next → Why a Neuron Needs an Activation Function: if layers only repeat XW+b, does depth actually buy us anything?