090 — Build a Tiny Autograd Engine

1. A value remembers how it was created

For a scalar teaching engine, each Value stores:

data       numeric result
grad       d(final_output)/d(this_value)
parents    values used to create it
_backward  local rule for passing gradient to parents

For

$c=a\times b,$

local derivatives are

$\frac{\partial c}{\partial a}=b,\qquad \frac{\partial c}{\partial b}=a.$

If upstream gradient is $g=\partial L/\partial c$ , then

a.grad += b.data * g
b.grad += a.data * g

The += is essential because one value may influence the loss through several paths.

2. Addition rule

For $c=a+b$ ,

$\frac{\partial c}{\partial a}=1,\qquad\frac{\partial c}{\partial b}=1.$

So the upstream gradient passes unchanged to both parents.

3. ReLU rule

For $r=\max(0,a)$ ,

a.grad += (a.data > 0) * r.grad

Now multiplication, addition and ReLU already let us build small neural expressions.

4. Why reverse topological order?

Consider

$L=(a\times b)+c.$

L depends on the multiply node, which depends on a and b. Before computing a node’s parent gradients, we must know the complete gradient arriving at that node from everything downstream.

So:

traverse the graph to create a topological order;
set L.grad = 1;
visit nodes in reverse topological order and call each local backward rule.

forward:   leaves ─────────→ loss
backward:  leaves ←───────── loss

5. Minimal skeleton

class Value:
    def __init__(self, data, parents=()):
        self.data = float(data)
        self.grad = 0.0
        self.parents = set(parents)
        self._backward = lambda: None

For multiplication:

def __mul__(self, other):
    out = Value(self.data * other.data, (self, other))
    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

6. Gradient checking

Never trust your engine merely because training loss falls. Compare analytical gradients with finite differences:

$\frac{dL}{dw}\approx\frac{L(w+\epsilon)-L(w-\epsilon)}{2\epsilon}.$

For sensible $\epsilon$ , the values should agree closely.

7. From scalar engine to tensor autograd

PyTorch generalizes this idea to tensor operations, broadcasting, saved intermediates, device kernels and memory management. But the conceptual engine is the same:

build a computation graph during the forward pass; apply local derivative rules backward with the chain rule.

What to Remember

Autograd is not a library spell. It is graph bookkeeping plus local derivatives plus reverse accumulation.

This closes Level 6: we can now explain what a neural network computes and how its gradients are obtained.

Next → Stochastic Gradient Descent: once gradients exist, how should we use noisy batches to move millions of parameters efficiently?