Computational Graphs: Turning Formulas Into Data Flow

Where We Are

Chapter 021 gave us the chain rule.

If

then

That solves the mathematics of composition.

But a neural network is not a neat two-function composition.

It may contain millions of operations:

input
 → matrix multiply
 → bias add
 → activation
 → matrix multiply
 → activation
 → ...
 → prediction
 → loss

Writing one giant symbolic formula becomes impossible to reason about.

We need a representation that remembers what was computed from what.

That representation is a computational graph.

1. The Problem: One Formula Can Hide Many Operations

Consider

It looks like one expression.

But a computer cannot perform “the whole formula” in one magical step.

It performs small operations:

1. multiply $w$ and $x$,
2. add $b$,
3. subtract $y$,
4. square the result.

Name the intermediate values:

Now the hidden structure is visible.

2. Draw the Forward Graph

w ──┐
    × ── a ──┐
x ──┘        + ── z ──┐
             b        - ── e ── square ── L
                      y

Every node is either:

• an input/value,
• an operation,
• an intermediate result.

Every arrow means:

this value is used to compute that value.

The graph is a map of dependency.

3. Why Name Intermediate Values?

Suppose

Forward pass:

The graph lets us store these values:

a = 6
z = 7
e = -3
L = 9

Later the backward pass will reuse them.

This is exactly what autodiff systems do internally.

4. Local Derivatives

Each operation knows only its own derivative.

For multiplication

local derivatives are

For addition

For subtraction

For squaring

No node needs to understand the entire network.

Each node only knows its own small rule.

5. Backward Pass as Chain Rule on the Graph

Start from the loss.

A quantity always changes one-for-one with itself:

Now move backward.

At the square node:

With

At subtraction:

At addition:

Finally through multiplication:

Since $x=3$:

Exactly as expected.

6. The “Upstream × Local” Rule

At every node, the pattern is the same:

For example, at

if upstream gradient is

then

This one pattern becomes the operational heart of backpropagation.

7. A Tempting Wrong Idea: Differentiate the Giant Formula Every Time

We could expand

and differentiate symbolically from scratch.

For a tiny expression, that is fine.

For a 100-layer network, it is disastrous.

Why?

Because the same intermediate calculations are reused throughout the network.

A computational graph exposes reuse and lets us apply small derivative rules locally.

A Tempting Wrong Idea

“Backpropagation means finding one enormous derivative formula for the whole network.”

No. Backpropagation is efficient because it avoids doing that. It reuses local derivatives and previously computed upstream gradients.

8. Branches: One Value Can Affect the Loss in Multiple Ways

Consider

The input x branches into two paths:

       ┌─ square ─┐
x ─────┤          + ── f
       └─ cube ───┘

Derivative through first branch:

Through second:

Total derivative:

When a value influences the output through multiple paths, contributions add.

This is crucial in real networks with branching, residual connections and parameter reuse.

9. Why Gradient Contributions Add at Branches

Suppose

Then

If x affects loss through multiple downstream paths, total sensitivity is the sum of all path sensitivities.

This gives the graph rule:

multiply along a path; add across alternative paths.

That sentence is worth remembering.

10. Example With a Branch

Let

At $x=2$:

Forward:

Backward:

Path through a:

Path through b:

Add:

Direct derivative confirms:

11. Topological Order

A graph has dependencies.

You cannot compute a node before its inputs exist.

So forward evaluation must respect dependency order.

For

x → a → c
 \→ b ↗

we must compute:

1. x,
2. a and b,
3. c.

This is a topological ordering of a directed acyclic graph (DAG).

Backward propagation uses the reverse topological order.

That is not accidental.

To know a node’s total influence on loss, we must first know the gradients arriving from everything downstream.

12. Why Neural Computation Is Usually a DAG

During one ordinary feed-forward evaluation, values flow forward without cycles:

inputs → hidden values → outputs → loss

That creates a directed acyclic graph.

Recurrent networks may represent repeated-time computation, but when unrolled across a finite sequence they also form a DAG for differentiation.

Autodiff engines exploit this structure.

13. Computational Graph vs Neural-Network Architecture Diagram

These are related but not identical.

A high-level architecture diagram may show:

Linear → ReLU → Linear

A computational graph can be much more detailed:

x
↓
matmul W1
↓
add b1
↓
ReLU
↓
matmul W2
↓
add b2
↓
loss

And underneath matrix multiplication itself are many scalar multiplications and additions.

The level of detail depends on what derivative rules the system treats as primitive.

14. Scalar Graph vs Tensor Graph

For teaching, scalar graphs are easiest.

Real frameworks operate on tensors.

A node may represent

rather than one scalar multiply.

Its backward rule returns tensor gradients:

The conceptual graph is identical:

X ─┐
W ─┼─ Linear ─ Z
b ─┘

Only the local derivative rule becomes matrix calculus instead of scalar calculus.

15. Shape Check on the Graph

Suppose

Then

has shape

If loss L is scalar, backward gradients must align with the quantities they belong to:

dL/dZ  → (B,h)
dL/dW  → (d,h)
dL/db  → (h,)
dL/dX  → (B,d)

Computational graphs are also shape-flow graphs.

16. Forward Values and Backward Gradients Are Different Things

At every graph node, keep two mental boxes.

Forward value

What numerical value did this operation produce?

Backward gradient

How sensitive is the final loss to this node’s value?

For node z:

forward:  z
backward: dL/dz

Confusing these is a common source of backprop mistakes.

17. Why We Cache Forward Values

Suppose

Backward rule needs

So during backward, we need the original x.

For sigmoid or softmax, backward may also reuse forward outputs.

Frameworks therefore store enough intermediate information from the forward pass to compute backward efficiently.

This stored information consumes memory.

That is why training often needs much more memory than inference.

18. Memory vs Compute Tradeoff

If we store every intermediate activation, backward is fast but memory-heavy.

If we discard some and recompute them later, memory decreases but compute increases.

This tradeoff appears in techniques such as activation checkpointing.

Even advanced training systems are still negotiating the simple graph question:

Which forward values should I remember for backward?

19. Reverse-Mode Automatic Differentiation Preview

Suppose a function has millions of inputs/parameters but one scalar loss.

We want

Reverse-mode autodiff starts from the one scalar output and propagates sensitivities backward through the graph.

This computes all parameter gradients efficiently in roughly a small multiple of the forward-pass work.

That is why reverse mode is ideal for neural-network training.

20. Forward-Mode vs Reverse-Mode Intuition

Imagine function

Forward mode

Track how one chosen input perturbation affects everything downstream.

Good when number of inputs is small.

Reverse mode

Start from one chosen output sensitivity and propagate backward to all inputs.

Good when outputs are few—especially scalar loss.

Deep learning usually has:

millions/billions of parameters → one scalar loss

So reverse mode is the natural fit.

21. A Full Scalar Graph by Hand

Use

Define

Forward

Backward seed

Square

Subtract

Add

Multiply

One graph gives every local sensitivity we care about.

22. Verify the Final Gradients Directly

Original loss:

Differentiate directly:

At our values:

Matches the graph.

Similarly:

Again matches.

The computational graph did not invent new calculus.

It organized the chain rule so we could apply it locally and systematically.

23. A Tempting Wrong Idea: Gradients “Move” Backward as Numbers Only

It is common to visualize gradients flowing backward like water.

Useful—but incomplete.

At every edge, the upstream sensitivity is transformed by the local derivative.

So backward propagation is not merely passing a number backward unchanged.

It is repeated multiplication, matrix multiplication, broadcasting reduction, masking, and accumulation according to local operation rules.

The graph tells us which rule applies where.

24. Stop-Gradient / Detach Intuition

Sometimes we intentionally do not want gradients to flow through one edge.

Conceptually, we cut that dependency for differentiation.

Frameworks expose operations such as detach or stop-gradient mechanisms.

Forward values can still be used, but backward sensitivity does not propagate through that path.

This is useful in many advanced algorithms.

The graph picture makes the idea immediately understandable: forward edge exists; derivative edge is blocked.

25. In-Place Mutation Can Break the Story

Suppose a backward rule needs an earlier forward value x.

If code overwrites x in place before backward, the graph may no longer have the information required to compute correct gradients.

Modern frameworks track such issues carefully.

This is why some in-place tensor operations interact badly with autograd.

Again, the underlying reason is not framework superstition.

The backward graph needs specific saved values.

26. Code: Tiny Manual Graph

x = 2.0
w = 3.0
b = 1.0
y = 8.0

# forward
a = w * x
z = a + b
e = z - y
loss = e ** 2

# backward
dloss_dloss = 1.0
dloss_de = dloss_dloss * (2 * e)
dloss_dz = dloss_de * 1.0
dloss_da = dloss_dz * 1.0
dloss_db = dloss_dz * 1.0
dloss_dw = dloss_da * x
dloss_dx = dloss_da * w

assert loss == 1.0
assert dloss_dw == -4.0
assert dloss_db == -2.0
assert dloss_dx == -6.0

This is backpropagation without a framework.

27. The Graph as a Debugging Tool

When a gradient is wrong, inspect the path.

Ask:

1. What forward value entered this node?
2. What local derivative should this operation have?
3. What upstream gradient arrived?
4. Were multiple downstream contributions added?
5. Are tensor shapes compatible?
6. Was a value broadcast in forward and therefore reduced in backward?

This turns “backprop feels magical” into local bookkeeping.

28. Machine-Learning Connection — Residual Networks

A residual block contains a skip path:

x ───────────────┐
│                + ── output
└→ F(x) ─────────┘

Backward gradients can travel through both paths.

Their contributions add at x.

The graph perspective makes the derivative structure obvious and helps explain why skip connections improve gradient flow.

We will return to this in Chapter 102.

29. Machine-Learning Connection — Shared Parameters

Suppose the same weight is used at several places in a model.

The graph contains multiple downstream paths from that parameter to the loss.

Its total gradient is the sum of contributions from every use.

This is common in:

• recurrent networks,
• convolution kernels,
• tied embeddings,
• repeated modules.

Parameter sharing therefore relies on gradient accumulation over graph branches.

30. History Lens — From Symbolic Differentiation to Automatic Differentiation

Automatic differentiation is distinct from both symbolic differentiation and numerical finite differences.

It exploits the fact that computer programs are compositions of elementary operations with known derivatives.

By recording those operations as a graph and systematically applying the chain rule, derivatives can be computed to machine precision with predictable cost.

Reverse-mode automatic differentiation became especially important for neural networks because one scalar objective depends on enormous numbers of parameters.

Backpropagation is the neural-network specialization of this broader idea.

31. Distinctions That Matter

32. What We Discovered

1. Complex expressions can be decomposed into small operations.
2. A computational graph records dependency between values.
3. Forward pass computes and often stores intermediate values.
4. Each operation needs only a local derivative rule.
5. Backward pass repeatedly uses “upstream gradient × local derivative.”
6. Gradients multiply along a path.
7. Contributions add when a value reaches the loss through multiple paths.
8. Forward uses topological order; reverse mode uses reverse topological order.
9. Reverse-mode autodiff is ideal for many parameters and one scalar loss.
10. Tensor computational graphs use the same logic with matrix/tensor local rules.
11. Graph structure explains gradient accumulation, detach, activation memory and checkpointing.
12. This graph machinery is the operational foundation of backpropagation and autograd.

33. One-Minute Explanation

A computational graph rewrites a complicated formula as a network of small operations. During the forward pass, values move from inputs toward the loss. During the backward pass, each node receives an upstream sensitivity, multiplies it by its local derivative, and passes the result toward its inputs. If one value affects the loss through multiple paths, those gradient contributions add. This is just the chain rule organized as a graph, but that organization is what makes automatic differentiation and backpropagation practical for enormous neural networks.

34. Mastery Check

1. Draw a computational graph for $(wx+b-y)^2$.
2. What is the difference between a node’s forward value and backward gradient?
3. Explain “upstream × local.”
4. Why do gradient contributions add at branches?
5. Why must forward execution respect topological order?
6. Why does backward run in reverse dependency order?
7. Why is reverse-mode autodiff a good fit for neural networks?
8. What information may need to be cached during forward?
9. Why can activation checkpointing trade compute for memory?
10. How does parameter sharing affect gradient accumulation?
11. How is autodiff different from finite-difference numerical differentiation?
12. Why is a computational graph more useful than one giant symbolic derivative for deep networks?

🔭 Bridge to Chapter 023

We now know:

• what a gradient is,
• how the chain rule propagates sensitivity,
• how a computational graph organizes the calculation.

But computing the gradient is not yet learning.

A gradient only tells us the local uphill direction.

We still need an update rule that repeatedly changes parameters and actually reduces loss.

How do we turn gradients into a learning algorithm?

That question leads to gradient descent.