🧪 Interactive Lab

The matching notebook is the complete hands-on laboratory for this lesson. It contains the runnable code, experiments, visualizations, and challenges.

🧭 Where this lesson fits

Previous lesson: Blog 16 — Attention: What Should I Look At?.

Today: Blog 17 — Transformers: Building With Attention.

Next lesson: Blog 18 — How Does a Language Model Learn to Predict Text?.

Student rule: if you cannot explain why this lesson follows the previous one, stop and reread the final takeaway of the previous blog. The equations below should feel like a continuation, not a new language.

1. From tokens to representations

Suppose the input is

The cat sat

The text is converted into tokens, then token embeddings.

But embeddings alone do not tell the model where each token occurs.

We therefore need positional information.

2. Position matters

These sequences contain the same words:

cat chased dog
dog chased cat

but their meanings differ.

A Transformer therefore needs information about token position.

One family of methods uses sinusoidal positional encodings such as

Modern models may instead use learned or relative/rotary positional methods. The central requirement is the same: the model needs access to positional information.

3. One Transformer block

A simplified Transformer block can be pictured as:

flowchart TD
    X[Token representations] --> A[Multi-head self-attention]
    A --> R1[Residual connection + normalization]
    R1 --> F[Feed-forward network]
    F --> R2[Residual connection + normalization]
    R2 --> O[Output representations]

Different Transformer variants use slightly different ordering and normalization conventions, but these are the major ideas.

4. Multi-head attention

One attention mechanism may learn one type of relationship.

Instead of using one head, Transformers use multiple heads.

For head $i$:

Then concatenate the heads:

Different heads can learn different interaction patterns.

Do not assume that every head has a clean human-readable role; learned representations can be distributed and redundant.

5. Feed-forward network

After attention, each token representation passes through a position-wise feed-forward network.

A simplified form is

Attention mixes information between positions.

The feed-forward network transforms each position’s representation independently.

This division is a useful mental model.

6. Residual connections

Suppose a block computes $F(x)$.

A residual connection produces

Instead of forcing a layer to learn an entirely new representation, the network can learn a useful modification of the existing one.

Residual connections also help optimization in deep networks.

7. Why Transformers train efficiently

RNNs naturally process tokens sequentially.

Self-attention can process many sequence positions in parallel during training.

For a sequence length $n$, the attention score matrix has shape

because every query can compare with every key.

That gives attention a major computational cost as sequence length grows, roughly quadratic in $n$ for standard full attention.

So Transformers trade sequential recurrence for highly parallel computation and a different scaling challenge.

8. Encoder and decoder families

The original Transformer architecture had an encoder and decoder.

Later models often use different subsets:

• encoder-only models for representation tasks;
• decoder-only models for autoregressive generation;
• encoder-decoder models for sequence-to-sequence tasks.

This is why “Transformer” describes an architecture family rather than one single model design.

9. A small PyTorch module

import torch
import torch.nn as nn

layer = nn.TransformerEncoderLayer(
    d_model=64,
    nhead=4,
    batch_first=True
)

x = torch.randn(8, 20, 64)
y = layer(x)

print(y.shape)

The shape remains

8 × 20 × 64

The layer transforms the representations while preserving batch size, sequence length and model dimension in this example.

10. The big picture

Text
 ↓
Tokens
 ↓
Embeddings + position information
 ↓
Transformer blocks
 ├─ Self-attention
 ├─ Residual + normalization
 ├─ Feed-forward network
 └─ Residual + normalization
 ↓
Contextual representations
 ↓
Prediction head

This architecture can be stacked many times.

Think Like a Scientist 🧠

Explain the difference between:

and

Then ask:

Why would the representation of “bank” need to change depending on the sentence?

If you can answer that, you understand one of the central motivations behind attention.

What you should remember

A Transformer combines attention, nonlinear transformations, residual connections, normalization and positional information into a highly scalable sequence architecture.

Attention answers “what should interact with what?”

The surrounding architecture makes those interactions trainable and reusable at scale.

Now we can finally ask the question that powers modern language models:

How does a model learn to predict text?

📚 Go Deeper — Understand the Architecture, Not Just the Name

3Blue1Brown is one of the best visual companions for understanding the mathematical structure behind neural networks and attention.

ZacharyLLM is particularly relevant here for connecting the Transformer architecture to modern LLMs.

Frame Zero is useful for first-principles explanations of modern ML systems.

Visual Kernel can provide another visual/technical perspective on model internals.

Welch Labs is valuable for its implementation-first philosophy and supporting code; its AI material explicitly combines exercises, graphics and code.

Use MrJensenMath10 when the underlying algebra, functions or matrix operations need reinforcement.

The architectural insight

A Transformer is not “just attention.”

It is a repeated block of cooperating ideas:

That repeated structure is what turns a single attention mechanism into a deep architecture.