Dot Product: Measuring Alignment

Where We Are

Chapter 006 gave us two distinct notions of similarity.

Distance asks:

How far apart are two points?

Direction asks:

How similarly do two arrows point?

We saw that

are far apart but point exactly the same way.

So Euclidean distance cannot answer every similarity question.

We need an operation that turns two vectors into one number and somehow captures alignment.

That operation is the dot product.

But rather than memorize it, we will force it to appear.

1. Start With the Easiest Directions Possible

Take the unit vector pointing right:

Compare it with itself.

Whatever our alignment score is, this should be strongly positive.

Now compare it with the unit vector pointing up:

These directions are perpendicular.

A natural alignment score should be neutral.

Finally compare right with left:

These point opposite ways.

A useful alignment score should become negative.

So we want something like:

same direction       → positive
perpendicular        → zero
opposite direction   → negative

2. First Attempt: Add Coordinates

For

adding coordinates gives

But the vectors are perpendicular.

The score should be neutral, not strongly positive.

So addition does not capture alignment.

What does?

Try pairing corresponding coordinates and multiplying.

3. The Discovery: Pair, Multiply, Add

For two vectors

compute

Test the three direction cases.

Same direction:

Perpendicular:

Opposite:

Exactly the behavior we wanted.

This operation is the dot product:

Read it as:

multiply matching coordinates, then add all the products.

4. Why “Dot” Product?

The notation

uses a centered dot between the vectors.

It is not ordinary scalar multiplication, because each object contains several numbers.

It is also not element-wise multiplication.

Element-wise multiplication produces another vector:

The dot product goes one step further and adds those results:

So:

vector + vector  → scalar

That shape change is part of the meaning.

5. Tiny Worked Example

Let

Then

Compute term by term:

One scalar comes out.

This exact operation is why a neuron can take many inputs and produce one pre-activation value.

6. The Machine-Learning Meaning: A Weighted Vote

Suppose a house is represented by

Suppose weights are

Then

Each feature contributes a vote:

rooms contribution = 6
area contribution  = 5
---------------------
total               = 11

Intuition — A dot product is a weighted vote. Each input contributes according to both its value and its weight.

Add a bias $b$ and the familiar linear model appears:

The model from Chapter 001 was already hiding a dot product in one dimension.

7. But Why Does This Measure an Angle?

So far the dot product behaves correctly in special cases.

Now we connect it to geometry.

Take two vectors $\mathbf{a}$ and $\mathbf{b}$ with angle $\theta$ between them.

Consider the triangle formed by

The law of cosines says

Now expand the left side using coordinates.

8. Derive the Geometric Dot-Product Formula

Start with

Because norm squared is a vector dotted with itself,

Expand:

But

and

So

Compare with the law of cosines:

The first two terms match.

Therefore the remaining terms must match:

Cancel 2:

This is the deep connection.

The coordinate formula and the geometric formula are the same operation viewed from two worlds.

9. What the Sign Means

Since vector lengths are non-negative, the sign of the dot product comes from

Recall:

Therefore:

The sign is geometric information.

10. Magnitude Still Matters

A common misunderstanding is:

“The dot product is only angle.”

Not quite.

The formula is

So it depends on three things:

1. length of $\mathbf{a}$,
2. length of $\mathbf{b}$,
3. angle between them.

Take

and

Dot product is 1.

Now scale both by 10:

Same angle, but

The direction did not change. Magnitude did.

So raw dot product is alignment weighted by magnitude.

11. Cosine Similarity Removes Magnitude

If we want direction only, divide out the lengths:

This is cosine similarity:

For nonzero vectors its value lies between -1 and 1.

+1  → same direction
 0  → perpendicular
-1  → opposite direction

This is why embeddings are often compared with cosine similarity.

12. Example: Two Documents

Pretend two numbers measure how strongly a document talks about:

1. machine learning,
2. cooking.

Document A:

Document B:

Document C:

B is exactly half of A.

So A and B point in the same direction even though their magnitudes differ.

Their cosine similarity is 1.

C points mostly toward cooking, so its angle with A is much larger.

A recommendation system may care more about this direction than about raw vector size.

13. Projection: How Much of One Vector Lies Along Another?

Suppose $\mathbf{u}$ is a unit vector.

Then

measures the signed amount of $\mathbf{x}$ in direction $\mathbf{u}$.

Why?

Because

Since

we get

That is exactly the scalar projection.

This idea will later reappear in:

• PCA,
• attention,
• linear layers,
• embeddings,
• orthogonal decompositions.

14. Shape Check

Let

Then

x                shape (d,)
w                shape (d,)
w * x elementwise shape (d,)
sum(w * x)       scalar
w · x             scalar

This is why a single neuron can map many features to one number.

The dot product reduces dimension.

15. From One Neuron to Many

Suppose one neuron has weights

and another has

Each computes

Stack the weight vectors as rows:

Then both dot products can be computed together as

That is our bridge to matrices.

A matrix-vector multiplication is many dot products at once.

16. Code From Scratch

def dot(a, b):
    total = 0.0
    for ai, bi in zip(a, b):
        total += ai * bi
    return total

assert dot([2, 3, 4], [5, 1, 2]) == 21

NumPy expresses the same operation directly:

import numpy as np

a = np.array([2.0, 3.0, 4.0])
b = np.array([5.0, 1.0, 2.0])

assert np.isclose(np.dot(a, b), 21.0)

Again: library convenience comes after mathematical understanding.

17. Cosine Similarity From Scratch

import math

def norm(v):
    return math.sqrt(sum(x*x for x in v))


def cosine_similarity(a, b):
    denom = norm(a) * norm(b)
    if denom == 0:
        raise ValueError("cosine similarity is undefined for a zero vector")
    return dot(a, b) / denom

The zero-vector check matters.

A zero vector has no direction.

So asking for its angle with another vector is not meaningful.

18. Break It

Failure 1 — Different shapes

A dot product requires matching dimensions.

has no ordinary dot-product meaning.

Failure 2 — Forgetting magnitude

A large raw dot product does not necessarily mean a smaller angle. Large vector norms can inflate it.

Failure 3 — Zero vector cosine similarity

Cosine similarity divides by

If either norm is zero, the denominator is zero.

Failure 4 — Assuming cosine similarity is always non-negative

If vectors point in opposing directions, cosine similarity is negative.

19. 🎯 Machine-Learning Connection — Attention

Much later, Transformers will compare a query vector with key vectors.

At the heart of that comparison is a dot product:

Why?

Because dot products measure compatibility / alignment.

Attention is not a disconnected trick.

It is this chapter, scaled up.

20. 🎯 Machine-Learning Connection — Linear Classifiers

A classifier may compute

Geometrically, $\mathbf{w}$ defines a direction.

The dot product tells us how much $\mathbf{x}$ points along that learned direction.

So weights are not merely arbitrary coefficients.

They define geometry in feature space.

21. History Lens — From Geometry to Vector Analysis

The modern dot product emerged from the development of vector analysis in the nineteenth century, particularly through work associated with Gibbs and Heaviside, building on earlier geometric and algebraic traditions.

Its power comes from unifying two descriptions:

• coordinate arithmetic: multiply matching entries and add,
• geometry: lengths times cosine of the angle.

That bridge is exactly why the dot product became foundational in physics, engineering and machine learning.

22. Distinctions That Matter

23. What We Discovered

1. Multiplying matching coordinates and summing gives the dot product.
2. The dot product maps two equal-length vectors to one scalar.
3. Its sign reveals broad directional relationship.
4. It satisfies

5. Raw dot product depends on magnitude and direction.
6. Cosine similarity divides out magnitude.
7. Dot products are weighted sums, projections and alignment scores.
8. Matrix-vector multiplication will turn out to be many dot products at once.

24. One-Minute Explanation

The dot product takes two vectors, multiplies matching coordinates, and adds the results. That simple computation has a geometric meaning: it equals the product of the vector lengths times the cosine of the angle between them. So positive dot products usually mean the vectors point generally together, zero means perpendicular, and negative means generally opposite. Because magnitude also affects the raw dot product, cosine similarity divides by both lengths to isolate direction. In machine learning, dot products appear everywhere: linear models, neural-network layers, embeddings and Transformer attention.

25. Mastery Check

1. Compute $[2,3,4]\cdot[5,1,2]$ by hand.
2. Why does a dot product produce a scalar?
3. Why is the dot product zero for perpendicular vectors?
4. What does a negative dot product mean geometrically?
5. Why can two pairs with the same angle have different dot products?
6. How does cosine similarity remove magnitude?
7. Why is cosine similarity undefined for the zero vector?
8. Why can a neuron be understood as a dot product plus bias?
9. How does this chapter foreshadow attention?
10. How does stacking weight vectors lead naturally to matrices?

🔭 Bridge to Chapter 008

One vector of weights can compute one weighted sum.

But real datasets contain many examples and neural networks contain many neurons.

Writing every dot product separately would become unbearable.

How can we organize many vectors so that many dot products happen together?

That question forces us into matrices.