Vectors and dot products

Almost everything in machine learning is a vector: a row of features, a word embedding, the weights of a model, its gradient. This lesson builds the one operation you will use more than any other, the dot product, and shows how it turns into the cosine similarity behind semantic search.

What a vector is

You can read a vector two ways. As data, $(42, 3, 1)$ might be a house with 42 m², 3 rooms and 1 bathroom. As geometry, $(2, 3)$ is an arrow from the origin to the point two steps right and three steps up. The geometric picture only works in two or three dimensions, but the algebra is identical in 768 dimensions, which is why we learn it in 2D and trust it everywhere.

Adding and scaling

Vectors of the same length add component by component, and multiplying by a number (a scalar) scales every component:

$$\vec{u} + \vec{v} = (u_1 + v_1, \dots, u_n + v_n), \qquad c\,\vec{v} = (c v_1, \dots, c v_n).$$

With $\vec{u} = (2, 3)$ and $\vec{v} = (3, -1)$ the sum is $(5, 2)$. Geometrically, you walk along $\vec{u}$ and then along $\vec{v}$:

Length

The length (or norm) of a vector comes from Pythagoras:

$$\|\vec{v}\| = \sqrt{v_1^2 + v_2^2 + \dots + v_n^2}.$$

For $\vec{u} = (2, 3)$, $\|\vec{u}\| = \sqrt{4 + 9} = \sqrt{13} \approx 3.606$. Dividing a vector by its length gives a unit vector that points the same way and has length 1.

The dot product

It takes two vectors and returns one number. Notice that $\vec{v} \cdot \vec{v} = \|\vec{v}\|^2$, so length is just a dot product in disguise.

The reason the dot product matters is its geometric meaning:

So the sign of the dot product tells you about direction:

• $\vec{u} \cdot \vec{v} > 0$: the angle is less than 90°, the vectors point roughly the same way.
• $\vec{u} \cdot \vec{v} = 0$: they are perpendicular (orthogonal).
• $\vec{u} \cdot \vec{v} < 0$: the angle is more than 90°, they point roughly opposite ways.

Drag the purple point. The readout shows its dot product with the fixed vector $(3, 1)$; find the line of points where it is exactly zero.

(3.00, 1.00) · (1.00, 3.00) = 6.00 · cos θ = 0.60 · θ = 53.1°

Cosine similarity

Rearranging the geometric form gives the cosine similarity, which compares directions and ignores lengths:

$$\operatorname{cos\_sim}(\vec{u}, \vec{v}) = \frac{\vec{u} \cdot \vec{v}}{\|\vec{u}\|\,\|\vec{v}\|} \in [-1, 1].$$

Text-embedding models map sentences with similar meaning to vectors that point in similar directions, so cosine similarity is the standard score in semantic search and RAG retrieval. If the embeddings are already normalized to length 1, the cosine similarity is just the dot product, which is why vector databases often store unit vectors.

In code

NumPy’s @ operator is the dot product for 1-D arrays:

import numpy as np

u = np.array([2.0, 3.0])
v = np.array([3.0, -1.0])

dot = u @ v                                    # 3.0
norm_u = np.linalg.norm(u)                     # 3.6056
cos_sim = dot / (norm_u * np.linalg.norm(v))   # 0.2631
angle = np.degrees(np.arccos(cos_sim))         # 74.74

Video

Exercise

Write a function cosine_similarity(a, b) in NumPy, then use it on sentence embeddings: embed five sentences (two pairs that mean the same thing and one unrelated sentence) with any open embedding model, and print the 5 × 5 similarity matrix. Check that each paraphrase pair scores higher than any pair involving the unrelated sentence.

Key ideas

• A vector is an ordered list of numbers, which you can picture as an arrow.
• $\vec{u} \cdot \vec{v} = \sum_i u_i v_i = \|\vec{u}\|\|\vec{v}\|\cos\theta$: one number that encodes length and angle.
• Cosine similarity is the dot product of unit vectors, and it is the default similarity score for embeddings.