A data point is a vector. A model's weights are a vector. A gradient is a vector. This lecture builds the object and its geometry — length, angle, projection — because every later result in the course is a geometric statement wearing algebraic clothes.
One house: 120 square metres, 3 bedrooms, 15 years old, 2 km from the station. Write it as an ordered list:
\[\vx = \begin{bmatrix} 120 \\ 3 \\ 15 \\ 2\end{bmatrix} \in \RR^4.\]
That is a vector: a point in a four-dimensional space, one axis per feature. A dataset of \(n\) houses is \(n\) such points. Stack them as rows and you have the design matrix \(\mX\in\RR^{n\times d}\), the object every supervised learning algorithm takes as input.
We cannot picture \(\RR^4\), let alone \(\RR^{784}\). But nearly everything we need is two-dimensional geometry that happens to be written in a way that does not care about the dimension. Learn it in the plane, where you can see it; the algebra then carries it anywhere.
Two conventions to fix now, because mismatching them causes more bugs than any other single thing in ML code. Vectors are columns by default: \(\vx\in\RR^d\) means a \(d\times1\) object. Data matrices are rows-as-samples: \(\mX\in\RR^{n\times d}\) has one row per data point. So the \(i\)-th sample is the column vector \(\vx_i = \mX[i,:]^\top\). This is why you constantly see \(\mX\vw\) (an \(n\)-vector of predictions) rather than \(\vw^\top\mX\).
A vector is not "a thing with magnitude and direction" — that is a description of one example. The real definition is behavioural: a vector is an element of a set on which you can add and scale, subject to rules.
A vector space over \(\RR\) is a set \(V\) with two operations, addition \(\vu+\vv\) and scalar multiplication \(\alpha\vu\), satisfying, for all \(\vu,\vv,\vw\in V\) and \(\alpha,\beta\in\RR\):
\(\RR^d\) with componentwise operations satisfies all of them — each axiom reduces to the corresponding property of real numbers, applied one coordinate at a time. But so do many other things, and that is the payoff of abstraction:
Every theorem we prove using only the axioms applies to all of them at once. That is why "function space" methods in machine learning are not an analogy: they are the same theorems.
A linear combination of \(\vv_1,\dots,\vv_k\) is any vector of the form
\[ \alpha_1\vv_1 + \alpha_2\vv_2 + \cdots + \alpha_k\vv_k, \qquad \alpha_i\in\RR. \]Their span is the set of all such combinations:
\[ \operatorname{span}\{\vv_1,\dots,\vv_k\} = \left\{\textstyle\sum_i \alpha_i\vv_i \;:\; \alpha_i\in\RR\right\}. \]It is always itself a vector space — a subspace of the ambient one.
\(\vv_1,\dots,\vv_k\) are linearly independent if the only way to write the zero vector as a combination of them is the trivial way:
\[ \alpha_1\vv_1+\cdots+\alpha_k\vv_k = \vzero \quad\Longrightarrow\quad \alpha_1=\cdots=\alpha_k=0. \]They are dependent otherwise — meaning at least one of them is a combination of the rest, and therefore redundant.
A basis for a space \(V\) is a set that is both independent and spans \(V\). Every basis of \(V\) has the same number of elements; that number is the dimension of \(V\).
For \(\vu,\vv\in\RR^d\),
\[ \vu\cdot\vv \;=\; \vu^\top\vv \;=\; \sum_{i=1}^{d}u_iv_i. \]The Euclidean norm is \(\|\vu\| = \sqrt{\vu\cdot\vu} = \sqrt{\sum_i u_i^2}\), which is Pythagoras in \(d\) dimensions.
Algebraically that is a sum of products, and it is not obvious it means anything. The following theorem says it means everything: it encodes both length and angle.
Given \(\vv\), what is the closest point to it on the line through \(\vu\)? Answering that gives least squares, PCA, Fourier analysis and Gram–Schmidt. Let us derive it two ways, because they give different insights.
Derive the same formula by minimising instead of imposing orthogonality. Define
\[g(k) = \|\vv - k\vu\|^2\]and minimise it over \(k\) using Lecture 7's machinery. You should get the same \(k\), and you should also be able to confirm it is a minimum rather than a maximum.
Expand using bilinearity of the dot product:
\[ g(k) = (\vv-k\vu)\cdot(\vv-k\vu) = \|\vv\|^2 - 2k\,(\vu\cdot\vv) + k^2\|\vu\|^2. \]This is a quadratic in \(k\) — the norms and the dot product are just numbers. Differentiate:
\[ g'(k) = -2(\vu\cdot\vv) + 2k\|\vu\|^2 \;\stackrel{!}{=}\;0 \quad\Longrightarrow\quad k = \frac{\vu\cdot\vv}{\|\vu\|^2}. \]Same answer ✓. And \(g''(k)=2\|\vu\|^2>0\) whenever \(\vu\neq\vzero\), so \(g\) is strictly convex and this critical point is the unique global minimum, by Lecture 7's convexity theorem.
Why two derivations matter. The orthogonality route says the best approximation leaves a perpendicular error — a geometric fact you can see. The minimisation route says the best approximation minimises squared distance — an optimisation fact you can compute. They are the same theorem, and the equivalence is precisely why least squares has a clean geometric interpretation. Every appearance of \((\mX^\top\mX)^{-1}\mX^\top\) in statistics is this equivalence in matrix form.
The Euclidean norm is one choice among a family:
\[\|\vx\|_p = \left(\sum_{i=1}^{d}|x_i|^p\right)^{1/p}, \qquad p\ge 1; \qquad \|\vx\|_\infty = \max_i |x_i|.\]
Cosine similarity. Rearranging the angle formula,
\[\cos\theta = \frac{\vu\cdot\vv}{\|\vu\|\,\|\vv\|} = \hat{\vu}\cdot\hat{\vv} \in[-1,1].\]This is how similarity between embeddings is measured in essentially every retrieval, recommendation and RAG system. Two reasons it is preferred over raw Euclidean distance:
Let \(\vu=(2,-1,3)\), \(\vv=(1,4,-2)\).
1. \(\vu\cdot\vv = 2(1)+(-1)(4)+3(-2) = 2-4-6 = -8\).
\(\|\vu\|=\sqrt{4+1+9}=\sqrt{14}\approx3.7417\), \(\|\vv\|=\sqrt{1+16+4}=\sqrt{21}\approx4.5826\).
\(\cos\theta = \frac{-8}{\sqrt{14}\sqrt{21}} = \frac{-8}{\sqrt{294}} \approx -0.46657\), so
\(\theta\approx 117.8^\circ\). Obtuse, consistent with the negative dot product.
2. \(k = \frac{\vu\cdot\vv}{\|\vu\|^2} = \frac{-8}{14} = -\frac47\).
\(\operatorname{proj}_{\vu}(\vv) = -\frac47(2,-1,3) = \left(-\frac87,\frac47,-\frac{12}{7}\right)
\approx(-1.1429, 0.5714, -1.7143)\).
Residual \(\vr = \vv - \vp = \left(1+\frac87,\ 4-\frac47,\ -2+\frac{12}{7}\right)
= \left(\frac{15}{7},\frac{24}{7},-\frac27\right)\).
Check: \(\vr\cdot\vu = \frac{30}{7}-\frac{24}{7}-\frac{6}{7} = 0\) ✓.
3. The cross product works in \(\RR^3\):
\(\vu\times\vv = \big((-1)(-2)-3(4),\ 3(1)-2(-2),\ 2(4)-(-1)(1)\big) = (2-12,\ 3+4,\ 8+1) = (-10,7,9)\).
Check: \((-10)(2)+7(-1)+9(3) = -20-7+27=0\) ✓ and \((-10)(1)+7(4)+9(-2) = -10+28-18=0\) ✓.
Alternatively, without the cross product: solve \(2a-b+3c=0\) and \(a+4b-2c=0\). Two equations, three
unknowns, so a one-dimensional family of solutions — which is what "perpendicular to a plane" means.
Prove the Cauchy–Schwarz inequality:
\[ |\vu\cdot\vv| \le \|\vu\|\,\|\vv\| \quad\text{for all }\vu,\vv\in\RR^d,\]with equality if and only if \(\vu\) and \(\vv\) are parallel.
Hint: consider \(g(t)=\|\vv-t\vu\|^2\ge0\) and use the fact that a non-negative quadratic has non-positive discriminant.
Assume \(\vu\neq\vzero\) (if \(\vu=\vzero\) both sides are 0 and the inequality holds trivially).
Define \(g(t)=\|\vv-t\vu\|^2\). A squared norm is a sum of squares, so \(g(t)\ge0\) for every real \(t\). Expanding as in the projection derivation:
\[g(t) = \|\vu\|^2 t^2 - 2(\vu\cdot\vv)\,t + \|\vv\|^2 .\]This is a quadratic \(at^2+bt+c\) with \(a=\|\vu\|^2>0\), \(b=-2(\vu\cdot\vv)\), \(c=\|\vv\|^2\).
An upward parabola that is never negative has at most one real root, so its discriminant satisfies \(b^2-4ac\le 0\):
\[4(\vu\cdot\vv)^2 - 4\|\vu\|^2\|\vv\|^2 \le 0 \quad\Longrightarrow\quad (\vu\cdot\vv)^2 \le \|\vu\|^2\|\vv\|^2.\]Taking non-negative square roots gives \(|\vu\cdot\vv|\le\|\vu\|\|\vv\|\). ∎
Equality. Equality in the discriminant means the parabola has exactly one real root \(t_0\), i.e. \(g(t_0)=\|\vv-t_0\vu\|^2=0\), i.e. \(\vv=t_0\vu\) — the vectors are parallel. Conversely if \(\vv=t_0\vu\) then \(|\vu\cdot\vv|=|t_0|\|\vu\|^2=\|\vu\|\|\vv\|\). ∎
Why it matters. Cauchy–Schwarz is what guarantees \(\cos\theta\in[-1,1]\), so the angle is well defined in any dimension. It also underlies the triangle inequality, the fact that correlation lies in \([-1,1]\), and Chebyshev's inequality in Lecture 14.
The vectors \(\vv_1=(1,1,0)\), \(\vv_2=(1,0,1)\), \(\vv_3=(0,1,1)\).
1. \(\alpha_1(1,1,0)+\alpha_2(1,0,1)+\alpha_3(0,1,1)=(0,0,0)\) gives
\[\alpha_1+\alpha_2=0,\qquad \alpha_1+\alpha_3=0,\qquad \alpha_2+\alpha_3=0.\]From the first two, \(\alpha_2=-\alpha_1\) and \(\alpha_3=-\alpha_1\). Substituting into the third: \(-2\alpha_1=0\), so \(\alpha_1=0\), and then \(\alpha_2=\alpha_3=0\). Only the trivial solution → independent ✓.
2. Solve \(\alpha_1+\alpha_2=2\), \(\alpha_1+\alpha_3=3\), \(\alpha_2+\alpha_3=4\).
Add all three: \(2(\alpha_1+\alpha_2+\alpha_3)=9\), so the total is \(4.5\). Subtracting each original
equation in turn: \(\alpha_3=4.5-2=2.5\), \(\alpha_2=4.5-3=1.5\), \(\alpha_1=4.5-4=0.5\).
Check: \(0.5(1,1,0)+1.5(1,0,1)+2.5(0,1,1)=(0.5+1.5,\ 0.5+2.5,\ 1.5+2.5)=(2,3,4)\) ✓.
3. Three independent vectors in \(\RR^3\) span a 3-dimensional subspace of \(\RR^3\), which must be all of \(\RR^3\). So they form a basis — and part 2 was guaranteed to succeed before we did any arithmetic. Note also that the solution in part 2 was unique, which is exactly what independence buys you: a basis gives every vector exactly one set of coordinates.
Coding + geometry. The curse of dimensionality, measured.
import numpy as np
rng = np.random.default_rng(0)
for d in (2, 10, 100, 1000, 10000):
u = rng.normal(size=(1000, d)); v = rng.normal(size=(1000, d))
u /= np.linalg.norm(u, axis=1, keepdims=True)
v /= np.linalg.norm(v, axis=1, keepdims=True)
cos = np.sum(u*v, axis=1)
ang = np.degrees(np.arccos(np.clip(cos, -1, 1)))
print(f"d={d:6d} mean angle {ang.mean():6.2f} deg sd {ang.std():5.2f}"
f" mean|cos| {np.abs(cos).mean():.4f} 1/sqrt(d) {1/np.sqrt(d):.4f}")1. The mean angle sits at \(90^\circ\) for every \(d\), but the spread collapses: at \(d=2\) the standard deviation is about \(52^\circ\), at \(d=100\) about \(5.7^\circ\), at \(d=10000\) about \(0.57^\circ\). In high dimensions two random vectors are almost exactly orthogonal, essentially always.
That is surprising because in the plane, random vectors are as likely to be nearly parallel as nearly perpendicular. High-dimensional space has vastly more "sideways" directions than "along" ones, and almost all of the volume sits near the equator relative to any fixed axis.
2. Write \(\hat{\vu}\cdot\hat{\vv}=\sum_i \hat u_i\hat v_i\). Since \(\vu\) and \(\vv\) are independent and each is symmetric about the origin, \(\E[\hat u_i\hat v_i]=\E[\hat u_i]\E[\hat v_i]=0\), so \(\E[\hat{\vu}\cdot\hat{\vv}]=0\) by linearity of expectation (Lecture 13).
For the variance: condition on \(\hat{\vu}\). By rotational symmetry of the Gaussian we may take \(\hat{\vu}=\ve_1\), so \(\hat{\vu}\cdot\hat{\vv}=\hat v_1\), the first coordinate of a uniformly random unit vector. By symmetry \(\E[\hat v_i^2]\) is the same for all \(i\), and \(\sum_i \hat v_i^2 = 1\), so \(\E[\hat v_1^2]=\frac1d\). Hence \(\Var(\hat{\vu}\cdot\hat{\vv})=\frac1d\) and the standard deviation is \(1/\sqrt d\) ✓ — which is exactly the last column your code prints, matching \(\E|\cos|\) up to the constant \(\sqrt{2/\pi}\approx0.8\).
3. Two consequences pulling in opposite directions.
Good news: a high-dimensional space has room for enormous numbers of nearly-orthogonal directions, so unrelated concepts can be given essentially non-interfering embeddings. This is why embedding dimensions of 768 or 1536 work at all, and it is the intuition behind the Johnson–Lindenstrauss lemma.
Bad news: the baseline similarity between unrelated items concentrates tightly around 0, so the entire useful signal lives in a narrow band, and small numerical or estimation errors eat a large fraction of it. It also means a raw cosine value is uninterpretable without a reference distribution — 0.3 might be a strong match in one embedding space and noise in another. This is why practical retrieval systems compare relative rankings rather than absolute thresholds, and why re-ranking models exist.
Next: vectors are the data; matrices are what we do to it. Lecture 9 derives matrix multiplication from the requirement that it represent composition of linear maps — which explains, among other things, why the dimensions have to line up the way they do.