A course built from first principles

The Mathematics
Behind Machine Learning

Phase A of a four-phase course. Fourteen interactive lectures that build the functions, calculus, linear algebra and probability you need — deriving every step, verifying every claim numerically, and implementing the ideas in NumPy before any library is allowed near them.

Lectures

Completed

Parts

Of four phases

Not started.

How to use this course

Do not scroll past a derivation. Each one is hidden step by step for a reason — predict the next line before you click, and notice when you cannot.
Answer the quiz before clicking. A wrong answer that you then understand is worth more than a right answer you guessed.
Run and break the code. Every Python block is editable. Change a number, make it fail, work out why.
Attempt every exercise on paper before revealing the solution. Bring me the ones you got stuck on — that is the most useful information you can give me.
Do not skip ahead. Each lecture assumes the previous ones. The order is the content.

Phase A · The lectures

I — The Language of Functions

What a Function Really Is

Every model you will ever train is a function. Before we can talk about learning one, we have to be ruthless about what the word means — and the part almost everyone skips is that a function is not just a formula.

State the formal definition of a function and say why \(\text{domain}\) is part of the object, not an afterthought
Compute the domain of an expression and justify each exclusion
Decide whether a function is injective, surjective, bijective — and therefore invertible

✓

Exponentials and Logarithms

Machine learning is addicted to logarithms. Log-likelihood, cross-entropy, logits, log-sum-exp, perplexity, KL divergence. This lecture builds the logarithm from its definition so that every one of those is transparent — and so that you never again wonder what log(−2) is.

Prove that \(a^x > 0\) for every real \(x\), and derive the domain of \(\log_b\) from it
Derive all four logarithm laws from the exponent laws rather than memorising them
Explain why \(e\) is the natural base, in terms of the derivative

✓

Limits and Continuity

A limit answers a question the function itself refuses to: what value is this heading towards at a point where it may not even be defined? Every derivative, every integral, and every convergence guarantee in optimisation is built on this one idea.

State what \(\lim_{x\to c} f(x) = L\) means, both informally and in \(\varepsilon\)–\(\delta\) form
Compute limits by factoring, and explain why \(\tfrac00\) is indeterminate rather than undefined
Distinguish 'the limit exists' from 'the function is defined' from 'the function is continuous'

✓

II — Calculus of One Variable

The Derivative from First Principles

A derivative is a limit of slopes. That single sentence, unpacked carefully, gives you tangent lines, local linear approximation, and the entire mechanism by which a neural network learns.

Derive \(f'(x)\) from the difference-quotient definition, for polynomials and \(1/x\)
Explain what the derivative means geometrically, physically, and as a linear approximation
Prove that differentiability implies continuity, and give a counterexample to the converse

✓

Differentiation Rules and the Chain Rule

Four rules replace every difference quotient you would otherwise have to compute. The last of them — the chain rule — is not merely convenient: it is the entire mathematical content of backpropagation, and therefore of how every neural network ever trained was trained.

Derive the sum, product, quotient and chain rules from the definition
Prove \(\frac{d}{dx}e^x = e^x\) and \(\frac{d}{dx}\ln x = \frac1x\), closing the debt from Lecture 2
Differentiate a long composition by peeling one layer at a time

✓

Taylor Series and Local Approximation

The tangent line is the best straight-line approximation to a curve. Taylor's theorem says what the best parabola is, the best cubic, and exactly how wrong each of them is — which is where second-order optimisers, learning-rate bounds, and half of numerical analysis come from.

Construct the degree-\(n\) Taylor polynomial by matching derivatives at a point
State Taylor's theorem with the Lagrange remainder and use it to bound an approximation error
Derive the series for \(e^x\), \(\sin x\), \(\ln(1+x)\), \(\tfrac1{1-x}\)

✓

Optimisation in One Dimension

Training a model is minimising a function. This lecture assembles the complete one-dimensional theory — critical points, the second-derivative test, and convexity — so that when we go to a million dimensions in Lecture 12, only the bookkeeping changes.

Find and classify critical points using the first and second derivative tests
State and use the definition of a convex function, and prove \(f''\ge0 \Rightarrow\) convex
Explain precisely what convexity buys you: every local minimum is global

✓

III — Linear Algebra

Vectors, Span, and Geometry

A data point is a vector. A model's weights are a vector. A gradient is a vector. This lecture builds the object and its geometry — length, angle, projection — because every later result in the course is a geometric statement wearing algebraic clothes.

State the vector space axioms and check them for \(\RR^d\)
Compute a span and decide whether a set of vectors is linearly independent
Derive \(\vu\cdot\vv = \|\vu\|\|\vv\|\cos\theta\) from the law of cosines

✓

Matrices as Linear Maps

A matrix is not a grid of numbers that you multiply by a strange rule. It is a linear function, written down. Once you see that, matrix multiplication stops being arbitrary — it is composition, and the shape rules are just Lecture 1's requirement that the codomain of one map match the domain of the next.

Define a linear map and prove every such map \(\RR^n\to\RR^m\) is a matrix
Read \(\mA\vx\) both as a stack of dot products and as a combination of columns
Derive the matrix multiplication formula from composition

✓

Least Squares and the Normal Equations

Real data gives you more equations than unknowns, and no exact solution. Least squares is the principled answer: project the target onto what your model can actually reach. This is linear regression, derived twice — geometrically and by calculus — and shown to be the same theorem.

Explain why \(\mX\vw=\vy\) is usually unsolvable and what we should ask instead
Derive the normal equations \(\mX^\top\mX\vw = \mX^\top\vy\) from orthogonality
Derive the same equations by minimising \(\|\mX\vw-\vy\|^2\)

✓

Eigenvalues, Eigenvectors, and Quadratic Forms

Some directions survive a linear map unchanged except for scaling. Finding them turns a matrix into a list of numbers, turns curvature into geometry, and explains — precisely — why gradient descent zig-zags, why learning rates have a ceiling, and what PCA is doing.

Define eigenvalues and eigenvectors and compute them for \(2\times2\) matrices by hand
State the spectral theorem for symmetric matrices and use the diagonalisation \(\mA=\mQ\Lambda\mQ^\top\)
Classify a quadratic form as positive definite, semi-definite or indefinite

✓

IV — Multivariable Calculus

Gradients, Jacobians, and Hessians

Calculus in one variable, plus linear algebra, equals the calculus that machine learning actually runs on. The gradient points uphill, the Jacobian is a derivative that is a matrix, the Hessian is curvature — and the multivariable chain rule is backpropagation.

Compute partial derivatives and assemble a gradient, with correct shapes
Prove that \(\grad f\) is the direction of steepest ascent and is perpendicular to level sets
Use the multivariable chain rule, and see it as the backpropagation recursion

✓

V — Probability and Statistics

Random Variables, Expectation, and Variance

Every loss you minimise is an expectation you cannot compute, estimated from a sample. Understanding what that sentence means — and how wrong the estimate can be — is the difference between running experiments and understanding them.

State the probability axioms and define a random variable properly
Prove linearity of expectation, and use it without assuming independence
Compute variance and covariance, and distinguish independent from uncorrelated

✓

Bayes, Distributions, and Maximum Likelihood

The last lecture of Phase A, and the one that ties it together. Maximum likelihood is a single principle from which squared error, cross-entropy, ridge and LASSO all fall out — and Bayes' theorem is the reason a 99%-accurate test can be wrong most of the time.

Derive Bayes' theorem and use it to resolve the base-rate fallacy quantitatively
Write down a likelihood, take its log, and maximise it
Derive the MLE for Bernoulli, Gaussian, linear regression and logistic regression

✓

What comes after

Phase A is the foundation. Here is the rest of the road, so you can see where each piece of mathematics is going. Nothing below is available yet — and that is deliberate.

Phase BClassical Machine Learningup next

Supervised vs unsupervised learning; the learning problem stated formally
Linear regression: least squares, the normal equations, the geometry of projection
Maximum likelihood, and why squared error is a Gaussian assumption in disguise
Logistic regression, the sigmoid, cross-entropy, and the logit link
MAP estimation, priors, and how L2/L1 regularisation falls out of them
Gradient descent, convexity, learning rates, momentum
Bias-variance decomposition, derived term by term
PCA from three directions: variance maximisation, reconstruction error, SVD
k-means and Gaussian mixture models; the EM algorithm
Naive Bayes; decision trees; random forests; gradient boosting
SVMs, the margin, duality, and the kernel trick
Model evaluation, cross-validation, and how to not fool yourself

Phase CStatistical Learning Theoryafter B

Empirical risk minimisation and what generalisation actually means
Concentration inequalities: Markov, Chebyshev, Hoeffding
PAC learning, sample complexity
VC dimension and shattering
Rademacher complexity; margin bounds
Regularisation theory; the bias-variance tradeoff revisited rigorously

Phase DDeep Learningafter B

Perceptrons, and the limits of linear models
Computational graphs and reverse-mode automatic differentiation
Backpropagation derived from the multivariable chain rule
Building a network from scratch in NumPy, then rebuilding it in PyTorch
Initialisation, normalisation, and why depth is hard to train
Optimisers: SGD, momentum, RMSProp, Adam — and their failure modes
Convolutional networks; weight sharing as an inductive bias
Sequence models, attention, and the transformer
Generative models: VAEs, flows, diffusion
Reading and reproducing research papers

What you should NOT study yet

Being specific about this saves more time than any study tip. Until Phase A is finished, avoid:

Transformers, LLMs, and attention. They are compositions of linear maps and softmaxes differentiated by the chain rule. Every one of those words is a Phase A topic. Reading about them now buys vocabulary, not understanding.
PyTorch and TensorFlow tutorials. loss.backward() will teach you nothing until you have written backpropagation by hand. Phase D, not before.
Kaggle competitions. They reward pipeline-assembly and hyperparameter luck. That is a real skill, but it is not the one you said you wanted.
Measure-theoretic probability. Genuinely necessary for research-level theory, genuinely useless before you are fluent with ordinary random variables. It belongs after Phase C.
Proof-heavy real analysis. You need the ideas of limits and continuity, which Lecture 3 gives you. You do not yet need epsilon-delta proofs of the Heine–Borel theorem.

Your progress is stored locally in this browser only. Nothing is uploaded anywhere. Interactive plots and derivations work offline; typeset mathematics and the in-page Python runner need an internet connection the first time a page loads.