Classical ML Phase B
Phase B of a four-phase course

Classical
Machine Learning

Twelve interactive lectures that build the algorithms — linear and logistic regression, regularisation, PCA, mixtures, trees, boosting, kernels — from the mathematics you already have. Nothing here is presented as a recipe. Every model is derived, every loss is traced back to an assumption, and every implementation is written in NumPy before a library is allowed near it.

12
Lectures
0
Completed
5
Parts
B
Of four phases
Not started.

Prerequisite — Phase A

This phase assumes the mathematics from Phase A and refers back to it constantly by lecture number. If the two folders sit side by side, this link works: → Phase A · The Mathematics Behind Machine Learning.

The dependency map below says which Phase A lecture each Phase B lecture leans on hardest. If a derivation here loses you, that is where to go back to.

Phase B lectureLeans hardest on
01 Learning problemA13 expectation and the empirical mean
02 Linear regressionA10 normal equations, A8 projection
03 Gradient descentA6 Taylor, A11 eigenvalues and conditioning
04 Maximum likelihoodA14 MLE, A2 logs and the argmax theorem
05 Logistic regressionA5 chain rule and σ′, A12 gradients
06 RegularisationA14 MAP and priors, A11 spectrum shifting
07 Bias–variance and evaluationA13 variance, A14 base rates
08 PCAA11 spectral theorem and Rayleigh quotients
09 k-means and mixturesA14 likelihood, A7 convexity
10 Naive BayesA14 Bayes' theorem and conditional independence
11 Trees and boostingA2 entropy, A4 gradients as residuals
12 SVMs and kernelsA8 inner products, A12 constrained optimisation

How to use this phase

  1. Implement before you import. Every lecture builds the algorithm in NumPy first. Only then does scikit-learn appear, and only to check the from-scratch version against it.
  2. Do not scroll past a derivation. Predict the next line before you click.
  3. Break the code. Change a hyperparameter until the method fails, then work out which assumption you violated. That is the fastest way to learn what an algorithm actually needs.
  4. Attempt every exercise on paper before revealing the solution.
  5. Notice the repetition. Nearly every model here is: choose a hypothesis class, choose a loss from a noise model, add a regulariser from a prior, minimise. Once you see that pattern the field stops being a list of tricks.

Phase B · The lectures

I — The Learning Problem
01
What Learning Actually Means
Before any algorithm: a precise statement of the problem all of them are trying to solve. Risk, empirical risk, hypothesis classes, and the single rule that separates real machine learning from fooling yourself.
  • State the supervised learning problem formally, and name every symbol
  • Distinguish true risk from empirical risk and explain why the gap exists
  • Explain what a hypothesis class is and why every model needs an inductive bias
✓
II — Linear Models
02
Linear Regression, Properly
Phase A gave you the normal equations. This lecture turns them into a model you can actually use — basis functions, feature design, coefficient interpretation, residual diagnostics — and is honest about the four ways a linear fit lies to you.
  • Explain why 'linear' means linear in the parameters, not in the inputs
  • Build design matrices with polynomial, RBF and categorical features
  • Choose between the closed form and an iterative solver from \(n\) and \(d\)
✓
03
Gradient Descent That Actually Works
Every model from here on is fitted by gradient descent, so it is worth doing properly. Convergence rates from the condition number, why stochastic gradients have a noise floor, and where momentum and Adam come from — each derived, not quoted.
  • Derive the convergence rate \(\big(\tfrac{\kappa-1}{\kappa+1}\big)^t\) and say what limits it
  • Explain the SGD noise floor and why every schedule decays the learning rate
  • Derive momentum and explain the \(\kappa \to \sqrt{\kappa}\) improvement
✓
04
Where Loss Functions Come From
You do not choose a loss function. You choose a noise model, and the loss is whatever its negative log-likelihood turns out to be. One principle generates squared error, absolute error, cross-entropy and Poisson loss — and tells you exactly when each is the wrong choice.
  • Turn any noise model into a loss by writing down its negative log-likelihood
  • Derive squared, absolute, Huber, cross-entropy and Poisson losses this way
  • Recognise the exponential family and why every gradient looks like \(\mX^\top(\hat{\vy}-\vy)\)
✓
05
Logistic Regression and Softmax
The workhorse classifier, and the one whose mathematics you must own completely — because the output layer of every classification network is exactly this, and its pathologies are the ones you will meet in every training run.
  • Explain why fitting a linear model to \(\{0,1\}\) labels with squared error is the wrong thing
  • Derive the gradient \(\mX^\top(\hat{\vp}-\vy)\) and the Hessian \(\mX^\top\mathbf{S}\mX\)
  • Prove the loss is convex, and say what that buys you
✓
06
Regularisation: Ridge, LASSO, and Priors
We have reached for a penalty term three times already. Here is the proper treatment: every regulariser is a prior, ridge shrinks eigen-directions by a computable factor, and the corners of the L1 ball are precisely why LASSO produces zeros.
  • Derive ridge and LASSO as MAP estimates under Gaussian and Laplace priors
  • Show ridge shrinks the \(i\)-th eigen-direction by \(\lambda_i/(\lambda_i+\alpha)\)
  • Explain sparsity from the geometry of the \(\ell_1\) ball and from the subgradient
✓
III — Evaluation
07
Bias, Variance, and Honest Evaluation
Why every model's error splits into three parts, why cross-validation is the right tool for choosing between them, and which of the metrics everyone reports are actively misleading.
  • Derive the bias–variance–noise decomposition term by term
  • Explain what \(k\)-fold cross-validation estimates and what it does not
  • Read a learning curve and say whether more data or more capacity is the fix
✓
IV — Unsupervised Learning
08
Principal Component Analysis
Three completely different questions — which direction has most variance, which subspace loses least information, and what does the SVD of the data matrix say — all have the same answer. That coincidence is PCA, and it is the most useful thing the spectral theorem does.
  • Derive PCA as variance maximisation using the Rayleigh quotient
  • Derive it again as minimum reconstruction error, and prove the two agree
  • Connect both to the SVD of the centred data matrix
✓
09
k-means, Gaussian Mixtures, and EM
Clustering without labels. k-means is alternating minimisation of a squared-error objective; Gaussian mixtures are its probabilistic upgrade; and EM — the algorithm that fits them — is one of the most reused ideas in the subject.
  • State the k-means objective and prove Lloyd's algorithm never increases it
  • Explain why k-means finds local minima, and what k-means++ does about it
  • Write down a Gaussian mixture and see why direct likelihood maximisation is hard
✓
V — Generative Models
10
Naive Bayes and the Generative–Discriminative Divide
Instead of modelling p(y|x) directly, model how the data was made and invert with Bayes' theorem. The 'naive' assumption is almost always false, the classifier is often excellent anyway, and understanding why teaches you more about learning than the algorithm itself.
  • Derive the generative decision rule from Bayes' theorem, and see why the evidence term drops out
  • Count the parameters the full joint needs, and what conditional independence buys
  • Derive the MLE for the class prior and the class-conditionals, constraint included
✓
VI — Nonlinear Methods
11
Decision Trees, Random Forests, and Gradient Boosting
Leave the hyperplane behind. A tree carves the input space into boxes; a forest averages many de-correlated trees; boosting fits each new tree to what the ensemble still gets wrong. Two of these are still the best thing to run on tabular data.
  • Derive the optimal leaf value for a fixed partition, and see why finding the best tree is intractable
  • Derive why misclassification error fails as a splitting criterion and Gini and entropy do not
  • Derive \(\Var\!\left(\bar f\right)=\rho\sigma^2+\frac{1-\rho}{B}\sigma^2\) and read random forests off it
✓
12
Support Vector Machines: Margins, Duality, and Kernels
Keep the linear boundary; change the space it lives in. Getting there requires the maximum-margin principle, Lagrangian duality, and the observation that the dual touches the data only through inner products — which is the whole kernel trick in one sentence.
  • Derive the distance from a point to a hyperplane, and turn 'maximise the margin' into a convex QP
  • Derive the soft-margin problem and show it is hinge loss plus \(L_2\) regularisation
  • Derive the Lagrangian dual, the representer form \(\vw=\sum_i\alpha_i y_i\vx_i\), and read the KKT conditions
✓

What comes after

Phase B gives you the algorithms. Phase C asks the uncomfortable question of why any of them should be expected to work on data they have not seen, and Phase D scales the whole apparatus up.

Phase DDeep Learningafter C

What you should still NOT study yet

Your progress is stored locally in this browser only, under a separate key from Phase A. Interactive plots and derivations work offline; typeset mathematics and the in-page Python runner need an internet connection the first time a page loads.