A derivative is a limit of slopes. That single sentence, unpacked carefully, gives you tangent lines, local linear approximation, and the entire mechanism by which a neural network learns.
Here is the problem gradient descent has to solve. You are standing at a point \(w\) on a loss curve \(L(w)\). You want to move downhill. Which way, and how far?
"Which way" is a question about slope. And slope is easy for a straight line — rise over run — but a loss curve is not straight. Its steepness changes from point to point. So we need a notion of slope at a single point, and that phrase is initially meaningless: a slope needs two points.
Zoom in far enough on a smooth curve and it looks straight. Photograph a one-metre stretch of the Earth's surface and you cannot see the curvature. The derivative is the slope of that straight-looking thing — the slope you would measure if you could zoom in infinitely. Limits are the tool that lets us say "infinitely" without hand-waving.
Pick a point \(x\) on the curve \(y=f(x)\), and a second point a distance \(h\) away. Two points determine a line — a secant. Its slope is the ordinary rise-over-run:
\[m_{\text{sec}}(h) = \frac{\text{rise}}{\text{run}} = \frac{f(x+h)-f(x)}{(x+h)-x} = \frac{f(x+h)-f(x)}{h}.\]
This expression is called the difference quotient, and it is the single most important formula in this half of the course. Now shrink \(h\). The second point slides towards the first, and the secant pivots towards the tangent. At \(h=0\) the formula reads \(\tfrac00\) and dies — which is exactly the situation limits were built for.
The derivative of \(f\) at \(x\) is
\[ f'(x) \;=\; \lim_{h\to 0}\frac{f(x+h)-f(x)}{h}, \]provided this limit exists. When it does, \(f\) is said to be differentiable at \(x\).
Equivalent notations, all meaning the same thing: \(\;f'(x)\), \(\;\dfrac{df}{dx}\), \(\;\dfrac{dy}{dx}\), \(\;Df(x)\), \(\;\dot f\) (physics, for time derivatives).
Three readings of the same object, and you should be able to move between them freely:
Notice the last line of step 3: the secant slope is exactly \(2x+h\). So the error between the secant slope and the true derivative is exactly \(h\) — which is what the readout in the figure reports, and which is the \(O(h)\) truncation error we predicted numerically in Lecture 3. The theory and the arithmetic agree, which is the kind of cross-check you should demand of yourself constantly.
Compute \(f'(x)\) from the definition for \(f(x)=\dfrac1x\), with \(x\neq0\).
Hint: the numerator \(\frac{1}{x+h}-\frac1x\) needs a common denominator before anything cancels.
Start with the definition:
\[ f'(x)=\lim_{h\to0}\frac{1}{h}\left(\frac{1}{x+h}-\frac{1}{x}\right). \]Common denominator \(x(x+h)\):
\[ \frac{1}{x+h}-\frac{1}{x} = \frac{x - (x+h)}{x(x+h)} = \frac{-h}{x(x+h)}. \]Divide by \(h\) — i.e. multiply by \(1/h\), and cancel the \(h\) (legal, \(h\neq0\)):
\[ \frac{1}{h}\cdot\frac{-h}{x(x+h)} = \frac{-1}{x(x+h)}. \]Take the limit. The expression is continuous in \(h\) near 0 provided \(x\neq0\):
\[ f'(x) = \frac{-1}{x\cdot x} = -\frac{1}{x^2}. \]Sanity checks. The sign is always negative, matching the fact that \(1/x\) is decreasing on both branches. At \(x=1\), \(f'(1)=-1\); at \(x=2\), \(f'(2)=-1/4\) — the curve flattens as \(x\) grows, which is what the graph shows. And a numerical check at \(x=2\) with \(h=10^{-6}\): \(\frac{1/2.000001 - 1/2}{10^{-6}} \approx -0.2499998\) ✓.
We wrote \(f'(x)=2x\), not "the derivative is 2". The derivative assigns a slope to every point in the domain — it is a new function, built from the old one. This is why you can differentiate again, getting \(f''\), and again.
If \(f\) is differentiable at \(c\), then \(f\) is continuous at \(c\).
Proof. We must show \(\lim_{x\to c}f(x)=f(c)\), equivalently \(\lim_{x\to c}\big[f(x)-f(c)\big]=0\). For \(x\neq c\) write the identity
\[ f(x)-f(c) = \frac{f(x)-f(c)}{x-c}\cdot (x-c), \]which is legitimate because we multiplied and divided by the nonzero quantity \(x-c\). Now take the limit as \(x\to c\). The first factor tends to \(f'(c)\) — that is precisely the hypothesis, in its \(\lim_{x\to c}\frac{f(x)-f(c)}{x-c}\) form. The second factor tends to \(0\). By the product law for limits, which applies because both limits exist,
\[ \lim_{x\to c}\big[f(x)-f(c)\big] = f'(c)\cdot 0 = 0. \qquad\blacksquare\]The converse is false, and the counterexample is one you will meet every day.
\(|x|\) is the shape of ReLU's kink, and this figure is the reason ReLU needs a subgradient convention. At \(x=0\) the set of valid slopes is the whole interval \([-1,1]\) for \(|x|\), or \([0,1]\) for \(\max(0,x)\); frameworks simply pick one (PyTorch picks 0) and carry on. Three consequences worth carrying with you:
Rearrange the definition. For small \(h\),
\[\frac{f(x+h)-f(x)}{h} \approx f'(x) \quad\Longrightarrow\quad \boxed{\,f(x+h) \approx f(x) + f'(x)\,h\,}\]
This is the first-order Taylor approximation, and it is the single most useful consequence of the derivative. It says: to predict how \(f\) responds to a small nudge, multiply the nudge by the derivative.
Now read gradient descent off it directly. We want to decrease \(f\). Choose the step \(h=-\eta f'(x)\) for some small \(\eta>0\). Then
\[f\big(x - \eta f'(x)\big) \approx f(x) + f'(x)\cdot\big(-\eta f'(x)\big) = f(x) - \eta\,\big(f'(x)\big)^2.\]
Since \(\eta>0\) and \((f'(x))^2 \ge 0\), the predicted new value is never larger than the old one, and is strictly smaller whenever \(f'(x)\neq0\). That is the entire justification for the update rule
\[x_{t+1} = x_t - \eta\, f'(x_t).\]
The word "approximately" is load-bearing. The guarantee holds for the linear approximation, which is only accurate for small \(h\). If \(\eta\) is too large the step leaves the region where the approximation is valid and the loss can increase — which is exactly what a diverging training run looks like. Lecture 6 quantifies the error term, and Lecture 7 turns that into an actual bound on \(\eta\).
Compute from the definition (no rules) and simplify fully:
(a) \(f(x)=3x+7\) (b) \(f(x)=x^3\) (c) \(f(x)=\sqrt{x}\), for \(x>0\)
(a) \(\frac{3(x+h)+7-(3x+7)}{h}=\frac{3h}{h}=3\), for every \(h\neq0\). So \(f'(x)=3\). The slope of a line is constant and equals its coefficient — reassuring, and a good check that the definition is behaving.
(b) \((x+h)^3 = x^3+3x^2h+3xh^2+h^3\), so the numerator is \(3x^2h+3xh^2+h^3\). Factor \(h\): \(h(3x^2+3xh+h^2)\). Divide:
\[\frac{f(x+h)-f(x)}{h}=3x^2+3xh+h^2 \xrightarrow{\;h\to0\;} 3x^2.\]So \(f'(x)=3x^2\).
(c) \(\tfrac00\) with a root — use the conjugate, as in Lecture 3:
\[\frac{\sqrt{x+h}-\sqrt x}{h}\cdot\frac{\sqrt{x+h}+\sqrt x}{\sqrt{x+h}+\sqrt x} =\frac{(x+h)-x}{h(\sqrt{x+h}+\sqrt x)}=\frac{h}{h(\sqrt{x+h}+\sqrt x)}=\frac{1}{\sqrt{x+h}+\sqrt x}.\]Let \(h\to0\): \(f'(x)=\dfrac{1}{2\sqrt x}\).
Note this blows up as \(x\to0^+\): \(\sqrt x\) has a vertical tangent at the origin, so it is continuous there but not differentiable — a second way the converse of the theorem fails, different from the corner in \(|x|\).
Let \(f(x)=x^2\). Using only the definition, prove the following exact statement:
\[\frac{f(x+h)-f(x)}{h} - f'(x) = h \quad \text{for every } h \neq 0.\]Then explain what this says about the truncation error of the forward difference, and reconcile it with the \(O(h)\) claim from Lecture 3.
From the derivation in the lecture, the difference quotient is exactly \(2x+h\) for every \(h\neq0\), and \(f'(x)=2x\). Subtracting: \((2x+h)-2x=h\). ∎ (No limit was needed — the identity is exact.)
Interpretation. The forward-difference estimate of \(f'(x)\) overshoots by exactly \(h\). So the truncation error is not merely \(O(h)\), it is precisely \(h\) — the constant is 1.
Reconciling with Lecture 3. The general claim was truncation error \(\approx \frac{f''(x)}{2}h\), which comes from the Taylor expansion \(f(x+h)=f(x)+f'(x)h+\frac{f''(x)}{2}h^2+\cdots\). Here \(f''(x)=2\), so \(\frac{f''}{2}h = h\) ✓, and there are no higher terms because \(x^2\) is exactly quadratic. The general formula reproduces the exact answer, which is a good sign that Lecture 6's machinery is trustworthy.
Define \[ f(x)=\begin{cases} x^2, & x \le 1\\ ax+b, & x>1.\end{cases}\]
Find \(a\) and \(b\) so that \(f\) is differentiable at \(x=1\). Explain why continuity alone is not enough, and sketch what goes wrong if you only impose continuity.
Continuity at 1 requires the two pieces to meet: \(1^2 = a(1)+b\), i.e. \(a+b=1\). That is one equation and two unknowns — a whole family of continuous choices.
Differentiability requires the two one-sided derivatives to agree, exactly as in the \(|x|\) figure. From the left, \(\frac{d}{dx}x^2 = 2x\), so the left slope at 1 is \(2\). From the right, \(\frac{d}{dx}(ax+b)=a\). Agreement forces \(a=2\), and then \(b=1-a=-1\).
So \(a=2,\ b=-1\), giving \(f(x)=2x-1\) for \(x>1\) — which is precisely the tangent line to \(x^2\) at \(x=1\). That is the geometric content of the answer: the only line that can be glued on smoothly is the tangent.
What continuity alone gives you. Any \((a,1-a)\) joins the pieces without a gap, but for \(a\neq2\) the graph has a corner at \(x=1\), just like \(|x|\) at 0. It is continuous, so you could draw it without lifting the pen, but there is no single tangent and \(f'(1)\) does not exist. Differentiability is "continuous and smooth"; continuity is only the first half.
Coding + analysis. Implement gradient descent for \(f(w)=w^4-3w^2+w+4\), whose derivative is \(f'(w)=4w^3-6w+1\).
import numpy as np
f = lambda w: w**4 - 3*w**2 + w + 4
fp = lambda w: 4*w**3 - 6*w + 1
def gd(w0, eta, n=200):
w = w0; hist=[w]
for _ in range(n):
w = w - eta*fp(w)
if not np.isfinite(w) or abs(w) > 1e6: return np.nan, hist
hist.append(w)
return w, hist
for w0 in (1.8, -1.8):
w, _ = gd(w0, 0.05); print(f"w0={w0:+.1f} -> w={w:.6f}, f={f(w):.6f}")
for eta in (0.01, 0.05, 0.10, 0.13, 0.20):
w, h = gd(1.8, eta); print(f"eta={eta:.2f} -> {w}")1. The critical points solve \(4w^3-6w+1=0\), whose roots are \(w\approx-1.30084,\ 0.16994,\ 1.13090\). From \(w_0=1.8\) descent lands at \(w\approx1.13090\) with \(f\approx2.9298\). From \(w_0=-1.8\) it lands at \(w\approx-1.30084\) with \(f\approx0.4861\). (The middle root has \(f''<0\) — it is the local maximum between the two wells, and descent never lands there.)
Same algorithm, same learning rate, different answers — and the left basin is the better one. This is non-convexity: gradient descent finds a local minimum, determined entirely by where you start. Every claim about deep learning optimisation lives with this fact. It is also why initialisation is a research topic and why people run multiple seeds.
2. \(\eta=0.01\) and \(0.05\): smooth convergence. \(\eta=0.10\): converges but with visible oscillation, overshooting the minimum and coming back. \(\eta=0.13\): oscillates without settling, or settles very slowly. \(\eta=0.20\): diverges to overflow within a few dozen steps.
3. \(f''(w)=12w^2-6\). At \(w^\star\approx1.13090\), \(f''\approx 12(1.2789)-6 \approx 9.347\). The quadratic-model threshold is \(\eta_{\text{crit}} = 2/f''(w^\star) \approx 0.214\).
Our sweep started diverging somewhere between \(0.13\) and \(0.20\), a bit earlier than \(0.213\) predicts. Both facts are informative: the order of magnitude is right, confirming the theory is the correct one, and the discrepancy is real because \(f\) is only approximately quadratic near the minimum — far from it, the quartic term dominates and \(f''\) is larger, so the effective threshold along the trajectory is tighter than the one computed at the minimum alone.
The general principle, which you will meet again as "the edge of stability": stable step sizes are governed by the largest curvature encountered, not the curvature at the destination. In many dimensions \(f''\) becomes the Hessian and the threshold becomes \(\eta<2/\lambda_{\max}\) — Lecture 11 and Lecture 12.
Next: computing every derivative from the definition is unsustainable. Lecture 5 derives the rules — sum, product, quotient, and above all the chain rule, which is the mathematical content of backpropagation.