Four rules replace every difference quotient you would otherwise have to compute. The last of them — the chain rule — is not merely convenient: it is the entire mathematical content of backpropagation, and therefore of how every neural network ever trained was trained.
By the end you can
Derive the sum, product, quotient and chain rules from the definition
Prove \(\frac{d}{dx}e^x = e^x\) and \(\frac{d}{dx}\ln x = \frac1x\), closing the debt from Lecture 2
Differentiate a long composition by peeling one layer at a time
Derive \(\sigma' = \sigma(1-\sigma)\) and see why it explains vanishing gradients
Implement forward-mode automatic differentiation in twenty lines of NumPy
Assumed
Lecture 4: the derivative as a limit of difference quotients
Lecture 3: limit laws for sums, products and quotients
1Linearity, and the power rule
The two easiest rules follow immediately from the corresponding limit laws, because the
difference quotient is linear in \(f\).
Theorem — linearity
If \(f\) and \(g\) are differentiable at \(x\) and
\(\alpha,\beta\) are constants, then
The \(k=0\) term cancels. As always, the constant term of the expansion is the thing we subtract. Every surviving term has at least one \(h\).
\[\frac{(x+h)^n-x^n}{h} = n x^{n-1} + \underbrace{\binom{n}{2}x^{n-2}h + \cdots + h^{n-1}}_{\text{every term contains } h}\]
Divide by \(h\). Legal for \(h\neq0\). The \(k=1\) term loses its \(h\) entirely and becomes the constant \(nx^{n-1}\); every term with \(k\ge2\) retains at least one factor of \(h\).
\[\frac{d}{dx}x^n = \lim_{h\to0}\left[n x^{n-1} + O(h)\right] = n x^{n-1}\]
Take the limit. The bracketed tail is a polynomial in \(h\) with no constant term, so it vanishes. Check against Lecture 4: \(n=2\) gives \(2x\) ✓, \(n=3\) gives \(3x^2\) ✓, \(n=1\) gives \(1\) ✓. The rule in fact holds for every real \(n\) — negative, fractional, irrational — but proving that needs the chain rule and the log derivative, both below.
2The product rule, and why it is not what you would guess
The natural guess is \((fg)' = f'g'\). It is wrong, and one example kills it: take
\(f=g=x\). Then \(fg = x^2\) with derivative \(2x\), while \(f'g' = 1\cdot1 = 1\). Not equal for any
\(x\neq\tfrac12\).
The correct rule has a picture behind it. Think of \(f(x)\) and \(g(x)\) as the side lengths of a
rectangle, so \(f(x)g(x)\) is its area. Nudge \(x\) by \(h\): both sides grow a little, and the area grows by
three pieces.
The product rule as growing area: Δ(fg) = f·Δg + g·Δf + Δf·ΔgInteractive
original area f·gg·Δff·ΔgΔf·Δg (negligible)
Blue is the original rectangle. Orange and purple are the two first-order strips. Pink is the tiny corner square. Shrink h and watch the corner vanish far faster than the strips — it is proportional to h², while the strips are proportional to h. Divide everything by h and the corner still carries a factor of h, so it dies in the limit. That is the whole proof, in a picture.
The product rule, proved0 / 4
\[\frac{f(x+h)g(x+h) - f(x)g(x)}{h}\]
The difference quotient of the product. The obstacle is that both factors changed at once, so nothing cancels directly.
\[= \frac{f(x+h)g(x+h) \;\underbrace{-\, f(x+h)g(x) \,+\, f(x+h)g(x)}_{\text{add and subtract}}\; -\, f(x)g(x)}{h}\]
The add-and-subtract trick. We insert a hybrid term \(f(x+h)g(x)\) — new \(f\), old \(g\). This changes nothing (we added zero) but lets us split the total change into 'change due to \(g\)' and 'change due to \(f\)'. This is the algebraic version of splitting the rectangle into strips.
Group and factor. The first two terms share \(f(x+h)\); the last two share \(g(x)\). Each bracket is now a difference quotient of a single function — the shapes we know how to take limits of.
\[(fg)'(x) = f(x)g'(x) + g(x)f'(x)\]
Take the limit. \(\frac{g(x+h)-g(x)}{h}\to g'(x)\) and \(\frac{f(x+h)-f(x)}{h}\to f'(x)\) by hypothesis. Crucially \(f(x+h)\to f(x)\) — and that is not automatic: it needs \(f\) to be continuous at \(x\), which we get free from Lecture 4's theorem, since \(f\) is differentiable there. ∎ Check: \(f=g=x\) gives \(x\cdot1+x\cdot1=2x\) ✓, matching \(\frac{d}{dx}x^2\).
Check yourself
Differentiate \(y = x^2 e^x\). (Take \(\frac{d}{dx}e^x = e^x\) as known — we prove it below.)
Correct. Correct. Product rule with \(f=x^2\), \(g=e^x\): \(f'g + fg' = 2x\,e^x + x^2 e^x\). Factoring
\(xe^x\) gives \(xe^x(2+x)\), which is worth doing because it exposes the roots: \(y'=0\) at \(x=0\) and
\(x=-2\). Since \(e^x>0\) always, the sign of \(y'\) is the sign of \(x(2+x)\).
Not quite. This differentiates only the first factor and leaves the second alone. The product rule has
two terms because both factors are changing simultaneously — that is exactly what the two rectangle
strips represent. Only if \(g\) were a constant would the second term vanish.
Not quite. This is the tempting \((fg)'=f'g'\), disproved at the top of this section. Test it on
\(f=g=x\): it would give \(1\), but \(\frac{d}{dx}x^2=2x\). Coincidentally \(2xe^x\) is also what the
previous wrong option gives, but for a different wrong reason.
3The quotient rule
You can derive it the same way, but it is faster to get it from the product rule once we have the
chain rule, so here it is stated and then re-derived below as a corollary.
The order of the numerator matters: it is low d-high minus high d-low, over low squared. Getting the
sign backwards is the single most common slip in this course, and the check is easy — try \(f=1,\ g=x\), which
must give \(-1/x^2\) to match Lecture 4.
4The chain rule
Now the important one. Recall from Lecture 1 that a neural network is a composition
\(f_L\circ\cdots\circ f_1\). To train it we need the derivative of a composition.
Intuition
Rates multiply. If a car travels 60 km per hour and burns 8 litres per 100 km, then it
burns \(60 \times 0.08 = 4.8\) litres per hour. The units tell you to multiply:
\(\frac{\text{km}}{\text{hr}}\times\frac{\text{L}}{\text{km}} = \frac{\text{L}}{\text{hr}}\), with kilometres
cancelling. The chain rule is that cancellation, made rigorous:
\(\frac{dy}{dx} = \frac{dy}{du}\cdot\frac{du}{dx}\).
The chain rule, proved0 / 6
\[y = g(u), \quad u = f(x) \quad\Longrightarrow\quad y = (g\circ f)(x) = g(f(x))\]
Set up. \(f\) is differentiable at \(x\); \(g\) is differentiable at \(u_0 = f(x)\). We want \((g\circ f)'(x)\).
The idea in one line. Multiply and divide by \(\Delta u = f(x+h)-f(x)\). The first factor is a difference quotient for \(g\) (with increment \(\Delta u\)), the second is one for \(f\). But this step is illegal whenever \(\Delta u = 0\), and \(\Delta u\) can be zero for arbitrarily small \(h\) — think of a constant \(f\). Step 4 repairs it.
Why the first factor converges. \(f\) is differentiable at \(x\), hence continuous there, so \(\Delta u\to0\) as \(h\to0\). The inner quotient is then \(g\)'s difference quotient with a vanishing increment, which tends to \(g'(u_0)\). Note we used the differentiable ⟹ continuous theorem again.
\[E(\Delta u) = \begin{cases}\dfrac{g(u_0+\Delta u)-g(u_0)}{\Delta u} & \Delta u \neq 0\\[6pt] g'(u_0) & \Delta u = 0\end{cases}\]
Patching the hole. Define \(E\) to equal the quotient where that makes sense, and to equal \(g'(u_0)\) at the bad point. By step 3, \(E\) is continuous at \(0\). And the identity \(g(u_0+\Delta u)-g(u_0) = E(\Delta u)\cdot\Delta u\) now holds for every \(\Delta u\) including zero, where both sides are 0. This is exactly the removable-discontinuity repair from Lecture 3, used to make a proof work.
Rewrite legally. Substituting the identity from step 4 — valid for all \(h\), with no division by \(\Delta u\) anywhere.
\[(g\circ f)'(x) = g'(f(x))\cdot f'(x)\]
Take the limit. \(E(\Delta u)\to E(0)=g'(u_0)=g'(f(x))\) by continuity of \(E\), and \(\Delta u/h \to f'(x)\) by definition. Product of limits. ∎ Read the final formula carefully: \(g'\) is evaluated at \(f(x)\), not at \(x\). That is the most common error in applying it.
Chain rule as multiplied rates: y = sin(u), u = x²Interactive
u = f(x) = x²y = g(u) = sin uy = g(f(x)) = sin(x²)
Top-left: the inner map, with its local slope f′(x) = 2x drawn as a tangent. Top-right: the outer map, with its slope g′(u) = cos u drawn at the point u = f(x). Bottom: the composite, whose slope is the product of the two. Move x and watch all three tangents update together. Where either factor is zero, the composite is momentarily flat.
Practice peeling
Differentiate, naming the outer and inner functions at each layer:
This is the unnormalised Gaussian density, and its derivative being \(-x\) times itself is exactly why the
Gaussian is the fixed point of so many arguments. Note \(y'=0\) only at \(x=0\), the peak.
The last step divides top and bottom by \(e^x\). So the derivative of the softplus function is
exactly the sigmoid — softplus is a smooth approximation to ReLU, and this identity is why.
The chain rule composes: for \(L\) layers you multiply \(L\) factors. That is literally the shape of
backpropagation through an \(L\)-layer network.
5Paying the Lecture 2 debt: the derivative of eˣ and ln x
We left two things unproved. Here they are.
d/dx ln x = 1/x, and hence C(a) = ln a0 / 5
\[y = \ln x \quad\Longleftrightarrow\quad e^{y} = x\]
Use the inverse relationship. Rather than attack \(\ln\) directly, convert to the exponential form — the move from Lecture 2. We take \(\frac{d}{dx}e^x=e^x\) as the definition of \(e\) (Lecture 2 established that some base makes \(C(a)=1\); call it \(e\)).
Differentiate both sides — implicitly. The left side is a composition: outer \(e^{(\cdot)}\), inner \(y(x)\). The chain rule gives \(e^y \cdot y'\). The right side differentiates to 1. This technique, differentiating an equation rather than a formula, is called implicit differentiation.
Solve and substitute back. \(e^y>0\) so the division is legal, and \(e^y = x\) by step 1. Hence \(\frac{d}{dx}\ln x = \frac1x\), valid for \(x>0\) — the domain of \(\ln\), as always.
Now the general base. The identity \(a = e^{\ln a}\) is just \(\exp\) and \(\ln\) undoing each other; raising to the \(x\) gives \(a^x = e^{x\ln a}\). Chain rule: outer \(e^u\), inner \(u=x\ln a\) with \(u' = \ln a\). So \(\frac{d}{dx}a^x = a^x \ln a\).
The debt is paid. \(C(a)\) was precisely the derivative of \(a^x\) at \(x=0\). So \(C(a)=\ln a\), and \(C(a)=1\) exactly when \(\ln a = 1\), i.e. \(a=e\). Numerical check against Lecture 2's interactive: at \(a=2\), \(C(2)=\ln 2 = 0.6931\) — and the finite-difference readout there showed \(0.69315\) ✓.
6The sigmoid derivative, and the first hint of vanishing gradients
Chain rule twice. Outer layer: \(\frac{d}{du}u^{-1}=-u^{-2}\). Inner layer: \(\frac{d}{dz}e^{-z} = e^{-z}\cdot(-1)\), itself a chain rule with inner \(-z\). Two minus signs are about to cancel.
Tidy up. The two negatives give a positive, confirming what the graph shows: \(\sigma\) is strictly increasing everywhere, so its derivative is positive everywhere.
The key regrouping. Split the fraction into two. The first factor is \(\sigma(z)\). For the second, recall from Lecture 2's exercise that \(1-\sigma(z) = \frac{e^{-z}}{1+e^{-z}}\) ✓. So the derivative is expressible entirely in terms of the function's own output — no exponentials needed. ∎
Practical engineering note
That last form is why every framework caches the output of a sigmoid layer during the
forward pass. To get the gradient you need only s * (1 - s): no exponentials, no recomputation.
The same trick appears for tanh (\(\tanh' = 1-\tanh^2\)) and softmax.
σ and σ′ — why deep sigmoid networks stopped trainingInteractive
σ(z)σ′(z) = σ(1−σ)product over L layers
σ′ peaks at 0.25 when z = 0 and collapses towards zero in both tails. The chain rule multiplies one such factor per layer. Raise the depth slider and read the product: even in the best case, 0.25 to the power L shrinks geometrically. At L = 10 the best possible gradient scale is about 10⁻⁶. This is the vanishing-gradient problem, and it is pure chain rule.
Where this shows up in ML
This figure is a piece of history. Networks deeper than a few layers were, for years,
untrainable with sigmoid activations, and the reason is one line of chain rule: the gradient reaching layer 1 is
a product of \(L\) factors each at most \(0.25\). Three of the ideas that unblocked deep learning attack exactly
this product:
ReLU has derivative exactly 1 on the positive side. Multiply as many ones as you like and
nothing shrinks.
Residual connections \(\vx + F(\vx)\) have derivative \(1 + F'\), so the product contains a
path of pure ones — the gradient has a route home that no layer can attenuate.
Normalisation layers keep pre-activations near zero, where \(\sigma'\) is largest, rather
than out in the saturated tails where it is essentially zero.
You are not expected to fully absorb these now. The point is that you can already see why they were
needed, from a rule you just proved.
7Automatic differentiation in twenty lines
The chain rule is mechanical, which means a computer can apply it. Here is the simplest version:
carry the value and the derivative together through every operation, applying the rule at each step. This is
forward-mode automatic differentiation, and it is exact — not a finite-difference
approximation.
Dual numbers: exact derivatives with no limits and no symbolsPython · NumPy
8Exercises
★
Differentiate, showing which rule you used at each step:
(c) Product rule, with a chain rule inside each factor:
\(\frac{d}{dx}e^{3x}=3e^{3x}\) and \(\frac{d}{dx}\sin(2x)=2\cos(2x)\). So
\[y' = 3e^{3x}\sin(2x) + 2e^{3x}\cos(2x) = e^{3x}\big(3\sin 2x + 2\cos 2x\big).\]
(d) Chain rule, outer \(\ln u\), inner \(u=x^2+4\):
\(y' = \frac{1}{x^2+4}\cdot 2x = \frac{2x}{x^2+4}\). Note the domain is all of \(\RR\) here because
\(x^2+4>0\) always — worth checking, since \(\ln\) is fussy.
★★
Derive the quotient rule from the product and chain rules, by writing
\(\dfrac{f}{g} = f\cdot g^{-1}\).
Then verify your result reproduces \(\frac{d}{dx}\frac1x = -\frac{1}{x^2}\).
First, \(\frac{d}{dx}g^{-1}\) by the chain rule: outer \(u^{-1}\) with derivative \(-u^{-2}\),
inner \(g\) with derivative \(g'\). So \(\frac{d}{dx}g^{-1} = -g^{-2}g'\).
Check. \(f=1,\ g=x\): \(f'=0,\ g'=1\), giving \(\frac{0\cdot x - 1\cdot 1}{x^2} =
-\frac{1}{x^2}\) ✓, matching the first-principles computation in Lecture 4.
This is worth internalising as a strategy: the quotient rule is not a fifth independent fact, it is a
consequence. Fewer things to remember, and you can always rebuild it.
★★
Prove the power rule \(\frac{d}{dx}x^{r} = r x^{r-1}\) for any real \(r\) and
\(x>0\), using \(x^r = e^{r\ln x}\).
Why did we need \(x>0\), and what is the corresponding statement for negative \(x\) when \(r\) is an
integer?
Write \(x^r = e^{r\ln x}\) — valid for \(x>0\), because that is the domain of \(\ln\).
Chain rule: outer \(e^u\), inner \(u = r\ln x\) with \(u' = r/x\).
Why \(x>0\). The identity \(x^r=e^{r\ln x}\) requires \(\ln x\) to exist. More
fundamentally, \(x^r\) for irrational \(r\) is only defined for \(x>0\) — recall Lecture 2, Stage 3:
\((-8)^{1/2}\) has no real value. So the restriction is not an artefact of the proof, it is inherited from the
function itself.
Negative \(x\), integer \(r\). Then \(x^r\) is perfectly well defined, and the rule still
holds — but it needs a different proof. For positive integers, the binomial argument at the top of this lecture
works for any real \(x\), positive or negative. For negative integers \(r=-n\), write \(x^{-n} = 1/x^n\) and
apply the quotient rule: \(\frac{0\cdot x^n - 1\cdot nx^{n-1}}{x^{2n}} = -n x^{-n-1}\) ✓. Three proofs for
three overlapping domains — a normal state of affairs in analysis.
★★★
The vanishing gradient, quantified. Consider the scalar 'network'
\[ y = \sigma(w_L \cdots \sigma(w_2\,\sigma(w_1 x))\cdots) \]
with \(L\) layers, each a weight followed by a sigmoid.
Write \(\frac{\partial y}{\partial w_1}\) as a product, using the chain rule.
Bound its magnitude, given that \(0<\sigma'\le\tfrac14\).
What must the weights satisfy for the gradient not to vanish? Comment on whether that condition
is compatible with the activations staying in the high-derivative region.
Extend the Dual class above to verify your bound numerically for \(L=10\).
1. Write \(a_0 = x\), \(z_\ell = w_\ell a_{\ell-1}\), \(a_\ell = \sigma(z_\ell)\),
\(y=a_L\). Peeling one layer at a time:
Each layer contributes exactly two factors: its activation derivative and its weight.
2. Since \(0<\sigma'(z)\le\frac14\) for all \(z\),
\[\left|\frac{\partial y}{\partial w_1}\right| \le |x|\cdot\prod_{\ell=1}^{L}\frac{|w_\ell|}{4}
= |x|\left(\frac{|w|}{4}\right)^{L} \quad\text{if all }|w_\ell|=|w|.\]
3. The product decays geometrically unless \(|w|\gtrsim 4\). With standard
initialisation \(|w|\approx 1\), the factor is \(\approx (1/4)^L\): at \(L=10\) that is \(10^{-6}\), at
\(L=20\) it is \(10^{-12}\) — below float32 resolution relative to typical weight magnitudes. The gradient
is not small, it is gone.
And the trap. Raising \(|w|\) to 4 or above does not rescue you, because \(\sigma'\)
reaches \(\tfrac14\) only at \(z=0\). Large weights drive \(z_\ell = w_\ell a_{\ell-1}\) into the saturated
tails, where \(\sigma'\) collapses towards zero far faster than \(|w|\) grows. The two requirements —
large weights, and pre-activations near zero — are in direct conflict. There is no setting of a plain
sigmoid stack that keeps the product near 1 at depth, which is precisely why the fix had to come from changing
the architecture (ReLU, residual connections) rather than from tuning.
import numpy as np
def sig(x): return 1/(1+exp(-x)) # uses the Dual exp above
def net(w1, ws, x0):
def f(w):
a = sig(w * x0)
for wl in ws: a = sig(Dual(wl) * a)
return a
return derivative(f, w1)
rng = np.random.default_rng(0)
for L in (1, 3, 5, 10, 20):
ws = rng.normal(0, 1, size=L-1)
g = net(1.0, ws, 1.0)
print(f"L={L:3d} dy/dw1 = {g:.3e} bound (1/4)^L = {0.25**L:.3e}")
You should see the measured gradient fall below the bound at every depth (the bound is loose because
\(\sigma'\) is well under \(\tfrac14\) once activations drift from zero) and reach \(10^{-8}\) or smaller by
\(L=10\).
Next: the linear approximation \(f(x+h)\approx f(x)+f'(x)h\) has an error we
have so far ignored. Lecture 6 pins it down exactly with Taylor's theorem — and in doing so explains why
second-order optimisers exist, and where the learning-rate bound \(\eta<2/f''\) comes from.