The tangent line is the best straight-line approximation to a curve. Taylor's theorem says what the best parabola is, the best cubic, and exactly how wrong each of them is — which is where second-order optimisers, learning-rate bounds, and half of numerical analysis come from.
Lecture 4 gave us \(f(x+h)\approx f(x)+f'(x)h\), and we used it to justify gradient descent while noting the word "approximately" was carrying weight. Now we make the error precise.
The tangent line agrees with \(f\) in two respects at the base point: same value, same slope. It disagrees in a third: curvature. A line has zero second derivative; a curve generally does not. So the natural repair is to add a quadratic term chosen to match the second derivative too. Then a cubic term to match the third. And so on.
A Taylor polynomial is a forgery. You are asked to fake a function using only polynomials, given complete information at a single point. All you can copy is the derivatives there — the value, the slope, the curvature, the rate of change of curvature. The remarkable fact is that for many functions, copying enough derivatives at one point reproduces the function over a whole interval.
Matching derivatives gives a polynomial that looks right near \(a\). Taylor's theorem turns "looks right" into an exact statement with a computable bound.
Suppose \(f\) is \((n+1)\)-times differentiable on an interval containing \(a\) and \(x\). Then there exists some point \(\xi\) strictly between \(a\) and \(x\) such that
\[ f(x) \;=\; \underbrace{\sum_{k=0}^{n}\frac{f^{(k)}(a)}{k!}(x-a)^k}_{P_n(x)} \;+\; \underbrace{\frac{f^{(n+1)}(\xi)}{(n+1)!}(x-a)^{n+1}}_{R_n(x)}. \]Three observations, in increasing order of usefulness:
The proof is a clever application of Rolle's theorem to an auxiliary function, and it is about half a page. I am deferring it rather than skipping it: it adds no conceptual content you need right now, and the statement is what does the work in everything below. Ask and we will do it properly.
Estimate \(\sqrt{e} = e^{0.5}\) using \(P_3\) about \(a=0\), and give a rigorous bound on the error using the Lagrange remainder. Then check your bound against the true error.
Hint: every derivative of \(e^x\) is \(e^x\), and on \([0,0.5]\) you can bound \(e^\xi\) by something concrete without knowing \(\xi\).
The polynomial. All derivatives of \(e^x\) at 0 equal 1, so \(P_3(x)=1+x+\frac{x^2}{2}+\frac{x^3}{6}\). At \(x=0.5\):
\[P_3(0.5)=1+0.5+0.125+0.0208\overline{3}=1.6458\overline{3}.\]The bound. The remainder is \(R_3(0.5)=\frac{e^{\xi}}{4!}(0.5)^4\) for some \(\xi\in(0,0.5)\). We do not know \(\xi\), but \(e^x\) is increasing, so \(e^\xi < e^{0.5}\). And \(e^{0.5}<e<3\), a bound we can assert without circularity. Hence
\[|R_3(0.5)| < \frac{3}{24}\cdot 0.0625 = 0.0078125.\]Check. The true value is \(\sqrt e = 1.6487212707\ldots\), so the actual error is \(1.6487213-1.6458333=0.0028879\). Our bound said "less than \(0.0078\)" ✓ — correct, and about a factor of 2.7 loose, which is the price of replacing \(e^\xi\) by its worst case.
That looseness is normal and acceptable. The value of the bound is that it is guaranteed: it holds without ever computing \(\sqrt e\). This is exactly how numerical libraries decide how many terms to use.
Each is derived the same way: compute derivatives at 0, divide by factorials.
\[\begin{aligned} e^{x} &= 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots = \sum_{k=0}^{\infty}\frac{x^k}{k!} && \text{for all } x\in\RR \\[4pt] \sin x &= x - \frac{x^3}{3!} + \frac{x^5}{5!} - \cdots = \sum_{k=0}^{\infty}\frac{(-1)^k x^{2k+1}}{(2k+1)!} && \text{for all } x\in\RR \\[4pt] \ln(1+x) &= x - \frac{x^2}{2} + \frac{x^3}{3} - \cdots = \sum_{k=1}^{\infty}\frac{(-1)^{k+1}x^k}{k} && \text{for } -1 < x \le 1 \\[4pt] \frac{1}{1-x} &= 1 + x + x^2 + x^3 + \cdots = \sum_{k=0}^{\infty}x^k && \text{for } |x| < 1 \end{aligned}\]
The domain restrictions on the last two are not decorative. The geometric series \(\sum x^k\) diverges for \(|x|\ge1\) — try \(x=2\) and you get \(1+2+4+8+\cdots\), which is not \(1/(1-2)=-1\). The radius of convergence is determined by how far the nearest singularity is from the expansion point: \(1/(1-x)\) blows up at \(x=1\), which sits at distance 1 from \(a=0\), and that distance is exactly the radius. You saw this in the figure above: raising the degree improved things inside the radius and made them dramatically worse outside it.
log1p exists in every maths library: computing
\(\ln(1+x)\) as log(1+x) loses precision when \(x\) is tiny, because \(1+x\) rounds to 1.Now the payoff for optimisation. Take the Taylor expansion of a loss \(f\) about the current point \(w\), keeping two terms:
\[f(w+h) \;\approx\; f(w) + f'(w)\,h + \tfrac12 f''(w)\,h^2 \;=:\; q(h).\]
This is a parabola in \(h\), and we know how to minimise a parabola exactly.
In one dimension Newton needs one number, \(f''\). In \(d\) dimensions it needs the Hessian: a \(d\times d\) matrix of second partial derivatives, which must then be inverted. For a model with \(d=10^9\) parameters that is \(10^{18}\) entries — not storable, let alone invertible. That, plus the negative-curvature failure you can see in the figure, is why deep learning uses first-order methods.
But the idea survives in disguise. Adam, RMSProp and their relatives maintain a running estimate of the diagonal of something Hessian-like and divide the gradient by it. They are cheap approximations to Newton, and the family name for such methods is quasi-Newton. You will build them in Phase D, and you now know what they are approximating.
We can also now derive the bound that Lecture 4's exercise had you discover numerically. Take \(f\) with \(f''(w) = L\) roughly constant near the minimum \(w^\star\) (i.e. \(f\) is locally quadratic). Gradient descent gives
\[\begin{aligned} w_{t+1}-w^\star &= w_t - \eta f'(w_t) - w^\star \\ &= (w_t - w^\star) - \eta\,L\,(w_t-w^\star) \qquad \text{since } f'(w)\approx L(w-w^\star)\\ &= (1-\eta L)\,(w_t-w^\star). \end{aligned}\]
So the distance to the optimum is multiplied by \(|1-\eta L|\) every step. Convergence requires that factor to be less than 1:
\[|1-\eta L| < 1 \quad\Longleftrightarrow\quad -1 < 1-\eta L < 1 \quad\Longleftrightarrow\quad 0 < \eta < \frac{2}{L}.\]
In many dimensions \(L\) becomes the largest eigenvalue of the Hessian, \(\lambda_{\max}\), and the bound becomes \(\eta<2/\lambda_{\max}\). We prove that in Lecture 11, once we have eigenvalues. The one-dimensional case here is the whole idea, with the linear algebra stripped away.
Find \(P_3(x)\) about \(a=0\) for each, by computing derivatives:
(a) \(f(x)=\cos x\) (b) \(f(x)=\sqrt{1+x}\) (c) \(f(x)=\sigma(x)=\frac{1}{1+e^{-x}}\)
(a) \(f=\cos,\ f'=-\sin,\ f''=-\cos,\ f'''=\sin\). At 0: \(1,0,-1,0\). So \(P_3(x)=1-\frac{x^2}{2}\). (The cubic coefficient is zero — \(\cos\) is even, so all odd terms vanish.)
(b) \(f=(1+x)^{1/2}\), \(f'=\frac12(1+x)^{-1/2}\), \(f''=-\frac14(1+x)^{-3/2}\), \(f'''=\frac38(1+x)^{-5/2}\). At 0: \(1,\ \frac12,\ -\frac14,\ \frac38\). Divide by factorials:
\[P_3(x)=1+\frac{x}{2}-\frac{x^2}{8}+\frac{x^3}{16}.\]Check at \(x=0.1\): \(P_3=1.0488125\) versus \(\sqrt{1.1}=1.0488088\) — four correct digits.
(c) Use \(\sigma'=\sigma(1-\sigma)\) from Lecture 5. At \(x=0\), \(\sigma=\frac12\), so \(\sigma'=\frac14\). Differentiating again: \(\sigma''=\sigma'(1-2\sigma)\), which at 0 is \(\frac14(1-1)=0\). And \(\sigma'''=\sigma''(1-2\sigma)-2(\sigma')^2\), which at 0 is \(0-2\cdot\frac1{16}=-\frac18\). So
\[P_3(x)=\frac12+\frac{x}{4}-\frac{x^3}{48}.\]The vanishing quadratic term reflects the fact that \(\sigma(x)-\frac12\) is an odd function.
Use the Lagrange remainder to determine how many terms of the series for \(e^x\) are needed to compute \(e^{1}\) with error below \(10^{-10}\).
Then verify your answer by direct computation.
With \(a=0\), \(x=1\), and \(f^{(n+1)}(\xi)=e^{\xi}\) for some \(\xi\in(0,1)\):
\[|R_n(1)| = \frac{e^{\xi}}{(n+1)!}\cdot 1^{n+1} < \frac{3}{(n+1)!},\]using \(e^\xi < e < 3\). We need \(\frac{3}{(n+1)!} < 10^{-10}\), i.e. \((n+1)! > 3\times10^{10}\).
Factorials: \(13!=6.227\times10^{9}\) (too small), \(14!=8.718\times10^{10}\) ✓. So \(n+1=14\), giving \(n=13\) — that is terms \(k=0\) through \(13\), fourteen terms.
import numpy as np, math
for n in (11, 12, 13, 14):
s = sum(1.0/math.factorial(k) for k in range(n+1))
print(n, f"{s:.15f}", f"err {abs(s-np.e):.2e}")You will find \(n=13\) gives error \(1.2\times10^{-11}\) — comfortably inside the target, and the bound correctly predicted it. \(n=12\) gives \(1.7\times10^{-10}\), just outside. So the bound identified the right cutoff exactly. It is loose by about a factor of 8 on the error itself, which is the price of replacing \(e^\xi\) by 3.
Show that for gradient descent on the exact quadratic \(f(w)=\tfrac{L}{2}(w-w^\star)^2\), the error after \(t\) steps is exactly
\[ w_t - w^\star = (1-\eta L)^{t}\,(w_0-w^\star). \]Then answer: with \(L=10\), how many steps are needed to reduce the error by \(10^{6}\), at (i) \(\eta=0.01\), (ii) \(\eta=0.1\), (iii) \(\eta=0.19\)? Comment on the pattern.
Derivation. \(f'(w)=L(w-w^\star)\), so
\[w_{t+1}-w^\star = w_t - \eta L(w_t-w^\star) - w^\star = (1-\eta L)(w_t-w^\star).\]Each step multiplies the error by the same constant \(r=1-\eta L\), so by induction \(w_t-w^\star = r^t(w_0-w^\star)\). ∎ (For an exact quadratic there is no approximation: the relation is an identity, not a local model.)
Step counts. Need \(|r|^t \le 10^{-6}\), i.e. \(t \ge \frac{-6\ln 10}{\ln|r|}\).
The pattern. Convergence speed is governed by \(|1-\eta L|\), which is a V-shape in \(\eta\) with its minimum (zero) at \(\eta=1/L\) and roots of the boundary at \(\eta=0\) and \(\eta=2/L\). Speed is symmetric about \(\eta=1/L\): being 90% too small and 90% too large cost exactly the same. This is why learning-rate tuning has a sweet spot rather than a "bigger is faster" regime, and why the standard advice is to increase \(\eta\) until training becomes unstable and then back off — you are searching for \(2/\lambda_{\max}\) empirically.
Coding + analysis. Newton's method for root-finding solves \(g(w)=0\) via \(w_{t+1}=w_t - g(w_t)/g'(w_t)\). Applying it to \(g=f'\) gives the optimiser above.
import numpy as np
g, gp = lambda w: w*w - 2, lambda w: 2*w
w, root = 1.0, np.sqrt(2)
for t in range(6):
e = abs(w - root)
print(f"t={t} w={w:.18f} err={e:.3e}")
w = w - g(w)/gp(w)1. The errors run \(4.14\times10^{-1},\ 8.58\times10^{-2},\ 2.45\times10^{-3},\ 2.12\times10^{-6},\ 1.59\times10^{-12},\ 0\). The number of correct digits doubles each iteration — from 0 to 1 to 2 to 5 to 11 to full machine precision. Five iterations from a poor starting guess.
2. \(C \approx e_{t+1}/e_t^2\). The successive ratios are \(0.500,\ 0.333,\ 0.353,\ 0.354\dots\) — settling on about \(0.354\). Theory: at \(w=\sqrt2\), \(g''=2\) and \(g'=2\sqrt2\), so \(C=\frac{2}{2\cdot 2\sqrt2}=\frac{1}{2\sqrt2}=0.35355\) ✓. The first ratio is off because the early iterates are still far from the root, where the quadratic model of the error is not yet accurate.
3. \(f'(w)=w^3-2w+0.3\) and \(f''(w)=3w^2-2\). At \(w_0=0.1\), \(f''(0.1)=0.03-2=-1.97\), which is negative. The quadratic model \(q(h)=f+f'h+\frac12 f''h^2\) therefore opens downwards: its critical point is a maximum, not a minimum. Newton dutifully jumps to it, moving uphill.
Concretely, \(f'(0.1)=0.001-0.2+0.3=0.101\), so the step is \(-0.101/(-1.97)=+0.0513\) — a step in the positive gradient direction, i.e. increasing \(f\). From there behaviour is erratic: the iterates wander in the negative-curvature region, and where \(f''\) passes through zero (at \(w=\pm\sqrt{2/3}\approx \pm0.8165\)) the step size blows up.
The general lesson. Newton finds critical points, not minima. It is attracted to maxima and saddle points just as readily. In high dimensions almost all critical points of a deep network's loss are saddles, which makes raw Newton actively harmful. Practical second-order methods therefore modify the curvature — trust regions, damping \((\mathbf{H}+\lambda \mI)^{-1}\), or Gauss–Newton approximations that are positive semi-definite by construction. Every one of those is a fix for the failure you just observed.
Next: Lecture 7 makes optimisation in one dimension complete and rigorous — critical points, the second-derivative test, convexity, and the guarantee that separates a problem you can solve from one you can only attempt.