A limit answers a question the function itself refuses to: what value is this heading towards at a point where it may not even be defined? Every derivative, every integral, and every convergence guarantee in optimisation is built on this one idea.
Consider
\[f(x) = \frac{x^2-9}{x-3}.\]
By the domain rules of Lecture 1, \(x=3\) is excluded — the denominator vanishes. The function genuinely does not exist there. And yet, everywhere near 3, the function is doing something very definite. Factor the numerator:
\[\frac{x^2-9}{x-3} = \frac{(x-3)(x+3)}{x-3} = x+3 \qquad \textbf{provided } x \neq 3.\]
That proviso is not a technicality — cancelling \((x-3)\) requires dividing by it, which requires it to be nonzero. So \(f\) is the line \(y=x+3\) with a single point punctured out at \(x=3\). As \(x\) approaches 3 from either side, \(f(x)\) approaches 6. It never gets there, because there is nothing there. But 6 is unmistakably the value it is heading for.
A limit is a statement about the neighbourhood of a point, deliberately excluding the point itself. "\(\lim_{x\to c}f(x)=L\)" says: I can force \(f(x)\) to be as close to \(L\) as you like, by keeping \(x\) close enough to \(c\) — and I never need to evaluate \(f\) at \(c\) to do it. That last clause is the whole point. It is what lets us define the derivative, which is a ratio that is \(\tfrac00\) at the one point we care about.
The informal version — "gets arbitrarily close" — is good intuition and useless for proofs, because "close" is not a number. The fix, due to Cauchy and Weierstrass, replaces both vague words with quantified numbers.
Let \(f\) be defined on an open interval around \(c\), except possibly at \(c\) itself. We say
\[\lim_{x\to c} f(x) = L\]if: for every \(\varepsilon>0\), there exists \(\delta>0\) such that
\[ 0 < |x-c| < \delta \quad\Longrightarrow\quad |f(x)-L| < \varepsilon. \]Read it as a game between two players. Your opponent names a tolerance \(\varepsilon\) — how close to \(L\) they demand the output be. You must respond with a radius \(\delta\) — how close to \(c\) you promise to keep the input. You win if your promise always delivers. The limit exists iff you have a winning response to every \(\varepsilon\), no matter how small.
Two details that carry the mathematical weight:
Why does \(\delta=\min(1,\varepsilon/5)\) always work for \(\lim_{x\to2}x^2=4\)? Try to reconstruct it: you need to bound \(|x^2-4|\) in terms of \(|x-2|\).
We must make \(|x^2-4|<\varepsilon\). Factor:
\[|x^2-4| = |x-2|\,|x+2|.\]The first factor is what we control. The second is a nuisance — it depends on \(x\). The standard trick is to bound the nuisance first by pre-committing to a modest \(\delta\).
Insist \(\delta\le1\). Then \(|x-2|<1\) gives \(1<x<3\), hence \(3<x+2<5\), hence \(|x+2|<5\). So
\[|x^2-4| = |x-2|\,|x+2| < 5\,|x-2|.\]Now to force \(|x^2-4|<\varepsilon\) it suffices to force \(5|x-2|<\varepsilon\), i.e. \(|x-2|<\varepsilon/5\).
Both requirements are met by \(\delta=\min(1,\varepsilon/5)\): the \(1\) keeps the bound on \(|x+2|\) valid, and the \(\varepsilon/5\) does the actual work. Given any \(\varepsilon>0\) this \(\delta\) is positive, so we have a winning response to every challenge. ∎
You will not do many of these by hand. The reason to do one is to see that "approaches" is a completely precise notion with an algorithm behind it — not an appeal to intuition. Every convergence theorem in optimisation ("SGD converges at rate \(O(1/\sqrt{T})\)") is a statement of exactly this shape, with \(\varepsilon\) traded for an error bound and \(\delta\) for an iteration count.
In practice you almost never use \(\varepsilon\)–\(\delta\). You use limit laws — sums, products, quotients of limits behave as you would hope — plus one strategy for the interesting cases.
When you substitute \(x=c\) and get an honest number, that is the answer (for continuous functions, which is most of them). When you get \(\tfrac00\), you have an indeterminate form. That phrase is precise and worth unpacking.
\(\tfrac00\) is not "undefined" in the way \(\tfrac10\) is. \(\tfrac10\) has no possible answer. \(\tfrac00\) has too many: it tells you the limit could be anything, and you must dig further. Compare, all as \(x\to0\):
\[\frac{x}{x}\to 1, \qquad \frac{x^2}{x}\to 0, \qquad \frac{x}{x^2}\to \pm\infty, \qquad \frac{\sin x}{x}\to 1, \qquad \frac{2x}{x}\to 2 .\]Every one of them is \(\tfrac00\) on substitution, and they have five different answers. The symbol \(\tfrac00\) carries no information; the rate at which each part goes to zero carries all of it.
Now we can say precisely what it means for a function to have no jumps, gaps, or holes.
\(f\) is continuous at \(c\) if all three of the following hold:
\(f\) is continuous on a set if it is continuous at every point of that set.
Three separate conditions, and each can fail independently. That is not pedantry — it is a complete classification of how continuity breaks:
Continuity is not decorative in machine learning — it is a hard requirement in one place and a violated-but-tolerated assumption in another.
A limit says: as \(h\) shrinks towards zero, the ratio approaches the answer. Your computer disagrees, and the way it disagrees is instructive.
This is why nobody computes gradients by finite differences in deep learning. The error is
U-shaped in \(h\): shrink \(h\) and truncation error falls but roundoff error rises, and the best you can hope
for is roughly \(\sqrt{\varepsilon_{\text{machine}}} \approx 10^{-8}\) relative accuracy — for one
partial derivative, at the cost of one extra function evaluation. A network with \(10^{9}\) parameters would
need \(10^{9}\) forward passes per gradient. Automatic differentiation, which we build in
Phase D, computes all \(10^{9}\) partials exactly (to machine precision) in the cost of about two forward
passes. Finite differences survive only as gradcheck: a unit test for your autodiff, run on tiny
inputs.
We flagged two promissory notes. Both are limits, and both can now at least be stated precisely.
I am marking both as genuinely deferred rather than skipped. If you want the completeness argument now, say so and we will do it — it is about a page and it is beautiful. Otherwise it belongs in a real analysis detour after Phase C.
Evaluate each limit, showing the algebra. Say which are indeterminate on substitution.
(a) \(\displaystyle\lim_{x\to2}\frac{x^2-4}{x-2}\) (b) \(\displaystyle\lim_{x\to0}\frac{\sqrt{x+4}-2}{x}\) (c) \(\displaystyle\lim_{x\to\infty}\frac{3x^2-x}{5x^2+7}\) (d) \(\displaystyle\lim_{x\to0}\frac{|x|}{x}\)
(a) \(\tfrac00\). Factor: \(\frac{(x-2)(x+2)}{x-2}=x+2\to 4\).
(b) \(\tfrac00\). No obvious factor, so rationalise — multiply by the conjugate:
\[\frac{\sqrt{x+4}-2}{x}\cdot\frac{\sqrt{x+4}+2}{\sqrt{x+4}+2} = \frac{(x+4)-4}{x(\sqrt{x+4}+2)} = \frac{x}{x(\sqrt{x+4}+2)} = \frac{1}{\sqrt{x+4}+2}.\]Now substitute: \(\frac{1}{2+2}=\frac14\). The conjugate trick is the square-root analogue of factoring — it manufactures the cancelling factor.
(c) Not \(\tfrac00\) but \(\tfrac\infty\infty\), also indeterminate. Divide top and bottom by the highest power \(x^2\): \(\frac{3-1/x}{5+7/x^2}\to\frac{3-0}{5+0}=\frac35\).
(d) \(\tfrac00\). From the right, \(|x|=x\) so the ratio is \(+1\). From the left, \(|x|=-x\) so the ratio is \(-1\). The one-sided limits differ, so the limit does not exist — a jump discontinuity. (This function is \(\operatorname{sign}(x)\), and its non-existent limit at 0 is exactly why \(|x|\) is not differentiable there, which is why ReLU is not differentiable at 0.)
Define \[ f(x) = \begin{cases} \dfrac{x^2-x-6}{x-3}, & x\neq 3\\[6pt] k, & x=3\end{cases}\]
Find the value of \(k\) that makes \(f\) continuous at \(3\), and justify each of the three continuity conditions.
Factor the numerator: \(x^2-x-6=(x-3)(x+2)\). For \(x\neq3\), \(f(x)=x+2\).
Condition 2 (limit exists): \(\lim_{x\to3}f(x)=\lim_{x\to3}(x+2)=5\). ✓ regardless of \(k\), since the limit ignores \(x=3\).
Condition 1 (value exists): \(f(3)=k\), which exists for any real \(k\). ✓
Condition 3 (they agree): requires \(k=5\).
So \(k=5\), and the discontinuity was removable — as it always is when the limit exists but the value is wrong or missing. Note the structure of the answer: we did not "plug in and hope", we checked the three conditions in order and found exactly one of them constrained \(k\).
Prove from the \(\varepsilon\)–\(\delta\) definition that \(\displaystyle\lim_{x\to4}(3x-5)=7\).
Then explain, in one sentence each, what would go wrong in your proof if the limit were claimed to be \(7.001\) instead.
Scratch work (find \(\delta\) first, then write the proof forwards). We need \(|(3x-5)-7|<\varepsilon\), i.e. \(|3x-12|<\varepsilon\), i.e. \(3|x-4|<\varepsilon\), i.e. \(|x-4|<\varepsilon/3\). So \(\delta=\varepsilon/3\) should work — and here, unusually, the chain is reversible, so no bounding trick is needed.
Proof. Let \(\varepsilon>0\) be given. Choose \(\delta=\varepsilon/3>0\). Suppose \(0<|x-4|<\delta\). Then
\[|(3x-5)-7| = |3x-12| = 3|x-4| < 3\delta = 3\cdot\frac{\varepsilon}{3} = \varepsilon.\]Since \(\varepsilon>0\) was arbitrary, \(\lim_{x\to4}(3x-5)=7\). ∎
Why \(7.001\) fails. Take the specific challenge \(\varepsilon=0.0005\). We would need \(|(3x-5)-7.001|<0.0005\) whenever \(|x-4|<\delta\). But \((3x-5)-7.001 = 3(x-4)-0.001\), and as \(x\to4\) this tends to \(-0.001\), whose magnitude is \(0.001 > 0.0005\). So for every \(\delta\) there are points \(x\) in the punctured interval where the inequality fails — no \(\delta\) can win that round. In general: the limit, when it exists, is unique, and any wrong candidate is defeated by choosing \(\varepsilon\) smaller than half the gap between it and the true limit.
Coding. The central difference approximates a derivative as \(\frac{f(x+h)-f(x-h)}{2h}\). Modify the numerical-differentiation code above to compare forward and central differences on \(f(x)=e^x\) at \(x=1\).
import numpy as np
f, x, exact = np.exp, 1.0, np.e
for k in range(1, 16):
h = 10.0**(-k)
fwd = (f(x+h) - f(x)) / h
ctr = (f(x+h) - f(x-h)) / (2*h)
print(f"{h:.0e} fwd err {abs(fwd-exact):.3e} ctr err {abs(ctr-exact):.3e}")1 & 2. Forward bottoms out around \(h\approx10^{-8}\) with error \(\sim10^{-8}\). Central bottoms out around \(h\approx10^{-5}\) to \(10^{-6}\) with error \(\sim10^{-11}\) — roughly a thousandfold improvement, for one extra function evaluation.
3. Predicting the optimum. Total error models:
\[E_{\text{fwd}}(h)\approx \frac{M h}{2}+\frac{2\varepsilon}{h}, \qquad E_{\text{ctr}}(h)\approx \frac{M h^2}{6}+\frac{\varepsilon}{h}.\]Differentiate each and set to zero. For forward: \(\frac{M}{2}-\frac{2\varepsilon}{h^2}=0 \Rightarrow h^\star = 2\sqrt{\varepsilon/M} \sim \sqrt{\varepsilon}\approx 1.5\times10^{-8}\) ✓.
For central: \(\frac{Mh}{3}-\frac{\varepsilon}{h^2}=0 \Rightarrow h^{\star3}=3\varepsilon/M \Rightarrow h^\star \sim \varepsilon^{1/3}\approx 6\times10^{-6}\) ✓, with resulting error \(\sim\varepsilon^{2/3}\approx 4\times10^{-11}\) ✓.
The cube root beats the square root, which is the whole story. This is why gradcheck
implementations universally use central differences — and why, even so, they are only a test and never the
training algorithm.
Next: Part II opens with the derivative. You have already computed one — the quiz limit \(\lim_{h\to0}\frac{(2+h)^2-4}{h}=4\) was \(f'(2)\) for \(f(x)=x^2\). Now we do it in general, and the whole of optimisation opens up.