Machine learning is addicted to logarithms. Log-likelihood, cross-entropy, logits, log-sum-exp, perplexity, KL divergence. This lecture builds the logarithm from its definition so that every one of those is transparent — and so that you never again wonder what log(−2) is.
Before the mathematics, the motive. Three facts about probabilities force logs on us whether we like them or not.
That third point is doing real work, and we will prove it. It is why "maximum likelihood" and "minimum negative log-likelihood" are the same procedure. But none of it is safe unless you know precisely where \(\log\) is defined, so we start there.
Fix a base \(a>0\), \(a \neq 1\). What does \(a^x\) mean? The honest answer is that it is built in four stages, each extending the last, and each forced by the requirement that one law keep holding.
For every base \(a>0\) and every real \(x\), \(\;a^x > 0\).
Proof. Take any real \(x\). Then
\[ a^x = a^{\frac{x}{2}+\frac{x}{2}} = a^{x/2} \cdot a^{x/2} = \left(a^{x/2}\right)^2 . \]A real number squared is \(\ge 0\), so \(a^x \ge 0\). It cannot equal \(0\): if \(a^x = 0\) then \(1 = a^0 = a^{x}a^{-x} = 0 \cdot a^{-x} = 0\), a contradiction. Hence \(a^x>0\). ∎
Look at how little that proof needed: the addition law and the fact that squares are non-negative. It holds for every real \(x\), no matter how large and negative. As \(x \to -\infty\) the value shrinks towards zero, but it is a strict inequality at every finite \(x\), and the limit is never attained.
For \(a>0,\ a\neq 1\), the function \(x \mapsto a^x\) is strictly monotone (increasing if \(a>1\), decreasing if \(a<1\)), hence injective. Its image is exactly \((0,\infty)\): every positive number is achieved, and nothing else is. So as a map
\[\exp_a : \RR \longrightarrow (0,\infty)\]
it is a bijection. By the theorem of Lecture 1, it has an inverse. We name it.
For \(a>0,\ a\neq 1\), the logarithm base \(a\) is the inverse of \(x\mapsto a^x\):
\[ \log_a : (0,\infty) \longrightarrow \RR, \qquad \boxed{\;\log_a(y) = c \iff a^c = y\;} \]Domain \((0,\infty)\). Codomain \(\RR\). The two are swapped relative to the exponential, exactly as the inverse-function theory demands.
Now we can settle the question directly, and I want you to see that it needs no rules at all — only the definition.
The claim "\(\log_2(-2) = -1\)" unpacks to the claim \(2^{-1} = -2\). But \(2^{-1} = \tfrac12\). So the claim is false.
Could some other exponent work? "\(\log_2(-2) = c\)" means \(2^c = -2\). The theorem above says \(2^c > 0\) for every real \(c\). A positive number is never \(-2\). So there is no \(c\), and \(\log_2(-2)\) does not name a real number at all. It is not \(-1\), not \(0\), not any value — it is undefined, in the same way \(1/0\) is undefined.
Where negative outputs really come from. \(\log_2\) is negative exactly when its input is strictly between \(0\) and \(1\): \(\log_2(0.5)=-1\) because \(2^{-1}=0.5\). Small positive input, negative output. It is the input being a fraction, never the input being negative.
You already used \(\log m + \log n = \log(mn)\) correctly. Now let us see where it comes from, because deriving it takes thirty seconds and it means you will never misremember its cousins.
The other three fall out the same way. Try each before revealing.
Using the same three-move pattern — name, convert, apply an exponent law, convert back — derive:
(a) \(\log_a\!\left(\dfrac{m}{n}\right) = \log_a m - \log_a n\) (b) \(\log_a(m^k) = k\log_a m\) (c) \(\log_a m = \dfrac{\log_b m}{\log_b a}\) (change of base)
(a) Quotient law. Let \(u=\log_a m\), \(v=\log_a n\), so \(a^u=m\), \(a^v=n\). Then \(\frac{m}{n}=\frac{a^u}{a^v}=a^{u-v}\), using \(a^{u-v}a^{v}=a^{u}\). Converting back, \(\log_a\!\frac{m}{n}=u-v\). ∎
(b) Power law. Let \(u=\log_a m\), so \(a^u=m\). Raise both sides to the \(k\): \(m^k=(a^u)^k=a^{uk}\), using the law \((a^u)^k=a^{uk}\). Converting back, \(\log_a(m^k)=uk=k\log_a m\). ∎
Sanity check with numbers: \(\log_2(8^2)=\log_2 64=6\), and \(2\log_2 8 = 2\cdot 3=6\) ✓.
(c) Change of base. Let \(u=\log_a m\), so \(a^u=m\). Apply \(\log_b\) to both sides — legal because both sides are positive and \(\log_b\) is a function, so equal inputs give equal outputs:
\[\log_b(a^u)=\log_b m.\]By the power law just proved, the left side is \(u\log_b a\). So \(u\log_b a=\log_b m\), and since \(a\neq1\) we have \(\log_b a\neq0\) and may divide:
\[ u=\frac{\log_b m}{\log_b a}. \qquad \blacksquare\]Why this one matters in practice: your computer only implements \(\ln\) (natural log). Every \(\log_2\) and \(\log_{10}\) you have ever computed was secretly \(\ln m / \ln 2\) or \(\ln m/\ln 10\). And in information theory, changing the base only rescales entropy by a constant — which is precisely why "bits" (base 2) and "nats" (base \(e\)) are interchangeable units.
Three non-laws that people invent. None of these is true:
\[\log(m+n) \neq \log m + \log n, \qquad \frac{\log m}{\log n} \neq \log\!\left(\frac{m}{n}\right), \qquad (\log m)^k \neq k \log m.\]Test the first with \(m=n=1\), base 10: the left is \(\log 2 \approx 0.301\), the right is \(0+0=0\).
The reason there is no law for \(\log(m+n)\) is structural: exponentials convert addition into
multiplication, so logs convert multiplication back into addition. Addition on the inside has no
counterpart. Notice that logsumexp, which we meet below, exists precisely because
\(\log(\text{a sum})\) is awkward.
Every base works. So why does all of mathematics and all of machine learning use \(e\approx 2.71828\)? The answer is a statement about derivatives, which we will prove properly in Lecture 5. Here is the statement and the numerical evidence.
Differentiate \(a^x\) from the definition of the derivative (Lecture 4):
\[\frac{d}{dx}a^x = \lim_{h\to0}\frac{a^{x+h}-a^x}{h} = a^x \cdot \underbrace{\lim_{h\to0}\frac{a^h-1}{h}}_{\displaystyle =: \,C(a)}\]
The factor \(a^x\) pulled straight out, because \(a^{x+h}=a^xa^h\) — the addition law again. So every exponential is proportional to its own derivative. The constant of proportionality \(C(a)\) depends on the base. The number \(e\) is defined as the base for which \(C(a)=1\):
\[\frac{d}{dx}e^x = e^x.\]
\(e^x\) is the function that is its own rate of change: at every point, the height equals the slope. That is the mathematical shape of "growth proportional to current size" — compound interest, population growth, radioactive decay, and, most relevant here, the way a gradient-descent step of the form \(w \leftarrow w(1-\eta)\) decays a weight geometrically. The natural log \(\ln = \log_e\) is its inverse, and \(\frac{d}{dx}\ln x = \frac1x\), which is why \(\ln\) shows up the moment you differentiate any log-likelihood.
This is the theorem that licenses every "we take logs for convenience" sentence in every paper you will read. It deserves a proof, not a wave.
Let \(g:\Theta\to(0,\infty)\) be any strictly positive function. Then
\[ \argmax_{\vt \in \Theta} g(\vt) \;=\; \argmax_{\vt \in \Theta} \log g(\vt), \]meaning the two problems have exactly the same set of maximisers.
Proof. For \(a>1\), \(\log_a\) is strictly increasing: if \(0<s<t\) then \(\log_a s < \log_a t\). (Reason: \(\log_a\) is the inverse of the strictly increasing \(a^x\), and the inverse of a strictly increasing bijection is strictly increasing — if it ever decreased, reflecting would contradict monotonicity of \(a^x\).)
Now suppose \(\vt^\star\) maximises \(g\), so \(g(\vt) \le g(\vt^\star)\) for all \(\vt\). Applying the increasing function \(\log\) preserves \(\le\), giving \(\log g(\vt) \le \log g(\vt^\star)\) for all \(\vt\); hence \(\vt^\star\) maximises \(\log g\).
Conversely, suppose \(\vt^\star\) maximises \(\log g\). Apply the increasing function \(\exp\) to \(\log g(\vt) \le \log g(\vt^\star)\) to get \(g(\vt)\le g(\vt^\star)\). So the maximiser sets are equal. ∎
This is why the following are all the same optimisation problem, and you will see all four written down as if they were interchangeable — because they are:
\[\argmax_{\vt}\ \prod_{i=1}^n p(y_i\mid \vx_i;\vt) \;=\;\argmax_{\vt}\ \sum_{i=1}^n \log p(y_i\mid \vx_i;\vt) \;=\;\argmin_{\vt}\ -\!\sum_{i=1}^n \log p(y_i\mid \vx_i;\vt) \;=\;\argmin_{\vt}\ -\frac1n\sum_{i=1}^n \log p(y_i\mid \vx_i;\vt)\]The first equality is the theorem plus the product law. The second is because maximising \(f\) is minimising \(-f\). The third is because dividing by the positive constant \(n\) cannot move a minimiser. That last object is the average negative log-likelihood, which for classification is exactly the cross-entropy loss. You have now derived the most common loss function in machine learning from the monotonicity of the logarithm.
Now the engineering payoff. Run this and watch a likelihood die.
The mirror-image problem appears in softmax. You need \(\log \sum_j e^{z_j}\), and if some
\(z_j\) is large, \(e^{z_j}\) overflows to inf. The fix is a one-line identity worth knowing by
heart.
inf in float64, and \(\log(\infty)=\infty\).Never write np.log(np.sum(np.exp(z))) in code you care about. Use
scipy.special.logsumexp, or torch.logsumexp, or the three-line stable version above.
Relatedly, never write np.log(softmax(z)) — use log_softmax(z) = z - logsumexp(z),
which is both faster and stable. If you have ever seen a training run produce nan loss on step
one, this is the first place to look.
One last pairing, because it is Lecture 1's inverse machinery applied to this lecture's functions, and it is the entire mathematical content of logistic regression.
Solve, showing every step, and state the domain restriction before you start solving:
(a) \(\log_3(x) + \log_3(x-6) = 3\) (b) \(\log_5(x+4) - \log_5(x) = 1\) (c) \(2\log_2(x) = \log_2(6-x) + 1\)
(a) Domain first: need \(x>0\) and \(x-6>0\), so \(x>6\).
\(\log_3(x(x-6))=3 \Rightarrow x^2-6x=27 \Rightarrow x^2-6x-27=0 \Rightarrow (x-9)(x+3)=0
\Rightarrow x=9\) or \(x=-3\).
\(x=-3\) fails \(x>6\), so reject it. Answer: \(x=9\) only.
Verify: \(\log_3 9 + \log_3 3 = 2+1=3\) ✓.
(b) Domain: \(x+4>0\) and \(x>0\), so \(x>0\).
\(\log_5\!\frac{x+4}{x}=1 \Rightarrow \frac{x+4}{x}=5^1=5 \Rightarrow x+4=5x \Rightarrow x=1\).
\(1>0\) ✓. Verify: \(\log_5 5 - \log_5 1 = 1-0=1\) ✓.
(c) Domain: \(x>0\) and \(6-x>0\), so \(0<x<6\).
\(\log_2(x^2)=\log_2(6-x)+\log_2 2=\log_2(2(6-x))\). Since \(\log_2\) is injective, equal logs force equal
arguments: \(x^2=12-2x \Rightarrow x^2+2x-12=0 \Rightarrow x=\frac{-2\pm\sqrt{4+48}}{2}=-1\pm\sqrt{13}\).
\(\sqrt{13}\approx3.606\), giving \(x\approx2.606\) or \(x\approx-4.606\). Only the first lies in \((0,6)\).
Answer: \(x=\sqrt{13}-1\).
Note the step "equal logs force equal arguments" is injectivity of \(\log_2\) — it is the reason the move
is legal, and it is worth naming.
Explain, using the definition rather than the rules, why each of these is undefined or ill-posed:
(a) \(\log_2(0)\) (b) \(\log_1(5)\) (c) \(\log_{-2}(4)\)
(a) \(\log_2(0)=c\) means \(2^c=0\). We proved \(2^c>0\) for every real \(c\).
No solution, so undefined. Note this is a limit case in a way that \(\log_2(-2)\) is not: as
\(y\to0^+\), \(\log_2 y\to-\infty\), so \(-\infty\) is a legitimate extended-real value. That is why
np.log(0.0) returns -inf with a warning, but np.log(-2.0) returns
nan.
(b) \(\log_1(5)=c\) means \(1^c=5\). But \(1^c=1\) for every \(c\). No solution. Worse, \(\log_1(1)=c\) means \(1^c=1\), which every \(c\) satisfies — so the value would not be unique. Base 1 fails both conditions of being a function: \(x\mapsto 1^x\) is constant, hence neither injective nor surjective onto \((0,\infty)\). That is exactly why the definition excludes \(a=1\).
(c) A negative base breaks at Stage 3 of the construction: \((-2)^{1/2}\) would be a real square root of \(-2\), which does not exist. So \(x\mapsto(-2)^x\) is not defined on \(\RR\) at all — it only makes sense at some rationals, giving a wildly discontinuous object. Since there is no exponential function to invert, there is no logarithm. (\(\log_{-2}(4)=2\) is tempting because \((-2)^2=4\), but a single lucky point does not make a function.)
Prove that \(\sigma(-z) = 1-\sigma(z)\), and use it to show that the binary cross-entropy loss
\[\ell(z,y) = -\big[y\log\sigma(z) + (1-y)\log(1-\sigma(z))\big], \qquad y\in\{0,1\}\]can be written in the single expression \(\ell = \log(1+e^{-(2y-1)z})\).
Why is the second form better numerically?
Part 1. \(\displaystyle 1-\sigma(z) = 1-\frac{1}{1+e^{-z}} = \frac{1+e^{-z}-1}{1+e^{-z}} = \frac{e^{-z}}{1+e^{-z}}\). Multiply top and bottom by \(e^{z}\): \(\displaystyle = \frac{1}{e^{z}+1} = \sigma(-z)\). ∎
Part 2. Note \(\log\sigma(z) = \log\frac{1}{1+e^{-z}} = -\log(1+e^{-z})\).
If \(y=1\): \(\ell = -\log\sigma(z) = \log(1+e^{-z})\). And \(2y-1=1\), so the claimed formula gives \(\log(1+e^{-z})\) ✓.
If \(y=0\): \(\ell = -\log(1-\sigma(z)) = -\log\sigma(-z) = \log(1+e^{z})\). And \(2y-1=-1\), so the formula gives \(\log(1+e^{-(-1)z}) = \log(1+e^{z})\) ✓.
Both cases match, and \(y\) takes only those two values, so the forms agree. ∎
Part 3 — why it is better. The first form computes \(\sigma(z)\) and then takes its
log. For \(z=-40\), \(\sigma(z)\approx 4\times10^{-18}\), which is representable — but for \(z=-800\),
\(\sigma(z)\) underflows to exactly \(0.0\) and \(\log 0 = -\infty\): the loss becomes inf and
the gradient nan, even though the true loss is a perfectly finite \(800\).
The combined form never materialises the probability. Writing \(t=-(2y-1)z\), the standard stable
implementation is \(\max(t,0) + \log(1+e^{-|t|})\), which is log-sum-exp applied to \(\log(e^0+e^t)\). This is
exactly why PyTorch offers BCEWithLogitsLoss and warns you not to chain
Sigmoid into BCELoss: fusing the two operations is a numerical necessity, not a
micro-optimisation.
Coding + reasoning. The entropy of a discrete distribution \(\mathbf{p}=(p_1,\dots,p_K)\) with \(\sum p_k=1\) is
\[ H(\mathbf p) = -\sum_{k=1}^{K} p_k \log_2 p_k \quad \text{bits.}\]entropy(p) in NumPy handling that case, and verify it on the uniform distribution
over \(K\) outcomes.1. Use \(0\). Justification: the term is \(p\log_2 p\), and \(\lim_{p\to0^+} p\log_2 p = 0\), because \(p\) goes to zero faster than \(\log_2 p\) goes to \(-\infty\). (Formally this is L'Hôpital on \(\frac{\log_2 p}{1/p}\), which we will have tools for in Lecture 4.) So \(p\log_2 p\) extends continuously to \(p=0\) with value 0. This is a genuine limit, not a convention of convenience — an outcome that never happens contributes no surprise.
import numpy as np
def entropy(p):
p = np.asarray(p, dtype=float)
if np.any(p < 0) or not np.isclose(p.sum(), 1.0):
raise ValueError("not a probability distribution")
nz = p > 0 # mask out the zeros
return -np.sum(p[nz] * np.log2(p[nz]))
for K in [2, 4, 8, 256]:
u = np.full(K, 1.0 / K)
print(K, entropy(u), np.log2(K))Uniform over \(K\) gives \(-\sum \frac1K\log_2\frac1K = -K\cdot\frac1K\cdot(-\log_2 K)=\log_2 K\). So \(K=256\) gives exactly 8 bits — one byte, which is the sanity check that the units are right.
3. With \(K=2\), write \(\mathbf p=(p,1-p)\) for \(p\in(0,1)\):
\[H(p) = -p\log_2 p-(1-p)\log_2(1-p).\]Differentiate. Using \(\frac{d}{dp}\log_2 p = \frac{1}{p\ln2}\) (change of base, then Lecture 5's rule):
\[H'(p) = -\log_2 p - \frac{1}{\ln 2} + \log_2(1-p) + \frac{1}{\ln2} = \log_2\!\frac{1-p}{p}.\]Setting \(H'(p)=0\) gives \(\frac{1-p}{p}=1\), so \(p=\tfrac12\). And \(H''(p) = \frac{d}{dp}\big[\log_2(1-p)-\log_2 p\big] = -\frac{1}{(1-p)\ln2}-\frac{1}{p\ln 2} < 0\) everywhere on \((0,1)\), so \(H\) is strictly concave and the critical point is the global maximum. \(H(\tfrac12)=1\) bit. ∎
Notice what \(H'(p)=\log_2\frac{1-p}{p}\) is: the negative of the logit. The same function keeps reappearing, which is not a coincidence — it is the reason cross-entropy has such a clean gradient, as you will see in Phase B.
Next: Lecture 3 makes the word "limit" precise. We used it twice in this lecture on promissory notes — to define \(a^x\) for irrational \(x\), and to define \(C(a)\). Time to pay both debts, because the derivative in Lecture 4 is built entirely out of limits.