What a Function Really Is

1Why we begin here

Supervised machine learning has a one-sentence summary: we are searching for a function. We have inputs \(\vx\) (pixels, words, sensor readings), we want outputs \(y\) (a label, a price, the next token), and we believe some function \(f\) maps one to the other. Training is the search. Everything else — gradients, loss functions, backpropagation — is machinery for conducting that search efficiently.

So the object at the centre of the entire field is the function. It is worth two hours of your life to know exactly what one is. And I mean exactly, because the sloppy version — "a function is a formula like \(f(x)=x^2\)" — is the source of a specific class of error that will bite you in probability, in optimisation, and in numerical code.

2The definition

We build it from sets. A set is just a collection of distinct objects; \(x \in A\) means "\(x\) is a member of \(A\)". The sets we will use constantly:

• \(\NN = \{0,1,2,3,\dots\}\) — the natural numbers
• \(\ZZ = \{\dots,-2,-1,0,1,2,\dots\}\) — the integers
• \(\RR\) — the real numbers, the whole continuous line
• \(\RR^d\) — the set of ordered lists of \(d\) real numbers. This is where your data lives.

Two words in that definition are doing all the work, and you should underline them:

1. every — the rule must be defined at every single point of \(A\). No exceptions, no gaps. This is called being total.
2. exactly one — no input is allowed to produce two different outputs. This is called being well-defined or single-valued.

The second condition is the one you may already know as the vertical line test: if a vertical line crosses the graph twice, the picture is not the graph of a function. But notice the test is a consequence of the definition, not the definition itself. The definition works for functions between any sets at all, including sets you cannot draw.

3The domain is part of the function

Here is the idea that most courses mention once and never return to, and that quietly causes real errors later. Changing the domain changes the function. The two objects

\[f:\RR\to\RR,\quad f(x)=x^2 \qquad\text{and}\qquad g:[0,\infty)\to\RR,\quad g(x)=x^2\]

are not the same function, even though the formula is identical. They have different graphs, different images, and — as we will see — one of them has an inverse and the other does not.

When you are handed a bare formula with no domain stated, the convention is to take the natural domain: the largest subset of \(\RR\) on which every operation in the formula is defined. In practice there are exactly three things to check, and you should check them mechanically, every time:

1. Division. A denominator may not be zero. Exclude every root of the denominator.
2. Even roots. \(\sqrt{u}\), \(\sqrt[4]{u}\), … require \(u \ge 0\) over the reals.
3. Logarithms. \(\log_b(u)\) requires \(u > 0\), strictly. Not \(\ge 0\) — strictly greater.

4Injective, surjective, bijective

The definition demands one arrow out of every domain element. It says nothing about arrows into codomain elements. Classifying what happens on the receiving end gives us three properties that decide, among other things, whether a function can be undone.

Notice that surjectivity depends on how you declared the codomain, which is a choice. \(f:\RR\to\RR,\;f(x)=x^2\) is not surjective (nothing maps to \(-1\)). But \(f:\RR\to[0,\infty),\;f(x)=x^2\) is surjective, with the same formula. This is the domain lesson again, wearing a different hat: the sets are part of the object.

5Inverses, and why they need injectivity

To "undo" \(f\) means to find \(f^{-1}\) with \(f^{-1}(f(a))=a\) for every \(a\in A\), and \(f(f^{-1}(b))=b\) for every \(b\in B\). Ask when this is possible.

Geometrically, \(f^{-1}\) swaps the roles of input and output, so its graph is the graph of \(f\) reflected across the line \(y=x\). That gives a fast visual test: reflect the curve; if the reflection fails the vertical line test, \(f\) was not injective and has no inverse.

6Composition: the shape of a neural network

Given \(f:A\to B\) and \(g:B\to C\), the composition \(g\circ f: A\to C\) is defined by

\[(g\circ f)(a) = g\big(f(a)\big).\]

Read it right to left: \(f\) fires first. The codomain of \(f\) must match (or sit inside) the domain of \(g\), or the pipeline does not type-check. This "shape matching" requirement is the abstract version of the matrix-dimension errors you will spend a lot of your life debugging.

7Functions in NumPy

Now let us make all of this concrete. The code below is live: edit it and press Run. The first run downloads Python and NumPy into your browser, which takes about ten seconds; after that it is instant.

Now the specific trap from the warning above. Run this and read the output carefully.

8The machine-learning restatement

Let me close the loop on why any of this matters. In supervised learning we posit a hypothesis class — a set \(\mathcal{H}\) of candidate functions — and search inside it:

\[\mathcal{H} = \{\, f_{\vt} : \RR^d \to \RR^k \;\mid\; \vt \in \Theta \,\}\]

Every symbol there is a thing you now have language for:

• \(\RR^d\) is the domain: your input space. \(d\) is the number of features. For a \(28\times 28\) grayscale image, \(d=784\).
• \(\RR^k\) is the codomain: the output space. \(k=1\) for scalar regression, \(k=10\) for ten-class classification.
• \(\vt\) is a parameter — a knob setting. Each choice of \(\vt\) picks out one specific function from the class. \(\Theta\) is the set of legal knob settings.
• Training means choosing \(\vt\). It does not mean choosing \(\mathcal{H}\) — that choice was made when you picked the architecture, and it constrains everything that follows.

And the domain lesson has one more payoff here. When you normalise your inputs, or clip them, or one-hot encode them, you are changing the domain of your model. That is why a model trained on normalised inputs produces nonsense on raw ones: you handed it an element that was never in \(A\).

9Exercises

Do these with pen and paper before opening the solutions. If you can only do some of them, that is useful information — tell me which ones and I will target the next lecture accordingly.

import numpy as np

def safe_log2(x):
    x = np.asarray(x, dtype=float)
    bad = x < 0
    if bad.any():
        raise ValueError(f"log2 undefined for negative inputs: {x[bad]}")
    out = np.full_like(x, -np.inf)
    pos = x > 0
    out[pos] = np.log2(x[pos])
    return out

Next: we take the logarithm apart from the definition up. You will see precisely why \(\log_2(-2)\) is undefined, why every log rule is a disguised exponent rule, and why machine learning is so addicted to logs that half its loss functions have one in the name.