{
"cells": [
{
"cell_type": "markdown",
"id": "75c8b1c8",
"metadata": {},
"source": [
"# Approximation and Expressivity\n",
"\n",
"*The Mathematics of Large Language Models ยท Chapter 2*\n",
"\n",
"Construct curves, count pieces, and separate three obstacles.\n",
"\n",
"These pages illustrate the mathematics. Read the saved figures and calculations in order, or open [the interactive browser edition](reader.html) to change the examples. No code editing is needed.\n",
"\n",
"Identify the input domain, target, error norm, candidate family, optimization evidence, and held-out population. Return separate assessments of representation, training, and generalization; compute only quantities justified by supplied numeric inputs."
]
},
{
"cell_type": "markdown",
"id": "6e86d36c",
"metadata": {},
"source": [
"Reproducing these calculations
\n",
"\n",
"The optional calculation cells use Python with NumPy, SciPy, and Matplotlib. All inputs are included in this file. The figures below are saved, so running these cells is optional.\n",
"\n",
"Show the answer
\n",
"\n",
"Three bends plus one ReLU for the starting slope, four in total. Counting pieces as bends would miss the first slope.\n",
"\n",
"Show the answer
\n",
"\n",
"No. T(x) = T(1-x), so any outer function gives the same value at x and 1-x. An asymmetric target cannot be copied this way.\n",
"\n",
"Show the answer
\n",
"\n",
"Four: the exponent is 1/2, so multiplying n by four multiplies error by 4^(-1/2) = 1/2.\n",
"\n",
"Show the answer
\n",
"\n",
"No. Optimization excess becomes zero but the floor 4/45 stays, because no constant equals x squared across the population.\n",
"\n",
"Show the answer
\n",
"\n",
"One hundred times larger, because error is proportional to D^(-1/2) and 100^(-1/2) = 1/10.\n",
"\n",
"Show the answer
\n",
"\n",
"The exponent gap is 1/2 - 1/4 = 1/4, so n* = 100^4 = 10^8: the dimension-free rate wins only at very large sizes.\n",
"\n",
"Show the answer
\n",
"\n",
"At least 63 neurons in one layer. The tent with 6 folds uses 3 x 6 = 18 neurons.\n",
"\n",
"Show the answer
\n",
"\n",
"About 16, since 2^(3+1) = 16. For this sine the measured ratio at 8 intervals is 18.6: it is not exactly 16 because the grid is still coarse. It wanders (15.1, 18.6, 18.8, 17.8 from 4 to 32 intervals) and settles toward 16 as the grid gets finer.\n",
"\n",
"