{
"cells": [
{
"cell_type": "markdown",
"id": "0018d093",
"metadata": {},
"source": [
"# Foundations and Notation\n",
"\n",
"*The Mathematics of Large Language Models ยท Chapter 1*\n",
"\n",
"Measure a profile, compress a table, score a forecast, trace a change, and know how much data is enough.\n",
"\n",
"These pages illustrate the mathematics. Read the saved figures and calculations in order, or open [the interactive browser edition](reader.html) to change the examples. No code editing is needed.\n",
"\n",
"Ask for the relevant numbers, their meanings and units, and the decision to support, then choose the matching equation. For profile questions, compare size and direction separately. For table simplification, report compression error alongside retained squared norm. For forecasts, check probability support before scoring. For sensitivity questions, show a local derivative and a finite-change check. For sample-size questions, state the range of the data and whether a worst-case or typical-case guarantee is wanted. Return only the relevant calculation in a form the reader can reproduce."
]
},
{
"cell_type": "markdown",
"id": "35ebbb53",
"metadata": {},
"source": [
"Reproducing these calculations
\n",
"\n",
"The optional calculation cells use Python with NumPy, SciPy, and Matplotlib. All inputs are included in this file. The figures below are saved, so running these cells is optional.\n",
"\n",
"Show the answer
\n",
"\n",
"The norms are 7, 5, and 4 for p = 1, 2, and infinity. Cosine 1 indicates the same positive direction; it provides no check of the records' accuracy.\n",
"\n",
"Show the answer
\n",
"\n",
"At k = 0 the approximation is zero and the error is sqrt(31). At k = 3 all directions are retained, so the error is zero apart from numerical roundoff.\n",
"\n",
"Show the answer
\n",
"\n",
"If q_i exceeds p_i, log(p_i/q_i) is negative for that outcome. Other terms offset it; Gibbs' inequality applies to the sum, not to every term. A matching forecast has total KL zero.\n",
"\n",
"Show the answer
\n",
"\n",
"The slope is 4, so the prediction is 4 times 0.5 = 2. The output goes from 1 to 4, an actual change of 3. The missing 1 equals 4 times 0.5 squared.\n",
"\n",
"Show the answer
\n",
"\n",
"n = ln(40) / (2 x 0.1^2) = 3.69 / 0.02 = 184. Halving the allowed miss multiplies the sample size by 4, because n scales with 1 / epsilon squared.\n",
"\n",
"Show the answer
\n",
"\n",
"It grows by the factor ln(80) / ln(40) = 1.19, only about 19%. The guarantee depends on the logarithm of the number of points, not on the number of points or on the original dimension.\n",
"\n",
"Show the answer
\n",
"\n",
"About 128. The gap scales as skewness over the square root of n, so doubling the skewness needs four times as many draws to keep the same gap.\n",
"\n",
"Show the answer
\n",
"\n",
"n = 1 / (0.01 x 0.2) = 500. Halving the variance always needs twice as many examples.\n",
"\n",
"