{
"cells": [
{
"cell_type": "markdown",
"id": "a9b144ba",
"metadata": {},
"source": [
"# Scaling Laws, In-Context Learning, and Abstract Foundations\n",
"\n",
"*The Mathematics of Large Language Models ยท Chapter 16*\n",
"\n",
"Distinguish resources, examples and the objective being rewarded.\n",
"\n",
"These pages illustrate the mathematics. Read the saved figures and calculations in order, or open [the interactive browser edition](reader.html) to change the examples. No code editing is needed.\n",
"\n",
"Choose an experiment that changes one relevant factor and measures task quality. Use numerical allocation only when the reader supplies a defensible quantitative law; use the toy models to understand mechanisms otherwise."
]
},
{
"cell_type": "markdown",
"id": "96c77f07",
"metadata": {},
"source": [
"Reproducing these calculations
\n",
"\n",
"The optional calculation cells use Python with NumPy, SciPy, and Matplotlib. All inputs are included in this file. The figures below are saved, so running these cells is optional.\n",
"\n",
"Show the answer
\n",
"\n",
"Not without the floor. If the floor is large, the reducible part can fall much faster than the total suggests; fit the excess above a stated floor.\n",
"\n",
"Show the answer
\n",
"\n",
"Both grow by the square root of 2, about 1.41 times: N* = sqrt(4 x 2000) = 89.4 and D* = 22.4. Equal exponents split the growth evenly.\n",
"\n",
"Show the answer
\n",
"\n",
"With K = 1 and x = 1 the step is y x = 1, and least squares is y x / x^2 = 1. They coincide only when the mean of x^2 equals 1.\n",
"\n",
"Show the answer
\n",
"\n",
"The starting distribution, for every beta: the common exponential factor cancels in the normalization.\n",
"\n",
"Show the answer
\n",
"\n",
"0.5^(1/100) = 0.9931, about 99.3%, so very high token accuracy is needed before long exact answers appear.\n",
"\n",
"Show the answer
\n",
"\n",
"By symmetry each direction has two neighbours at 72 degrees with overlap cos 72 = 0.31; the error is 2 x 0.31^2 = 0.19 for each, so five features add up to 0.95 p.\n",
"\n",
"Show the answer
\n",
"\n",
"Above 1.1: past that the clipped term is constant at 1.1 x A, so the slope is zero.\n",
"\n",
"Show the answer
\n",
"\n",
"0.5: sigma(0) = 1/2, so an undecided pair always gets half weight, whatever beta is.\n",
"\n",
"