{
"cells": [
{
"cell_type": "markdown",
"id": "c6350c57",
"metadata": {},
"source": [
"# Optimization and Training Dynamics\n",
"\n",
"*The Mathematics of Large Language Models ยท Chapter 3*\n",
"\n",
"Inspect stable steps, noisy gradients, adaptive moments, schedules, curvature limits and batch size.\n",
"\n",
"These pages illustrate the mathematics. Read the saved figures and calculations in order, or open [the interactive browser edition](reader.html) to change the examples. No code editing is needed.\n",
"\n",
"Request the objective, gradient or measured response, curvature estimate, step units, noise model, and stopping target. Compute quadratic stability when justified, compare steps or batches, and return a small reproducible experiment with stopping criteria. Explain nonconvex and adaptive-method limits."
]
},
{
"cell_type": "markdown",
"id": "a78b596b",
"metadata": {},
"source": [
"Reproducing these calculations
\n",
"\n",
"The optional calculation cells use Python with NumPy, SciPy, and Matplotlib. All inputs are included in this file. The figures below are saved, so running these cells is optional.\n",
"\n",
"Show the answer
\n",
"\n",
"The curvature at the bottom is 12 x 4 - 16 = 32, so the limit is 2/32 = 0.0625. A wider, steeper well needs a much smaller step.\n",
"\n",
"Show the answer
\n",
"\n",
"(1 + 1 + 2 x 0.5)/4 = 0.75, larger than the independent value 0.5. Correlation limits how much averaging helps.\n",
"\n",
"Show the answer
\n",
"\n",
"m1 = 1.5, m2 = 0.5 x 1.5 + 0.5 x 3 = 2.25. Corrected: 2.25/(1 - 0.25) = 3. The correction removes the shrinkage from starting at zero.\n",
"\n",
"Show the answer
\n",
"\n",
"At update 50 the cosine term is 0, so the step is 0.4/2 = 0.2. At update 100 it reaches 0, but that final value is not used for an update.\n",
"\n",
"Show the answer
\n",
"\n",
"The edge is 2/1.5 = 1.33. The floor 1 is below it, so the bottom itself is stable and sharpness ends at 1. If instead eta were above 2, even the floor would sit past the edge and the walk could not settle.\n",
"\n",
"Show the answer
\n",
"\n",
"kappa = 40, so the shrink per update is 1 - 1/40 = 0.975. ln(1,000,000)/(-ln 0.975) = 13.8/0.0253, about 546 updates (an upper estimate; the real run is faster).\n",
"\n",
"Show the answer
\n",
"\n",
"Before. ln(6/0.01 - 1)/6 = ln(599)/6 = 1.07, against 1.9 for target 3. Larger targets are learned sooner.\n",
"\n",
"Show the answer
\n",
"\n",
"Updates = 1000 x (1 + 1) = 2000, twice the floor, and examples = 1024 x 2000, twice the minimum 1000 x 1024. The knee costs a factor of two in both.\n",
"\n",
"