{
"cells": [
{
"cell_type": "markdown",
"id": "d1d9d120",
"metadata": {},
"source": [
"# Generalization and Implicit Bias\n",
"\n",
"*The Mathematics of Large Language Models ยท Chapter 4*\n",
"\n",
"Separate repeated-fit uncertainty, bounds, interpolation, and solution choice.\n",
"\n",
"These pages illustrate the mathematics. Read the saved figures and calculations in order, or open [the interactive browser edition](reader.html) to change the examples. No code editing is needed.\n",
"\n",
"Ask for sample selection, target population, loss definition, candidate class, fitting procedure, and untouched evaluation data. Return an evaluation design and supported uncertainty calculations. Distinguish repeated-sample expectation, one observed test score, and a theorem under stated assumptions."
]
},
{
"cell_type": "markdown",
"id": "f7cf394d",
"metadata": {},
"source": [
"Reproducing these calculations
\n",
"\n",
"The optional calculation cells use Python with NumPy, SciPy, and Matplotlib. All inputs are included in this file. The figures below are saved, so running these cells is optional.\n",
"\n",
"Show the answer
\n",
"\n",
"The average prediction is 2, so bias squared is (2-1)^2 = 1. Variance is ((1-2)^2 + (3-2)^2)/2 = 1. Total error is (0 + 4)/2 = 2 = 1 + 1.\n",
"\n",
"Show the answer
\n",
"\n",
"The count of weights is not the count of candidate models. Real weights allow infinitely many different models, so a different capacity argument is needed.\n",
"\n",
"Show the answer
\n",
"\n",
"Four. The noise term goes to 0, while 4 x (d - n)/d goes to 4: the second descent need not reach zero.\n",
"\n",
"Show the answer
\n",
"\n",
"Zero and one, since the prediction at x = 0 is 0.5 + w. Both still predict one at x = -1 and x = 1.\n",
"\n",
"Show the answer
\n",
"\n",
"0.04. The number of points is 25 times larger, the square root is 5, so the score falls to 0.2/5.\n",
"\n",
"Show the answer
\n",
"\n",
"sqrt(0.25) = 0.5, so the lower edge is (1 - 0.5)^2 = 0.25 and the upper edge is (1 + 0.5)^2 = 2.25.\n",
"\n",
"Show the answer
\n",
"\n",
"It more than doubles. The factor 1/lambda doubles, and the loss range c grows too because weights may be larger.\n",
"\n",
"Show the answer
\n",
"\n",
"The divergence is 0. What is left is the training error plus the best joint error, which still includes the label noise twice.\n",
"\n",
"