{
"cells": [
{
"cell_type": "markdown",
"id": "a6e2d80e",
"metadata": {},
"source": [
"# The Mathematics of Truth, Error, and Abstention\n",
"\n",
"*The Mathematics of Large Language Models ยท Chapter 17*\n",
"\n",
"Separate what was seen, what is supported, and what can responsibly be answered.\n",
"\n",
"These pages illustrate the mathematics. Read the saved figures and calculations in order, or open [the interactive browser edition](reader.html) to change the examples. No code editing is needed.\n",
"\n",
"Break the answer into checkable claims and create a table containing the claim, its actual source, the relevant passage or calculation, and a check status. Distinguish supported, contradicted, and unresolved claims. Use error rates only when measured on a relevant population; use costs to discuss abstention, and leave unsupported claims unresolved instead of treating fluent confidence as verification."
]
},
{
"cell_type": "markdown",
"id": "97688dca",
"metadata": {},
"source": [
"Reproducing these calculations
\n",
"\n",
"The optional calculation cells use Python with NumPy, SciPy, and Matplotlib. All inputs are included in this file. The figures below are saved, so running these cells is optional.\n",
"\n",
"Show the answer
\n",
"\n",
"Only the third type is unseen, so actual missing mass is 0.2. One type occurs exactly once among three draws, giving the estimate 1/3. The quantities need not coincide in a realized sample.\n",
"\n",
"Show the answer
\n",
"\n",
"Risk is (0.2 times 0.02 + 0.3 times 0.08)/0.5 = 0.056. The break-even is (0 + 4)/(1 + 4) = 0.8. At exactly 0.8, answering and abstaining tie in expected reward.\n",
"\n",
"Show the answer
\n",
"\n",
"The faithful final stage succeeds only after retrieval and evidence quality have succeeded, so the rate is 0.9 times 0.95 times 0.98 = 0.8379. These are conditional factors from the probability chain rule; independence is not required.\n",
"\n",
"Show the answer
\n",
"\n",
"Availability is 1 - 0.7 squared = 0.51. The exact checker accepts 51% of batches, returns a correct answer on every accepted batch, and abstains otherwise. Both candidates are correct with probability 0.09; exactly one is correct with probability 0.42. The imperfect selector's success is 0.09 + 0.8 times 0.42 = 0.426, so v is about 0.8353. At p = 0 the exact checker accepts nothing; its conditional accuracy is undefined, not zero or one.\n",
"\n",
"Show the answer
\n",
"\n",
"Wald gives 0 to 0, a zero-width interval. The Wilson interval is about 0 to 0.161: the lower limit is 0 and the upper limit is positive, because 20 questions cannot rule out a small true accuracy.\n",
"\n",
"Show the answer
\n",
"\n",
"You lose (0.6 - 0.8)^2 = 0.04. Check: honest -0.8 x 0.2 = -0.16; reported -(0.8 x 0.16 + 0.2 x 0.36) = -0.2; the difference is 0.04.\n",
"\n",
"Show the answer
\n",
"\n",
"The gap is 0.5 - 1.5 = -1, times 2 gives -2, and sigma(-2) = 1/(1 + e^2) = 0.119. The model is weaker than the item, so the chance is well below one half.\n",
"\n",
"Show the answer
\n",
"\n",
"The gap is 2, so X wins sigma(2) = 1/(1 + e^-2) = 0.881 of the time and Y wins 0.119. The two chances sum to 1.\n",
"\n",
"