{
"cells": [
{
"cell_type": "markdown",
"id": "689a49cd",
"metadata": {},
"source": [
"# Attention and Transformers\n",
"\n",
"*The Mathematics of Large Language Models ยท Chapter 8*\n",
"\n",
"Follow information through attention, sampling, position, and growing context.\n",
"\n",
"These pages illustrate the mathematics. Read the saved figures and calculations in order, or open [the interactive browser edition](reader.html) to change the examples. No code editing is needed.\n",
"\n",
"Identify whether the practical issue is missing context, different sampled wording, positional relationships, memory cost, speed or numeric precision. Use the relevant small calculation, state the assumptions, and propose a check such as supplying the needed evidence, comparing answers to the source, or reducing irrelevant context. Explain that low temperature does not verify claims."
]
},
{
"cell_type": "markdown",
"id": "05e5b409",
"metadata": {},
"source": [
"Reproducing these calculations
\n",
"\n",
"The optional calculation cells use Python with NumPy, SciPy, and Matplotlib. All inputs are included in this file. The figures below are saved, so running these cells is optional.\n",
"\n",
"Show the answer
\n",
"\n",
"The weights are (1, 0, 0), so the output is V1 = (1, 0). Future key 3 is masked before normalization and contributes zero.\n",
"\n",
"Show the answer
\n",
"\n",
"Its final probability is 0.4/0.8 = 0.5. This is its sampling probability among the retained tokens, with no factual verification implied.\n",
"\n",
"Show the answer
\n",
"\n",
"The raw score is cos(0.02) + cos(0.0002), about 1.9997999867. Dividing by sqrt(4) gives about 0.9998999933.\n",
"\n",
"Show the answer
\n",
"\n",
"There are 16 squared = 256 scores per head and 2 times 16 times 8 times 32 = 8,192 full-layer KV values. Doubling d leaves the score count unchanged while doubling KV storage and the QK-plus-AV arithmetic count.\n",
"\n",
"Show the answer
\n",
"\n",
"(1 - 0.5^4)/(1 - 0.5) = 0.9375/0.5 = 1.875 tokens. The ceiling as gamma grows is 1/(1 - 0.5) = 2.\n",
"\n",
"Show the answer
\n",
"\n",
"ln 2 / 0.125 = 0.693 / 0.125, about 5.5 tokens. Each factor-of-two drop in slope doubles the half-distance.\n",
"\n",
"Show the answer
\n",
"\n",
"s = sqrt(8) / sqrt(0.5) = 4. The activation range becomes 8/4 = 2 and the weight range 0.5 x 4 = 2.\n",
"\n",
"Show the answer
\n",
"\n",
"4 x (0.5 x 0.25 + 0.5 x 0.25) = 4 x 0.25 = 1. Even router weights give 1 whatever the traffic split, which is why the penalty acts through both f and p.\n",
"\n",
"