{
"cells": [
{
"cell_type": "markdown",
"id": "4a423443",
"metadata": {},
"source": [
"# Recurrent and State-Space Models\n",
"\n",
"*The Mathematics of Large Language Models ยท Chapter 7*\n",
"\n",
"Track decay, Jacobian products, gated retention, and convolutional memory.\n",
"\n",
"These pages illustrate the mathematics. Read the saved figures and calculations in order, or open [the interactive browser edition](reader.html) to change the examples. No code editing is needed.\n",
"\n",
"Request sampling interval, transition and input/output coefficients, initial state, input history, and whether parameters or gates vary. Return exact history weights, recurrence outputs, stability conditions and decay-envelope lookback when defined. For conversation memory, explain architectures without assuming a product implements them."
]
},
{
"cell_type": "markdown",
"id": "0febeeca",
"metadata": {},
"source": [
"Reproducing these calculations
\n",
"\n",
"The optional calculation cells use Python with NumPy, SciPy, and Matplotlib. All inputs are included in this file. The figures below are saved, so running these cells is optional.\n",
"\n",
"Show the answer
\n",
"\n",
"2 x (-0.5)^3 = -0.25. The size shrinks by half each step while the sign flips, so it fades; it is a signed weight, not an ordinary average.\n",
"\n",
"Show the answer
\n",
"\n",
"Lined up: 1.2^10 = 6.19. Alternating: 0.144^5 = 0.0000619. The ratio is 100000 (0.1^5 inverted), a gap that grows with every pair.\n",
"\n",
"Show the answer
\n",
"\n",
"0.9^10 = 0.349, about 35%. A gate of 0.9 sounds high, but it still loses two thirds of the value in 10 steps.\n",
"\n",
"Show the answer
\n",
"\n",
"C A-bar^(t+1) h(-1), the fading echo of the starting memory. The exponent is t + 1 because the first update already applies A-bar once.\n",
"\n",
"Show the answer
\n",
"\n",
"A degree m-1 polynomial has at most m-1 crossings, so m-1 must be at least 7 and m at least 8. That is necessary only; the shape must also fit.\n",
"\n",
"Show the answer
\n",
"\n",
"Euler needs |1 - 4 Delta| below 1, so Delta below 0.5. ZOH multiplies by e^(-4 Delta), always between 0 and 1, so it has no limit.\n",
"\n",
"Show the answer
\n",
"\n",
"e^(-Delta) is near 1, so the old state is kept almost unchanged, and the write 1 - e^(-Delta) is near 0, so the filler is skipped.\n",
"\n",
"Show the answer
\n",
"\n",
"No. |1 - lambda| is about 0.62, so the response is about 1.6. Only a positive real entry with the same size would give 100.\n",
"\n",
"