{
"cells": [
{
"cell_type": "markdown",
"id": "7393e100",
"metadata": {},
"source": [
"# Continuous-Depth and Implicit Models\n",
"\n",
"*The Mathematics of Large Language Models ยท Chapter 12*\n",
"\n",
"Separate equilibria, convergence and numerical artifacts.\n",
"\n",
"These pages illustrate the mathematics. Read the saved figures and calculations in order, or open [the interactive browser edition](reader.html) to change the examples. No code editing is needed.\n",
"\n",
"Request the exact recurrence or differential equation, parameter values, input, initial state, time horizon and numerical step. Solve fixed points before assessing iteration convergence; for z_next=theta z+x report x/(1-theta) when theta differs from one, handle theta=1 separately and test |theta|<1 for global contraction. Return a trajectory, residual, sensitivity and a before/after perturbation calculation. For ODEs distinguish mathematical stability from the chosen solver, name the solver and its order, compare the step with 2 divided by the fastest rate before trusting an explicit method, and name the update equations rather than saying only symplectic."
]
},
{
"cell_type": "markdown",
"id": "6aa8be0b",
"metadata": {},
"source": [
"Reproducing these calculations
\n",
"\n",
"The optional calculation cells use Python with NumPy, SciPy, and Matplotlib. All inputs are included in this file. The figures below are saved, so running these cells is optional.\n",
"\n",
"Show the answer
\n",
"\n",
"The factor is 1 - 1.5 = -0.5. Its size is below one, so it shrinks, but it alternates sign, unlike the exact positive decay.\n",
"\n",
"Show the answer
\n",
"\n",
"With x = 0 every value is fixed, so the start never moves. With x = 1 no fixed point exists and the value grows by one each round.\n",
"\n",
"Show the answer
\n",
"\n",
"It is -2/(0.5)^2 = -8, so increasing theta slightly lowers the equilibrium. The size grows near one; the sign stays negative.\n",
"\n",
"Show the answer
\n",
"\n",
"Each step multiplies the energy by 1 + 0.25 = 1.25, so after two steps it is 0.5 x 1.25^2 = 0.78125.\n",
"\n",
"Show the answer
\n",
"\n",
"No. Staying on the axis makes it a one-dimensional flow again, and order cannot change. The freedom comes from leaving the line.\n",
"\n",
"Show the answer
\n",
"\n",
"About 1/16 of it, near 0.00000092. The measured value is 0.00000083, close to this.\n",
"\n",
"Show the answer
\n",
"\n",
"About 1,001 forward solves, one baseline plus one nudged run per parameter. The adjoint method needs one forward and one backward solve, whatever the number of parameters.\n",
"\n",
"Show the answer
\n",
"\n",
"The largest step is 2/500 = 0.004, so 2/0.004 = 500 steps. Implicit Euler with h = 0.1 needs only 20 steps and stays stable.\n",
"\n",
"