{
"cells": [
{
"cell_type": "markdown",
"id": "03a948b3b34a",
"metadata": {},
"source": [
"# Chapter 14: Building a World Inside\n",
"\n",
"A planner evaluates tomorrow through an approximate transition model. Small one-step errors can matter more when the controller looks farther ahead. Yet a large worst-case bound does not mean the actual planning error reached it. Both quantities deserve separate places in the report.\n",
"\n",
"This notebook compares two finite kernels under one fixed policy and one common immediate reward vector. It computes their transition mismatch, propagates finite-horizon values, and plots actual value error beside the chapter's conditional horizon bound. The changed case lengthens the horizon without changing the model. The transfer case makes the kernels identical, giving an exact zero-error check before any more complicated interpretation.\n",
"\n",
"**Outcome:** Calculate uniform transition error, actual fixed-policy finite values, and conditional bounds.\n",
"\n",
"- Inspect the declared input contract\n",
"- Predict the hand-checkable case\n",
"- Run the shared computation\n",
"- Change the critical assumption\n",
"- Apply the method to the transfer data\n",
"\n",
"**Guided route:** Run the worked calculation, inspect its figure, change the stated assumption, and try the transfer case. Read the explanations beside each result before opening the answers.\n",
"\n",
"**Deeper route:** First read the mathematics and canonical equation reference. Audit the input contract, predict the changed result, then inspect the shared chapter implementation and solve the questions independently. Both routes use the same calculations and preserve the equations."
]
},
{
"cell_type": "markdown",
"id": "a7781f3ee7dc",
"metadata": {},
"source": [
"## Technical Requirements\n",
"\n",
"Python 3.11 or later, the complete laboratory folder, and the notebook dependencies listed in `requirements-notebooks.txt` (the launcher's **Install notebook tools** choice installs them; see START-HERE). Standard-library chapter commands also support Python 3.10. No API key, model account or network call is used by this experiment.\n",
"\n",
"Prior knowledge:\n",
"\n",
"- Python lists and dictionaries\n",
"- The mapped chapter and its notation"
]
},
{
"cell_type": "markdown",
"id": "c5c3e4d9b3f6",
"metadata": {},
"source": [
"## The question and its mathematics\n",
"\n",
"For each state, total variation between the true and modeled transition rows is half the sum of absolute probability differences. Uniform error epsilon is the maximum of those row distances. An average row error cannot replace that maximum in a uniform bound.\n",
"\n",
"With common expected immediate rewards in [0,R], one fixed policy, and discount gamma<1, the chapter's discounted value bound is gamma epsilon R/(1-gamma)^2. This quantity applies under those prerequisites; it is not an estimate fitted to the finite error curve. Reward-model mismatch would need another term.\n",
"\n",
"For H rewards and zero terminal continuation, the undiscounted finite-horizon bound is R epsilon H(H-1)/2. It also bounds the discounted finite calculation because discounting can only reduce these nonnegative error contributions. The implementation computes V_h=r+gamma P V_(h-1) and modeled W_h=r+gamma Q W_(h-1), starting both at zero, then records max_s |V_h(s)-W_h(s)|.\n",
"\n",
"The kernels already absorb a fixed policy. A state-value bound alone does not provide the uniform action-value error and action gap needed to certify a choice ranking. Keep that missing prerequisite explicit."
]
},
{
"cell_type": "markdown",
"id": "b7d0db24f36c",
"metadata": {},
"source": [
"## A calculation you can run\n",
"\n",
"Provide two aligned square probability matrices and one nonnegative reward vector. Every row must sum to one. Declare discount and horizon. The function computes the uniform total-variation error, maximum reward, both finite value recurrences, the actual maximum error at each horizon, and the two conditional bounds.\n",
"\n",
"Figures use reward horizon on the horizontal axis and reward units on the vertical axis. Plotting bound and actual error separately avoids disguising a small error beneath a large envelope. In the changed case increase horizon from five to twenty and predict how the quadratic envelope changes. Inspect the actual curve independently; it need not grow at the same rate. Export common-reward and fixed-policy assumptions with the numerical result.\n",
"\n",
"The next cell finds the bundle and imports the same computation used by the chapter skill. It does not change your system Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "42ebefc0e5c1",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys, json\n",
"LAB_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"lab-manifest.json\").is_file()), None)\n",
"if LAB_ROOT is None:\n",
" raise RuntimeError(\"Open this notebook from the complete extracted laboratory folder.\")\n",
"sys.path.insert(0, str(LAB_ROOT / \"src\"))\n",
"from math_ai_agents.core import analyze, report_text\n",
"from math_ai_agents.plotting import figure_svg\n",
"from IPython.display import SVG, display\n"
]
},
{
"cell_type": "markdown",
"id": "7fffde8c0edb",
"metadata": {},
"source": [
"Set the declared inputs below. These are constructed teaching values, not measurements from a production agent. Change a value only after predicting what it should change."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "a09bdc7645c0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 14: transition-model-bound\n",
"How much value error can this approximate world model introduce?\n",
"Evidence: constructed teaching example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"uniform_tv_error\": 0.02,\n",
" \"true_values\": [\n",
" 0.55354113,\n",
" 2.98801774\n",
" ],\n",
" \"model_values\": [\n",
" 0.6403010023,\n",
" 2.921214829\n",
" ],\n",
" \"max_finite_value_error\": 0.08675987227,\n",
" \"discounted_infinite_bound\": 1.8,\n",
" \"finite_horizon_bound\": 0.2\n",
"}\n",
"\n",
"Interpretation:\n",
"Both kernels share the same immediate reward vector under one fixed policy. The uniform total-variation error controls the conditional bound; the actual finite error can be much smaller.\n",
"\n",
"Assumptions:\n",
"- Fixed policy absorbed into each transition matrix.\n",
"- Common expected immediate reward in [0,R].\n",
"- Finite calculation has zero terminal continuation; discounted bound assumes gamma below one.\n",
"\n",
"Limitations:\n",
"- A bound is not a measured planning error.\n",
"- This state-value bound alone does not certify an action ranking.\n",
"- Reward-model error would require an additional term.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"chapter = 14\n",
"inputs = {'true_transition': [[0.9, 0.1], [0.2, 0.8]],\n",
" 'model_transition': [[0.88, 0.12], [0.22, 0.78]],\n",
" 'rewards': [0, 1],\n",
" 'discount': 0.9,\n",
" 'horizon': 5}\n",
"report = analyze(chapter, inputs)\n",
"# This input was explicitly taken from the teaching fixture.\n",
"report['evidence_kind'] = 'constructed teaching example'\n",
"print(report_text(report))"
]
},
{
"cell_type": "markdown",
"id": "0fbf4dcdb600",
"metadata": {},
"source": [
"Both default row distances are 0.02, so epsilon=0.02. With R=1 and gamma=0.9, the discounted infinite bound is 1.8. At horizon 5 the finite bound is 0.02(5)(4)/2=0.2. The actual finite error is calculated from the two recurrences and must remain below the finite envelope.\n",
"\n",
"At horizon 20 the finite bound grows to 3.8, even though each transition row changed by the same amount. That growth is a sensitivity statement under a worst-case construction. It does not say that the actual error became 3.8, and it does not establish that any particular action ranking reversed.\n",
"\n",
"Two bounds are reported and both are valid. At horizon 20 the finite-horizon bound (3.8) is the looser one and the discounted infinite-horizon bound (1.8) the tighter, so the tightest valid statement is their minimum. The finite bound is not a stronger warning.\n",
"\n",
"The plot below uses the calculated quantities. Read each panel's units before comparing its values."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "590f57e74048",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Matplotlib is building the font cache; this may take a moment.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(SVG(figure_svg(report)))"
]
},
{
"cell_type": "markdown",
"id": "1169a1f69a1e",
"metadata": {},
"source": [
"**Figure 14.L1:** Calculated chapter experiment. Each panel labels its input and output units; interpret it under the assumptions printed in the report."
]
},
{
"cell_type": "markdown",
"id": "4b3b8fdec2ca",
"metadata": {},
"source": [
"## Change the assumption\n",
"\n",
"A common misuse inserts average model error into a uniform bound. A model may be accurate on frequently observed states and badly wrong at the one state that changes the plan. The maximum row distance protects the stated domain, while an average describes a different estimand.\n",
"\n",
"Another misuse compares reward functions that differ and then attributes all value error to transition mismatch. This input contract intentionally supplies only one common reward vector. If the model invents immediate reward, the implementation's bound assumptions no longer apply.\n",
"\n",
"Finally, an error bound is not a decision certificate. Even a small value bound can overwhelm a smaller action gap, and this fixed-policy state calculation does not inspect alternative actions. Use it to audit model sensitivity, then state the extra action-value and gap information required for ranking guarantees."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "d2f981055e16",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 14: transition-model-bound\n",
"How much value error can this approximate world model introduce?\n",
"Evidence: constructed changed-assumption example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"uniform_tv_error\": 0.02,\n",
" \"true_values\": [\n",
" 2.027264312,\n",
" 4.729704829\n",
" ],\n",
" \"model_values\": [\n",
" 2.231030443,\n",
" 4.694010975\n",
" ],\n",
" \"max_finite_value_error\": 0.2037661306,\n",
" \"discounted_infinite_bound\": 1.8,\n",
" \"finite_horizon_bound\": 3.8\n",
"}\n",
"\n",
"Interpretation:\n",
"Both kernels share the same immediate reward vector under one fixed policy. The uniform total-variation error controls the conditional bound; the actual finite error can be much smaller.\n",
"\n",
"Assumptions:\n",
"- Fixed policy absorbed into each transition matrix.\n",
"- Common expected immediate reward in [0,R].\n",
"- Finite calculation has zero terminal continuation; discounted bound assumes gamma below one.\n",
"\n",
"Limitations:\n",
"- A bound is not a measured planning error.\n",
"- This state-value bound alone does not certify an action ranking.\n",
"- Reward-model error would require an additional term.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"changed_inputs = {'true_transition': [[0.9, 0.1], [0.2, 0.8]],\n",
" 'model_transition': [[0.88, 0.12], [0.22, 0.78]],\n",
" 'rewards': [0, 1],\n",
" 'discount': 0.9,\n",
" 'horizon': 20}\n",
"changed = analyze(chapter, changed_inputs)\n",
"changed['evidence_kind'] = 'constructed changed-assumption example'\n",
"print(report_text(changed))\n",
"display(SVG(figure_svg(changed)))"
]
},
{
"cell_type": "markdown",
"id": "8e65b81fc527",
"metadata": {},
"source": [
"**Figure 14.L2:** The changed-assumption result. Compare the printed quantities and the stated assumptions with the first run. A different input need not imply a causal effect in a deployed agent."
]
},
{
"cell_type": "markdown",
"id": "a875d7a8d653",
"metadata": {},
"source": [
"## Try a new case\n",
"\n",
"The transfer kernels are identical. Epsilon is zero, finite values agree at every stage, and both bounds are zero. This identity is a useful check of matrix alignment and recurrence behavior. If an implementation returned nonzero error here, investigate the computation before interpreting a more realistic model.\n",
"\n",
"For reader data, explain how the true kernel was measured or constructed. Usually a deployed world does not provide exact transition probabilities. In that case the calculated epsilon is conditional on estimates and their support, not a known universal error. Preserve uncertainty or specify the measurements needed to bound it on the relevant policy domain."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "9581f4057d65",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 14: transition-model-bound\n",
"How much value error can this approximate world model introduce?\n",
"Evidence: constructed transfer example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"uniform_tv_error\": 0.0,\n",
" \"true_values\": [\n",
" 1.875,\n",
" 3.75\n",
" ],\n",
" \"model_values\": [\n",
" 1.875,\n",
" 3.75\n",
" ],\n",
" \"max_finite_value_error\": 0.0,\n",
" \"discounted_infinite_bound\": 0.0,\n",
" \"finite_horizon_bound\": 0.0\n",
"}\n",
"\n",
"Interpretation:\n",
"Both kernels share the same immediate reward vector under one fixed policy. The uniform total-variation error controls the conditional bound; the actual finite error can be much smaller.\n",
"\n",
"Assumptions:\n",
"- Fixed policy absorbed into each transition matrix.\n",
"- Common expected immediate reward in [0,R].\n",
"- Finite calculation has zero terminal continuation; discounted bound assumes gamma below one.\n",
"\n",
"Limitations:\n",
"- A bound is not a measured planning error.\n",
"- This state-value bound alone does not certify an action ranking.\n",
"- Reward-model error would require an additional term.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"transfer_inputs = {'true_transition': [[1, 0], [0, 1]],\n",
" 'model_transition': [[1, 0], [0, 1]],\n",
" 'rewards': [1, 2],\n",
" 'discount': 0.5,\n",
" 'horizon': 4}\n",
"transfer = analyze(chapter, transfer_inputs)\n",
"transfer['evidence_kind'] = 'constructed transfer example'\n",
"print(report_text(transfer))"
]
},
{
"cell_type": "markdown",
"id": "f73c717a106e",
"metadata": {},
"source": [
"## Apply the method to your inputs\n",
"\n",
"The example file below has the exact input shape the method accepts. Copy it to a new file, replace its values, then point `reader_file` at your copy. Run the cell again. Supplied inputs retain their stated provenance; the program cannot establish that they are representative observations.\n",
"\n",
"- **true_transition:** Square row-stochastic true kernel after fixing a policy.\n",
"- **model_transition:** Same-size approximate row-stochastic kernel for that policy.\n",
"- **rewards:** Common immediate state reward vector, finite and nonnegative.\n",
"- **discount:** Gamma in [0,0.9999].\n",
"- **horizon:** Integer reward horizon 1..1000 with zero terminal continuation."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "5e78438044b6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 14: transition-model-bound\n",
"How much value error can this approximate world model introduce?\n",
"Evidence: supplied local inputs; provenance not independently verified\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"uniform_tv_error\": 0.0,\n",
" \"true_values\": [\n",
" 1.875,\n",
" 3.75\n",
" ],\n",
" \"model_values\": [\n",
" 1.875,\n",
" 3.75\n",
" ],\n",
" \"max_finite_value_error\": 0.0,\n",
" \"discounted_infinite_bound\": 0.0,\n",
" \"finite_horizon_bound\": 0.0\n",
"}\n",
"\n",
"Interpretation:\n",
"Both kernels share the same immediate reward vector under one fixed policy. The uniform total-variation error controls the conditional bound; the actual finite error can be much smaller.\n",
"\n",
"Assumptions:\n",
"- Fixed policy absorbed into each transition matrix.\n",
"- Common expected immediate reward in [0,R].\n",
"- Finite calculation has zero terminal continuation; discounted bound assumes gamma below one.\n",
"\n",
"Limitations:\n",
"- A bound is not a measured planning error.\n",
"- This state-value bound alone does not certify an action ranking.\n",
"- Reward-model error would require an additional term.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"reader_file = LAB_ROOT / 'data/examples/ch14.json'\n",
"reader_inputs = json.loads(reader_file.read_text())\n",
"reader_report = analyze(chapter, reader_inputs)\n",
"print(report_text(reader_report))"
]
},
{
"cell_type": "markdown",
"id": "51f32353714b",
"metadata": {},
"source": [
"## Questions\n",
"\n",
"1. Compute default epsilon.\n",
"\n",
"2. Compute horizon 20 finite bound.\n",
"\n",
"3. What if the reward model also differs?\n",
"\n",
"Answers: [separate solutions](../solutions/ch14.md). Try the calculation before opening them."
]
},
{
"cell_type": "markdown",
"id": "9629bab42f4e",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"Transition models accumulate errors through future value. Uniform total variation, common immediate rewards, and a fixed policy are essential to the implemented bounds. The notebook separates actual finite error from conditional envelopes and checks the identical-kernel case. Extending horizon changes sensitivity without changing one-step mismatch. A bound is neither a measured planning error nor an action-ranking certificate; report its prerequisites and the evidence supporting the supplied kernels.\n",
"\n",
"Limits of this experiment:\n",
"\n",
"- This is a local calculation under declared inputs, not an empirical claim about a deployed agent.\n",
"- Read the returned assumptions and limitations before applying the numerical result.\n",
"\n",
"The assistant skill is [`maa-14-transition-model-bound`](../skills/maa-14-transition-model-bound/SKILL.md). It uses this notebook's tested computation and input contract."
]
},
{
"cell_type": "markdown",
"id": "9cc9f25ba898",
"metadata": {},
"source": [
"## Equations from the chapter\n",
"\n",
"These are the unchanged display equations and their explanations from the canonical chapter. They are a reference for the experiment, not a claim that every equation is numerically implemented by this one method."
]
},
{
"cell_type": "markdown",
"id": "bf22b8c0dc5d",
"metadata": {},
"source": [
"### Equation 14.1\n",
"\n",
"\n",
"\n",
"Equation (14.1) measures how wrong the model is about a single step, at its worst state and action.\n",
"\n",
"For each state and action, add up how much the two distributions disagree, halve it, and take the largest such disagreement anywhere.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\epsilon \\;=\\; \\max_{x,a}\\; \\tfrac{1}{2}\\sum_{x'}\\big|\\hat P(x'\\mid x,a) - P(x'\\mid x,a)\\big|.\n",
"\\tag{14.1}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "62317d9cb72c",
"metadata": {},
"source": [
"### Equation 14.2\n",
"\n",
"\n",
"\n",
"Equation (14.2) bounds how far the value computed inside the model can sit from the value in the world.\n",
"\n",
"Multiply the one-step error by the discount factor and largest reward, then divide by the square of one minus the discount factor.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\big|\\hat V^{\\pi}(x) - V^{\\pi}(x)\\big| \\;\\le\\; \\frac{\\gamma\\,\\epsilon\\,R}{(1-\\gamma)^2}.\n",
"\\tag{14.2}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "6c8284c7758c",
"metadata": {},
"source": [
"### Equation 14.3\n",
"\n",
"\n",
"\n",
"Equation (14.3) measures how much better the best action at a state is than the runner-up.\n",
"\n",
"Subtract the value of the second best action from the value of the best one.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\Delta(x) \\;=\\; \\max_{a}Q^{\\pi}(x,a) \\;-\\; \\max_{a \\neq a^{*}}Q^{\\pi}(x,a).\n",
"\\tag{14.3}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "6dca0c956dfe",
"metadata": {},
"source": [
"### Equation 14.4\n",
"\n",
"\n",
"\n",
"Equation (14.4) gives the largest positive integer horizon certified to preserve a unique action ranking under the stated finite-model assumptions.\n",
"\n",
"Within the declared planning cap `H_{\\max}`, find the largest positive integer horizon at which twice the finite-horizon action-value error still fits inside the action gap.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"H^{*} \\;=\\; \\max\\Big\\{ H \\in \\mathbb{Z}_{>0} \\;:\\; H \\leq H_{\\max},\\;\n",
"R\\,\\epsilon\\,H(H-1) \\;<\\; \\Delta \\Big\\}.\n",
"\\tag{14.4}\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Mathematics of AI Agents",
"language": "python",
"name": "maa-lab"
},
"lab_chapter": 14,
"lab_execution": {
"code_cells": 6,
"created_utc": "2026-10-02T05:15:00.798566+00:00",
"elapsed_seconds": 8.909545915899798,
"method": "fresh process; new ipykernel InProcessKernelManager; cells submitted as Jupyter execute requests",
"network_transport_tested": false,
"python": "3.11.15",
"source_sha256": "16ca0c2ea1bd24b7abbb889350e2d2d7ef93ba8b26b233be0cd813942b067cfe"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}