{
"cells": [
{
"cell_type": "markdown",
"id": "1d1bfee532b3",
"metadata": {},
"source": [
"# Chapter 12: Credit for Consequences\n",
"\n",
"The draft step earns no immediate reward. The review step produces the eventual benefit. If the controller learns only from immediate outcomes, it may never give preparation the credit it deserves. If it waits for every complete episode, it may learn slowly or fail to use partial traces.\n",
"\n",
"This notebook puts three credit rules beside the same trajectory. Monte Carlo uses the discounted later return. TD(0) bootstraps from the next state's current value. Eligibility traces spread a later TD error backward through recently visited states. The changed case marks the end as truncation rather than true termination, exposing the value that can be lost when those two boundaries are confused.\n",
"\n",
"**Outcome:** Compare Monte Carlo, TD(0), and accumulating TD(lambda) updates on one trajectory.\n",
"\n",
"- Inspect the declared input contract\n",
"- Predict the hand-checkable case\n",
"- Run the shared computation\n",
"- Change the critical assumption\n",
"- Apply the method to the transfer data\n",
"\n",
"**Guided route:** Run the worked calculation, inspect its figure, change the stated assumption, and try the transfer case. Read the explanations beside each result before opening the answers.\n",
"\n",
"**Deeper route:** First read the mathematics and canonical equation reference. Audit the input contract, predict the changed result, then inspect the shared chapter implementation and solve the questions independently. Both routes use the same calculations and preserve the equations."
]
},
{
"cell_type": "markdown",
"id": "a7781f3ee7dc",
"metadata": {},
"source": [
"## Technical Requirements\n",
"\n",
"Python 3.11 or later, the complete laboratory folder, and the notebook dependencies listed in `requirements-notebooks.txt` (the launcher's **Install notebook tools** choice installs them; see START-HERE). Standard-library chapter commands also support Python 3.10. No API key, model account or network call is used by this experiment.\n",
"\n",
"Prior knowledge:\n",
"\n",
"- Python lists and dictionaries\n",
"- The mapped chapter and its notation"
]
},
{
"cell_type": "markdown",
"id": "2c3007cd2922",
"metadata": {},
"source": [
"## The question and its mathematics\n",
"\n",
"For a complete trajectory, the return target is G_t=r_t+gamma G_(t+1), with terminal continuation zero. For a truncated trace, this implementation initializes the final continuation from the supplied value estimate instead. The backward recurrence makes that distinction visible in every earlier target.\n",
"\n",
"Monte Carlo updates V(s_t) by alpha[G_t-V(s_t)]. TD(0) uses delta_t=r_t+gamma V(s_(t+1))-V(s_t), replacing the next value with zero only at true terminal transition. Accumulating TD(lambda) first adds one to the current state's eligibility, updates every value by alpha delta_t e_t(s), then decays eligibility by gamma lambda.\n",
"\n",
"All three rules receive the same reward sequence and initial values, but their online updates differ. A state revisited within the trace can accumulate eligibility or receive multiple updates. This notebook implements one sequential pass, not an asymptotic convergence experiment. Lambda controls how far a later error reaches backward; it does not manufacture missing future rewards.\n",
"\n",
"How this relates to Equation (12.4). The chapter writes the forward view: each visited state moves toward its lambda-return. This notebook implements the backward view, accumulating eligibility traces applied online during one pass. For a terminal episode the two give identical updates when the updates are offline, meaning the value estimates are held fixed during the episode and the summed increments are applied at the end. With online updates, as here, they still agree exactly when no state repeats within the episode, because every update touches only states already visited and no later TD error reads them. When a state repeats, online updates change estimates that later errors use, and the result can differ from the Equation (12.4) update. The default trajectory has no repeated state, so its values 0.4 and 0.5 equal the forward-view updates."
]
},
{
"cell_type": "markdown",
"id": "7200038ca259",
"metadata": {},
"source": [
"## A calculation you can run\n",
"\n",
"Give the ordered state list, rewards, initial values, discount, learning rate, lambda, and exact terminal flag. The state list must have one more element than the reward list. Values must exist for every named state. The code computes backward return targets, then applies each update rule to its own independent copy of the initial value map.\n",
"\n",
"The return plot shows the target associated with each transition, while the trace-error plot shows the online TD errors used by eligibility updates. Consult the three final value maps to compare credit assignment. In the changed case the final state's estimate is 2 and terminal is false. Predict how that bootstrap changes the targets before running the calculation.\n",
"\n",
"The next cell finds the bundle and imports the same computation used by the chapter skill. It does not change your system Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "42ebefc0e5c1",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys, json\n",
"LAB_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"lab-manifest.json\").is_file()), None)\n",
"if LAB_ROOT is None:\n",
" raise RuntimeError(\"Open this notebook from the complete extracted laboratory folder.\")\n",
"sys.path.insert(0, str(LAB_ROOT / \"src\"))\n",
"from math_ai_agents.core import analyze, report_text\n",
"from math_ai_agents.plotting import figure_svg\n",
"from IPython.display import SVG, display\n"
]
},
{
"cell_type": "markdown",
"id": "7fffde8c0edb",
"metadata": {},
"source": [
"Set the declared inputs below. These are constructed teaching values, not measurements from a production agent. Change a value only after predicting what it should change."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "0fd5c55fd2f8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 12: trajectory-credit\n",
"How should a late consequence change values assigned to earlier states?\n",
"Evidence: constructed teaching example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"returns\": [\n",
" 1.0,\n",
" 1.0\n",
" ],\n",
" \"monte_carlo_values\": {\n",
" \"review\": 0.5,\n",
" \"done\": 0.0,\n",
" \"draft\": 0.5\n",
" },\n",
" \"td_zero_values\": {\n",
" \"review\": 0.5,\n",
" \"done\": 0.0,\n",
" \"draft\": 0.0\n",
" },\n",
" \"td_lambda_values\": {\n",
" \"review\": 0.5,\n",
" \"done\": 0.0,\n",
" \"draft\": 0.4\n",
" },\n",
" \"bootstrap_value\": 0\n",
"}\n",
"\n",
"Interpretation:\n",
"All three estimators see the same rewards. A truncated trajectory retains its supplied final bootstrap value; a terminal trajectory sets that continuation to zero.\n",
"\n",
"Assumptions:\n",
"- One sequential pass with accumulating eligibility traces.\n",
"- Initial values, discount, step size, and terminal status are declared.\n",
"\n",
"Limitations:\n",
"- This finite update does not demonstrate convergence.\n",
"- Bootstrapping trades dependence on later observations for dependence on current estimates.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"chapter = 12\n",
"inputs = {'states': ['draft', 'review', 'done'],\n",
" 'rewards': [0, 1],\n",
" 'discount': 1,\n",
" 'learning_rate': 0.5,\n",
" 'lambda': 0.8,\n",
" 'terminal': True,\n",
" 'values': {'draft': 0, 'review': 0, 'done': 0}}\n",
"report = analyze(chapter, inputs)\n",
"# This input was explicitly taken from the teaching fixture.\n",
"report['evidence_kind'] = 'constructed teaching example'\n",
"print(report_text(report))"
]
},
{
"cell_type": "markdown",
"id": "46c0ef955e06",
"metadata": {},
"source": [
"Default returns are [1,1]. With alpha=0.5, Monte Carlo assigns draft and review values 0.5. TD(0) updates draft from a zero next value, leaving it at zero, then assigns review 0.5. TD(lambda) carries the later error back with eligibility 0.8, giving draft 0.4 and review 0.5.\n",
"\n",
"In the changed truncated case, final bootstrap is 2. Returns become [3,3]. Treating that trace as terminal would incorrectly erase the supplied continuation estimate. Whether 2 is a good estimate is a separate question; the credit arithmetic simply makes its dependence explicit.\n",
"\n",
"The plot below uses the calculated quantities. Read each panel's units before comparing its values."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "590f57e74048",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Matplotlib is building the font cache; this may take a moment.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(SVG(figure_svg(report)))"
]
},
{
"cell_type": "markdown",
"id": "5d6b177a6a53",
"metadata": {},
"source": [
"**Figure 12.L1:** Calculated chapter experiment. Each panel labels its input and output units; interpret it under the assumptions printed in the report."
]
},
{
"cell_type": "markdown",
"id": "eb8db8e63760",
"metadata": {},
"source": [
"## Change the assumption\n",
"\n",
"Bootstrapping can propagate a bad value estimate. Monte Carlo avoids that particular dependence in complete episodes, but it depends on the observed later return and can have high sampling variability. Neither target dominates in every setting.\n",
"\n",
"The terminal flag is especially important in limited-budget logs. An observation window ending is not proof that the underlying task terminated. Setting terminal true after a timeout can systematically suppress continuation value. Setting it false at actual termination can add imaginary future reward.\n",
"\n",
"Trace conventions also matter. Replacing traces and accumulating traces are different algorithms when states repeat. This implementation uses accumulating traces. Compare it with equations under that convention, and avoid claiming convergence from one pass over a short supplied trajectory. Learning-rate schedules, visitation, and process assumptions would need their own study."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "9a5dd2096dfc",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 12: trajectory-credit\n",
"How should a late consequence change values assigned to earlier states?\n",
"Evidence: constructed changed-assumption example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"returns\": [\n",
" 3.0,\n",
" 3.0\n",
" ],\n",
" \"monte_carlo_values\": {\n",
" \"review\": 1.5,\n",
" \"done\": 2.0,\n",
" \"draft\": 1.5\n",
" },\n",
" \"td_zero_values\": {\n",
" \"review\": 1.5,\n",
" \"done\": 2.0,\n",
" \"draft\": 0.0\n",
" },\n",
" \"td_lambda_values\": {\n",
" \"review\": 1.5,\n",
" \"done\": 2.0,\n",
" \"draft\": 1.2\n",
" },\n",
" \"bootstrap_value\": 2.0\n",
"}\n",
"\n",
"Interpretation:\n",
"All three estimators see the same rewards. A truncated trajectory retains its supplied final bootstrap value; a terminal trajectory sets that continuation to zero.\n",
"\n",
"Assumptions:\n",
"- One sequential pass with accumulating eligibility traces.\n",
"- Initial values, discount, step size, and terminal status are declared.\n",
"\n",
"Limitations:\n",
"- This finite update does not demonstrate convergence.\n",
"- Bootstrapping trades dependence on later observations for dependence on current estimates.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"changed_inputs = {'states': ['draft', 'review', 'done'],\n",
" 'rewards': [0, 1],\n",
" 'discount': 1,\n",
" 'learning_rate': 0.5,\n",
" 'lambda': 0.8,\n",
" 'terminal': False,\n",
" 'values': {'draft': 0, 'review': 0, 'done': 2}}\n",
"changed = analyze(chapter, changed_inputs)\n",
"changed['evidence_kind'] = 'constructed changed-assumption example'\n",
"print(report_text(changed))\n",
"display(SVG(figure_svg(changed)))"
]
},
{
"cell_type": "markdown",
"id": "000cb42319e6",
"metadata": {},
"source": [
"**Figure 12.L2:** The changed-assumption result. Compare the printed quantities and the stated assumptions with the first run. A different input need not imply a causal effect in a deployed agent."
]
},
{
"cell_type": "markdown",
"id": "b2859e5ce987",
"metadata": {},
"source": [
"## Try a new case\n",
"\n",
"The transfer case sets lambda to zero. Accumulating TD(lambda) then updates only the currently eligible state before immediate decay, so its final values should agree with TD(0) under the same sequential convention. This is an independent structural check rather than a visual similarity claim.\n",
"\n",
"For local use, preserve reward timing and state identity, and mark whether the final boundary is real termination or missing observation. If the final value is a fitted estimate, record its provenance. A trajectory-credit report should show targets and bootstrap assumptions as well as updated values, so a surprising change can be traced to evidence rather than hidden continuation."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "3d4ebad464a9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 12: trajectory-credit\n",
"How should a late consequence change values assigned to earlier states?\n",
"Evidence: constructed transfer example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"returns\": [\n",
" 2.62,\n",
" 1.8,\n",
" 2.0\n",
" ],\n",
" \"monte_carlo_values\": {\n",
" \"d\": 0.0,\n",
" \"c\": 0.4,\n",
" \"a\": 0.524,\n",
" \"b\": 1.16\n",
" },\n",
" \"td_zero_values\": {\n",
" \"d\": 0.0,\n",
" \"c\": 0.4,\n",
" \"a\": 0.38,\n",
" \"b\": 0.8\n",
" },\n",
" \"td_lambda_values\": {\n",
" \"d\": 0.0,\n",
" \"c\": 0.4,\n",
" \"a\": 0.38,\n",
" \"b\": 0.8\n",
" },\n",
" \"bootstrap_value\": 0\n",
"}\n",
"\n",
"Interpretation:\n",
"All three estimators see the same rewards. A truncated trajectory retains its supplied final bootstrap value; a terminal trajectory sets that continuation to zero.\n",
"\n",
"Assumptions:\n",
"- One sequential pass with accumulating eligibility traces.\n",
"- Initial values, discount, step size, and terminal status are declared.\n",
"\n",
"Limitations:\n",
"- This finite update does not demonstrate convergence.\n",
"- Bootstrapping trades dependence on later observations for dependence on current estimates.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"transfer_inputs = {'states': ['a', 'b', 'c', 'd'],\n",
" 'rewards': [1, 0, 2],\n",
" 'discount': 0.9,\n",
" 'learning_rate': 0.2,\n",
" 'lambda': 0,\n",
" 'terminal': True,\n",
" 'values': {'a': 0, 'b': 1, 'c': 0, 'd': 0}}\n",
"transfer = analyze(chapter, transfer_inputs)\n",
"transfer['evidence_kind'] = 'constructed transfer example'\n",
"print(report_text(transfer))"
]
},
{
"cell_type": "markdown",
"id": "ca6668fc3ccb",
"metadata": {},
"source": [
"## Apply the method to your inputs\n",
"\n",
"The example file below has the exact input shape the method accepts. Copy it to a new file, replace its values, then point `reader_file` at your copy. Run the cell again. Supplied inputs retain their stated provenance; the program cannot establish that they are representative observations.\n",
"\n",
"- **states:** Trajectory state identifiers, one more than reward count.\n",
"- **rewards:** Finite observed immediate rewards.\n",
"- **discount:** Gamma in [0,1].\n",
"- **learning_rate:** Step size in [0,1].\n",
"- **lambda:** Trace decay parameter in [0,1].\n",
"- **terminal:** Exact boolean distinguishing true termination from truncation.\n",
"- **values:** Initial finite value for every trajectory state."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "744d6c133898",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 12: trajectory-credit\n",
"How should a late consequence change values assigned to earlier states?\n",
"Evidence: supplied local inputs; provenance not independently verified\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"returns\": [\n",
" 2.62,\n",
" 1.8,\n",
" 2.0\n",
" ],\n",
" \"monte_carlo_values\": {\n",
" \"d\": 0.0,\n",
" \"c\": 0.4,\n",
" \"a\": 0.524,\n",
" \"b\": 1.16\n",
" },\n",
" \"td_zero_values\": {\n",
" \"d\": 0.0,\n",
" \"c\": 0.4,\n",
" \"a\": 0.38,\n",
" \"b\": 0.8\n",
" },\n",
" \"td_lambda_values\": {\n",
" \"d\": 0.0,\n",
" \"c\": 0.4,\n",
" \"a\": 0.38,\n",
" \"b\": 0.8\n",
" },\n",
" \"bootstrap_value\": 0\n",
"}\n",
"\n",
"Interpretation:\n",
"All three estimators see the same rewards. A truncated trajectory retains its supplied final bootstrap value; a terminal trajectory sets that continuation to zero.\n",
"\n",
"Assumptions:\n",
"- One sequential pass with accumulating eligibility traces.\n",
"- Initial values, discount, step size, and terminal status are declared.\n",
"\n",
"Limitations:\n",
"- This finite update does not demonstrate convergence.\n",
"- Bootstrapping trades dependence on later observations for dependence on current estimates.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"reader_file = LAB_ROOT / 'data/examples/ch12.json'\n",
"reader_inputs = json.loads(reader_file.read_text())\n",
"reader_report = analyze(chapter, reader_inputs)\n",
"print(report_text(reader_report))"
]
},
{
"cell_type": "markdown",
"id": "75ac82e79fb8",
"metadata": {},
"source": [
"## Questions\n",
"\n",
"1. Compute default trace update to draft.\n",
"\n",
"2. What are changed return targets?\n",
"\n",
"3. What should lambda = 0 match?\n",
"\n",
"Answers: [separate solutions](../solutions/ch12.md). Try the calculation before opening them."
]
},
{
"cell_type": "markdown",
"id": "100563c39e03",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"Credit rules differ in which future information they use. Monte Carlo follows the observed return, TD(0) bootstraps locally, and eligibility traces spread later errors through recent states. Terminal versus truncated boundaries alter continuation. The default is hand-checkable, the changed case exposes bootstrap dependence, and lambda zero checks the transfer identity. One finite update demonstrates mechanics, not convergence or reliable learning in a deployed agent.\n",
"\n",
"Limits of this experiment:\n",
"\n",
"- This is a local calculation under declared inputs, not an empirical claim about a deployed agent.\n",
"- Read the returned assumptions and limitations before applying the numerical result.\n",
"\n",
"The assistant skill is [`maa-12-trajectory-credit`](../skills/maa-12-trajectory-credit/SKILL.md). It uses this notebook's tested computation and input contract."
]
},
{
"cell_type": "markdown",
"id": "9cc9f25ba898",
"metadata": {},
"source": [
"## Equations from the chapter\n",
"\n",
"These are the unchanged display equations and their explanations from the canonical chapter. They are a reference for the experiment, not a claim that every equation is numerically implemented by this one method."
]
},
{
"cell_type": "markdown",
"id": "6992ac5ab549",
"metadata": {},
"source": [
"### Equation 12.1\n",
"\n",
"\n",
"\n",
"Equation (12.1) drags each state's estimate toward the return that actually followed it.\n",
"\n",
"Take the gap between what happened and what was predicted, keep a fraction of it, and add that to the prediction.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\hat V_{t+1}(x_t) \\;=\\; \\hat V_t(x_t) \\;+\\; \\alpha\\big[\\operatorname{Ret}_t - \\hat V_t(x_t)\\big].\n",
"\\tag{12.1}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "3dd03bff3e95",
"metadata": {},
"source": [
"### Equation 12.2\n",
"\n",
"\n",
"\n",
"Equation (12.2) measures how much the agent's own opinion changed between one step and the next.\n",
"\n",
"Compare what was predicted from here against the reward received plus what is predicted from there.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\delta_t \\;=\\; r_t \\;+\\; \\gamma\\,\\hat V_t(x_{t+1}) \\;-\\; \\hat V_t(x_t).\n",
"\\tag{12.2}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "9aa5a4cf597c",
"metadata": {},
"source": [
"### Equation 12.3\n",
"\n",
"\n",
"\n",
"Equation (12.3) moves each estimate toward the estimate that came immediately after it.\n",
"\n",
"Nudge the earlier prediction in the direction its own successor disagreed with it.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\hat V_{t+1}(x_t) \\;=\\; \\hat V_t(x_t) \\;+\\; \\alpha\\,\\delta_t.\n",
"\\tag{12.3}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "a314604b700c",
"metadata": {},
"source": [
"### Equation 12.4\n",
"\n",
"\n",
"\n",
"Equation (12.4) combines geometrically weighted intermediate targets with the remaining weight on the complete terminal return.\n",
"\n",
"Weight intermediate lookahead targets geometrically, then place all remaining weight on the realized return through termination.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{Ret}^{\\lambda}_t \\;=\\; (1-\\lambda)\\sum_{k=1}^{N-t-1}\\lambda^{\\,k-1}\\Big[\\textstyle\\sum_{j=0}^{k-1}\\gamma^{\\,j} r_{t+j} \\;+\\; \\gamma^{\\,k}\\hat V_t(x_{t+k})\\Big] \\;+\\; \\lambda^{N-t-1}\\operatorname{Ret}_t.\n",
"\\tag{12.4}\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Mathematics of AI Agents",
"language": "python",
"name": "maa-lab"
},
"lab_chapter": 12,
"lab_execution": {
"code_cells": 6,
"created_utc": "2026-10-02T05:14:42.572917+00:00",
"elapsed_seconds": 8.881980917067267,
"method": "fresh process; new ipykernel InProcessKernelManager; cells submitted as Jupyter execute requests",
"network_transport_tested": false,
"python": "3.11.15",
"source_sha256": "48a14eb8f44398d8d48099113f773a5925ff6215f452cba6115bae2e0da31437"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}