{
"cells": [
{
"cell_type": "markdown",
"id": "df12292296fc",
"metadata": {},
"source": [
"# Chapter 8: Acting in the Dark\n",
"\n",
"The document appears ready, but the controller cannot directly see whether the hidden review state is valid. Acting on the most likely state may work, yet an additional observation could change which action is best. The observation's value comes from its effect on a decision, not from how detailed its text looks.\n",
"\n",
"This notebook maintains a probability distribution over hidden states. It first predicts through a transition kernel, then updates with the observed signal. A separate calculation averages over every possible signal to price gathering information before acting. The changed case uses an observation that looks different on the surface but has identical state-conditioned distributions, exposing the difference between receiving data and learning something useful.\n",
"\n",
"**Outcome:** Update a belief and calculate gross and net one-step information value.\n",
"\n",
"- Inspect the declared input contract\n",
"- Predict the hand-checkable case\n",
"- Run the shared computation\n",
"- Change the critical assumption\n",
"- Apply the method to the transfer data\n",
"\n",
"**Guided route:** Run the worked calculation, inspect its figure, change the stated assumption, and try the transfer case. Read the explanations beside each result before opening the answers.\n",
"\n",
"**Deeper route:** First read the mathematics and canonical equation reference. Audit the input contract, predict the changed result, then inspect the shared chapter implementation and solve the questions independently. Both routes use the same calculations and preserve the equations."
]
},
{
"cell_type": "markdown",
"id": "a7781f3ee7dc",
"metadata": {},
"source": [
"## Technical Requirements\n",
"\n",
"Python 3.11 or later, the complete laboratory folder, and the notebook dependencies listed in `requirements-notebooks.txt` (the launcher's **Install notebook tools** choice installs them; see START-HERE). Standard-library chapter commands also support Python 3.10. No API key, model account or network call is used by this experiment.\n",
"\n",
"Prior knowledge:\n",
"\n",
"- Python lists and dictionaries\n",
"- The mapped chapter and its notation"
]
},
{
"cell_type": "markdown",
"id": "30186f819169",
"metadata": {},
"source": [
"## The question and its mathematics\n",
"\n",
"The predictive belief is b_minus(j)=sum_i b(i)P(j|i). For observation o, its probability is Z(o)=sum_j b_minus(j)O(o|j), and Bayes' rule gives b_plus(j)=b_minus(j)O(o|j)/Z(o). An observation with Z(o)=0 is impossible under the declared model and is rejected.\n",
"\n",
"For reward vector r_a, acting now has value max_a sum_j b_minus(j)r_a(j). If the controller first observes, its gross expected decision value is sum_o Z(o) max_a sum_j b_plus_o(j)r_a(j). Gross value of information is the difference between that quantity and acting now. Net value subtracts the observation cost.\n",
"\n",
"The maximum is taken after each possible observation because the action can adapt to the signal. Placing the maximum before averaging would price a fixed action and lose the decision benefit. An informative signal can still have zero value when it never changes the optimal action. Identical observation rows convey no state information at all. These are one-step values; the calculation does not solve a general partially observable planning problem."
]
},
{
"cell_type": "markdown",
"id": "f61d79e2c3de",
"metadata": {},
"source": [
"## A calculation you can run\n",
"\n",
"Provide a prior belief, transition matrix, observation matrix, reward vector for each action, observed index, and observation cost. Kernel dimensions must align, all probability rows must normalize, and the observed index must exist. The function predicts belief, evaluates the observation likelihood, forms its posterior, and selects the best posterior action.\n",
"\n",
"It then repeats the posterior calculation for every possible observation to obtain information value. The first plot compares prior, predictive, and posterior probability of state zero. The second varies observation cost while holding gross benefit fixed. In the changed case, replace the observation rows with identical distributions. Predict both the posterior and information value before execution.\n",
"\n",
"The next cell finds the bundle and imports the same computation used by the chapter skill. It does not change your system Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "42ebefc0e5c1",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys, json\n",
"LAB_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"lab-manifest.json\").is_file()), None)\n",
"if LAB_ROOT is None:\n",
" raise RuntimeError(\"Open this notebook from the complete extracted laboratory folder.\")\n",
"sys.path.insert(0, str(LAB_ROOT / \"src\"))\n",
"from math_ai_agents.core import analyze, report_text\n",
"from math_ai_agents.plotting import figure_svg\n",
"from IPython.display import SVG, display\n"
]
},
{
"cell_type": "markdown",
"id": "7fffde8c0edb",
"metadata": {},
"source": [
"Set the declared inputs below. These are constructed teaching values, not measurements from a production agent. Change a value only after predicting what it should change."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "844dcaea3f26",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 8: belief-information\n",
"Should a controller act now or buy an observation before choosing?\n",
"Evidence: constructed teaching example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"predictive_belief\": [\n",
" 0.5,\n",
" 0.5\n",
" ],\n",
" \"posterior\": [\n",
" 0.9,\n",
" 0.1\n",
" ],\n",
" \"observed_probability\": 0.5,\n",
" \"act_now_value\": 0.0,\n",
" \"observe_then_act_value\": 7.0,\n",
" \"gross_value_of_information\": 8.0,\n",
" \"net_value_of_information\": 7.0,\n",
" \"posterior_action\": 0\n",
"}\n",
"\n",
"Interpretation:\n",
"Bayes' rule updates the belief, while information value averages optimal later decisions over all possible observations.\n",
"\n",
"Assumptions:\n",
"- Transition precedes observation.\n",
"- Information arrives before one decision.\n",
"- Observation incurs the declared cost.\n",
"\n",
"Limitations:\n",
"- A likely observation is not necessarily useful; identical observation rows give no information.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"chapter = 8\n",
"inputs = {'belief': [0.5, 0.5],\n",
" 'transition': [[1, 0], [0, 1]],\n",
" 'observation': [[0.9, 0.1], [0.1, 0.9]],\n",
" 'observed': 0,\n",
" 'action_rewards': [[10, -10], [-10, 10]],\n",
" 'observation_cost': 1}\n",
"report = analyze(chapter, inputs)\n",
"# This input was explicitly taken from the teaching fixture.\n",
"report['evidence_kind'] = 'constructed teaching example'\n",
"print(report_text(report))"
]
},
{
"cell_type": "markdown",
"id": "293920aa495e",
"metadata": {},
"source": [
"The default prior is equally split and the transition preserves it. Observation zero produces posterior [0.9,0.1]. Acting now has value zero because either action has equally weighted gains and losses. After observing, the controller can choose the appropriate action, giving expected value 8 before cost and 7 after cost.\n",
"\n",
"The gross information value is therefore 8. With identical observation rows in the changed case, the posterior remains [0.5,0.5] regardless of the signal. Gross information value falls to zero and net information value becomes -1. The signal still arrives, but it does not earn its cost.\n",
"\n",
"The plot below uses the calculated quantities. Read each panel's units before comparing its values."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "590f57e74048",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Matplotlib is building the font cache; this may take a moment.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(SVG(figure_svg(report)))"
]
},
{
"cell_type": "markdown",
"id": "f59ea1a43316",
"metadata": {},
"source": [
"**Figure 8.L1:** Calculated chapter experiment. Each panel labels its input and output units; interpret it under the assumptions printed in the report."
]
},
{
"cell_type": "markdown",
"id": "83e40ae9f4b7",
"metadata": {},
"source": [
"## Change the assumption\n",
"\n",
"A belief update can be mathematically correct under an incorrect observation model. If a tool's signal changes distribution, or if hidden states were defined too coarsely, the posterior can look confident without being calibrated. Inspect the likelihood source rather than treating normalization as validation of the world model.\n",
"\n",
"Information also has opportunity cost. The implemented cost is a single utility deduction; a deadline or state transition during observation would require a larger model. Likewise, this function assumes the same action set remains available after the observation.\n",
"\n",
"An impossible signal should not be assigned an arbitrary uniform posterior. It is evidence that the declared boundary is incomplete or the input is inconsistent. The rejection asks you to repair that model rather than fabricate a belief. Preserve unmodeled outcomes in the next measurement design."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "7e98c6953bdb",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 8: belief-information\n",
"Should a controller act now or buy an observation before choosing?\n",
"Evidence: constructed changed-assumption example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"predictive_belief\": [\n",
" 0.5,\n",
" 0.5\n",
" ],\n",
" \"posterior\": [\n",
" 0.5,\n",
" 0.5\n",
" ],\n",
" \"observed_probability\": 0.5,\n",
" \"act_now_value\": 0.0,\n",
" \"observe_then_act_value\": -1.0,\n",
" \"gross_value_of_information\": 0.0,\n",
" \"net_value_of_information\": -1.0,\n",
" \"posterior_action\": 0\n",
"}\n",
"\n",
"Interpretation:\n",
"Bayes' rule updates the belief, while information value averages optimal later decisions over all possible observations.\n",
"\n",
"Assumptions:\n",
"- Transition precedes observation.\n",
"- Information arrives before one decision.\n",
"- Observation incurs the declared cost.\n",
"\n",
"Limitations:\n",
"- A likely observation is not necessarily useful; identical observation rows give no information.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"changed_inputs = {'belief': [0.5, 0.5],\n",
" 'transition': [[1, 0], [0, 1]],\n",
" 'observation': [[0.5, 0.5], [0.5, 0.5]],\n",
" 'observed': 0,\n",
" 'action_rewards': [[10, -10], [-10, 10]],\n",
" 'observation_cost': 1}\n",
"changed = analyze(chapter, changed_inputs)\n",
"changed['evidence_kind'] = 'constructed changed-assumption example'\n",
"print(report_text(changed))\n",
"display(SVG(figure_svg(changed)))"
]
},
{
"cell_type": "markdown",
"id": "5d80f363cce5",
"metadata": {},
"source": [
"**Figure 8.L2:** The changed-assumption result. Compare the printed quantities and the stated assumptions with the first run. A different input need not imply a causal effect in a deployed agent."
]
},
{
"cell_type": "markdown",
"id": "b9be244200c7",
"metadata": {},
"source": [
"## Try a new case\n",
"\n",
"The transfer case predicts a new belief before receiving observation one. Its transition mixes the initial states, so the prior is not the correct distribution to insert directly into Bayes' rule. First calculate the predictive belief [0.6,0.4], then weight it by observation-one likelihoods [0.2,0.7] and normalize.\n",
"\n",
"Use local data only when the state, observation, and action definitions share one boundary. If likelihoods are fitted estimates, retain their sample sizes and calibration limits outside this exact calculation. Report gross information value separately from cost so a negative net result can be traced to unhelpful evidence, high price, or both."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "8728d0e8ba74",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 8: belief-information\n",
"Should a controller act now or buy an observation before choosing?\n",
"Evidence: constructed transfer example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"predictive_belief\": [\n",
" 0.6,\n",
" 0.4\n",
" ],\n",
" \"posterior\": [\n",
" 0.3,\n",
" 0.7\n",
" ],\n",
" \"observed_probability\": 0.4,\n",
" \"act_now_value\": 1.4,\n",
" \"observe_then_act_value\": 2.28,\n",
" \"gross_value_of_information\": 1.08,\n",
" \"net_value_of_information\": 0.88,\n",
" \"posterior_action\": 1\n",
"}\n",
"\n",
"Interpretation:\n",
"Bayes' rule updates the belief, while information value averages optimal later decisions over all possible observations.\n",
"\n",
"Assumptions:\n",
"- Transition precedes observation.\n",
"- Information arrives before one decision.\n",
"- Observation incurs the declared cost.\n",
"\n",
"Limitations:\n",
"- A likely observation is not necessarily useful; identical observation rows give no information.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"transfer_inputs = {'belief': [0.8, 0.2],\n",
" 'transition': [[0.7, 0.3], [0.2, 0.8]],\n",
" 'observation': [[0.8, 0.2], [0.3, 0.7]],\n",
" 'observed': 1,\n",
" 'action_rewards': [[5, -4], [0, 2]],\n",
" 'observation_cost': 0.2}\n",
"transfer = analyze(chapter, transfer_inputs)\n",
"transfer['evidence_kind'] = 'constructed transfer example'\n",
"print(report_text(transfer))"
]
},
{
"cell_type": "markdown",
"id": "9f3512a08b2a",
"metadata": {},
"source": [
"## Apply the method to your inputs\n",
"\n",
"The example file below has the exact input shape the method accepts. Copy it to a new file, replace its values, then point `reader_file` at your copy. Run the cell again. Supplied inputs retain their stated provenance; the program cannot establish that they are representative observations.\n",
"\n",
"- **belief:** Normalized prior state probability list.\n",
"- **transition:** Square row-stochastic state transition matrix.\n",
"- **observation:** State by observation row-stochastic matrix.\n",
"- **observed:** Zero-based observation index.\n",
"- **action_rewards:** Action by state finite reward matrix.\n",
"- **observation_cost:** Nonnegative cost paid before the one later action."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "1e2d1885a48d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 8: belief-information\n",
"Should a controller act now or buy an observation before choosing?\n",
"Evidence: supplied local inputs; provenance not independently verified\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"predictive_belief\": [\n",
" 0.6,\n",
" 0.4\n",
" ],\n",
" \"posterior\": [\n",
" 0.3,\n",
" 0.7\n",
" ],\n",
" \"observed_probability\": 0.4,\n",
" \"act_now_value\": 1.4,\n",
" \"observe_then_act_value\": 2.28,\n",
" \"gross_value_of_information\": 1.08,\n",
" \"net_value_of_information\": 0.88,\n",
" \"posterior_action\": 1\n",
"}\n",
"\n",
"Interpretation:\n",
"Bayes' rule updates the belief, while information value averages optimal later decisions over all possible observations.\n",
"\n",
"Assumptions:\n",
"- Transition precedes observation.\n",
"- Information arrives before one decision.\n",
"- Observation incurs the declared cost.\n",
"\n",
"Limitations:\n",
"- A likely observation is not necessarily useful; identical observation rows give no information.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"reader_file = LAB_ROOT / 'data/examples/ch08.json'\n",
"reader_inputs = json.loads(reader_file.read_text())\n",
"reader_report = analyze(chapter, reader_inputs)\n",
"print(report_text(reader_report))"
]
},
{
"cell_type": "markdown",
"id": "45d24f7dcc7d",
"metadata": {},
"source": [
"## Questions\n",
"\n",
"1. Compute default posterior.\n",
"\n",
"2. What observation cost makes default information break even?\n",
"\n",
"3. Compute transfer posterior after observation 1.\n",
"\n",
"Answers: [separate solutions](../solutions/ch08.md). Try the calculation before opening them."
]
},
{
"cell_type": "markdown",
"id": "93912f2e458a",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"Beliefs predict through transitions and update through observation likelihoods. Information is worth its decision improvement after cost, not its volume or apparent specificity. The default signal changes the optimal action; the changed signal is uninformative; the transfer case requires transition prediction before Bayes' rule. Keep normalization, model provenance, and action timing explicit. This notebook prices one observation and one decision within a declared hidden-state model.\n",
"\n",
"Limits of this experiment:\n",
"\n",
"- This is a local calculation under declared inputs, not an empirical claim about a deployed agent.\n",
"- Read the returned assumptions and limitations before applying the numerical result.\n",
"\n",
"The assistant skill is [`maa-08-belief-information`](../skills/maa-08-belief-information/SKILL.md). It uses this notebook's tested computation and input contract."
]
},
{
"cell_type": "markdown",
"id": "9cc9f25ba898",
"metadata": {},
"source": [
"## Equations from the chapter\n",
"\n",
"These are the unchanged display equations and their explanations from the canonical chapter. They are a reference for the experiment, not a claim that every equation is numerically implemented by this one method."
]
},
{
"cell_type": "markdown",
"id": "47534cad9957",
"metadata": {},
"source": [
"### Equation 8.1\n",
"\n",
"\n",
"\n",
"Equation (8.1) turns yesterday's belief, one action, and one observation into today's belief.\n",
"\n",
"Push the old belief forward through the action, weight each destination by how well it explains what was seen, then rescale so the weights sum to one.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\mathbf{b}_{t+1}(x') \\;=\\;\n",
"\\frac{\\begin{gathered}\n",
"\\operatorname{Obs}(o\\mid x',a)\\\\\n",
"\\cdot\\sum_{x\\in\\mathcal{X}}P(x'\\mid x,a)\\,\\mathbf{b}_t(x)\n",
"\\end{gathered}}\n",
"{\\begin{gathered}\n",
"\\sum_{x''\\in\\mathcal{X}}\\operatorname{Obs}(o\\mid x'',a)\\\\\n",
"\\cdot\\sum_{x\\in\\mathcal{X}}P(x''\\mid x,a)\\,\\mathbf{b}_t(x)\n",
"\\end{gathered}}.\n",
"\\tag{8.1}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "c544b2cc2bdb",
"metadata": {},
"source": [
"### Equation un-numbered display 2\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"T_E=\\begin{pmatrix}\n",
"0.1&0.9&0&0\\\\\n",
"0.1&0&0.9&0\\\\\n",
"0&0.1&0&0.9\\\\\n",
"0&0&0.1&0.9\n",
"\\end{pmatrix},\\qquad\n",
"\\operatorname{Obs}(\\text{non-goal}\\mid x')=(1,1,0,1).\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "49a8e045cfdc",
"metadata": {},
"source": [
"### Equation 8.2\n",
"\n",
"\n",
"\n",
"Equation (8.2) prices an action under uncertainty about where the agent is.\n",
"\n",
"Average the reward the action would earn in each state, weighting each state by how much belief sits on it.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\bar r(\\mathbf{b},a)\\;=\\;\\sum_{x\\in\\mathcal{X}}\\mathbf{b}(x)\\,r(x,a).\n",
"\\tag{8.2}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "f3826669ef78",
"metadata": {},
"source": [
"### Equation 8.3\n",
"\n",
"\n",
"\n",
"Equation (8.3) values a belief by the best action available from it and the beliefs that action can lead to.\n",
"\n",
"Take the immediate expected reward, add the discounted average value of the belief each possible observation would produce, and keep the best action.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\begin{aligned}\n",
"V(\\mathbf{b})\n",
"&=\\max_{a\\in\\mathcal{A}}\\Big[\\,\\bar r(\\mathbf{b},a)\\\\\n",
"&\\quad+\\gamma\\sum_{o}\\Pr(o\\mid \\mathbf{b},a)\\,\n",
"V\\big(\\mathbf{b}'(\\mathbf{b},a,o)\\big)\\Big].\n",
"\\end{aligned}\n",
"\\tag{8.3}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "8617c1796583",
"metadata": {},
"source": [
"### Equation 8.4\n",
"\n",
"\n",
"\n",
"Equation (8.4) represents finite-horizon belief value as best dot product from finite policy-vector set.\n",
"\n",
"Score each contingent plan against current belief, then select largest expected return.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"V_H(\\mathbf b)=\\max_{\\alpha\\in\\Gamma_H}\\alpha^\\top\\mathbf b.\n",
"\\tag{8.4}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "d77fd90e7d73",
"metadata": {},
"source": [
"### Equation 8.5\n",
"\n",
"\n",
"\n",
"Equation (8.5) prices a look by how much better the decision becomes once its answer is known.\n",
"\n",
"Average the best achievable reward across the beliefs each possible answer would leave, then subtract the best achievable reward without looking.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\begin{aligned}\n",
"\\operatorname{VOI}(O)\n",
"&=\\sum_{o}\\Pr(o\\mid\\mathbf{b})\\,\n",
"\\max_{a\\in\\mathcal{A}}\\bar r\\big(\\mathbf{b}'(\\mathbf{b},o),a\\big)\\\\\n",
"&\\quad-\\max_{a\\in\\mathcal{A}}\\bar r(\\mathbf{b},a).\n",
"\\end{aligned}\n",
"\\tag{8.5}\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Mathematics of AI Agents",
"language": "python",
"name": "maa-lab"
},
"lab_chapter": 8,
"lab_execution": {
"code_cells": 6,
"created_utc": "2026-10-02T05:14:06.139594+00:00",
"elapsed_seconds": 8.867219042032957,
"method": "fresh process; new ipykernel InProcessKernelManager; cells submitted as Jupyter execute requests",
"network_transport_tested": false,
"python": "3.11.15",
"source_sha256": "a85c8b0a9f634d9f51c8fe3bc58cc7077f2caf5beec642c69f612f7e55b71039"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}