{
"cells": [
{
"cell_type": "markdown",
"id": "67570be72469",
"metadata": {},
"source": [
"# Chapter 22: Safe Enough to Act\n",
"\n",
"The most rewarding action is not always one the controller may take. A fast release can have high expected utility while lacking current approval. A different action can be authorized and still exceed a declared risk budget. Those are separate reasons to refuse execution.\n",
"\n",
"This notebook compares three decision boundaries: unconstrained penalty maximization, penalty maximization after authority filtering, and reward maximization after both authority and expected-risk filtering. The example includes explicit abstention. Its purpose is to show what each mathematical formulation permits, rather than treating a large penalty as an invisible substitute for an action prohibition. The changed case relaxes only the expected-risk threshold.\n",
"\n",
"**Outcome:** Compare penalty choice with hard authorization and expected-risk constraints.\n",
"\n",
"- Inspect the declared input contract\n",
"- Predict the hand-checkable case\n",
"- Run the shared computation\n",
"- Change the critical assumption\n",
"- Apply the method to the transfer data\n",
"\n",
"**Guided route:** Run the worked calculation, inspect its figure, change the stated assumption, and try the transfer case. Read the explanations beside each result before opening the answers.\n",
"\n",
"**Deeper route:** First read the mathematics and canonical equation reference. Audit the input contract, predict the changed result, then inspect the shared chapter implementation and solve the questions independently. Both routes use the same calculations and preserve the equations."
]
},
{
"cell_type": "markdown",
"id": "a7781f3ee7dc",
"metadata": {},
"source": [
"## Technical Requirements\n",
"\n",
"Python 3.11 or later, the complete laboratory folder, and the notebook dependencies listed in `requirements-notebooks.txt` (the launcher's **Install notebook tools** choice installs them; see START-HERE). Standard-library chapter commands also support Python 3.10. No API key, model account or network call is used by this experiment.\n",
"\n",
"Prior knowledge:\n",
"\n",
"- Python lists and dictionaries\n",
"- The mapped chapter and its notation"
]
},
{
"cell_type": "markdown",
"id": "2d8f387a1a4d",
"metadata": {},
"source": [
"## The question and its mathematics\n",
"\n",
"Let reward R(a), expected adverse-event probability rho(a), penalty lambda>=0, and current permission A(a) describe an action. Penalty choice maximizes R(a)-lambda rho(a). A finite lambda trades reward against risk; it does not enforce rho(a)<=b or A(a)=true.\n",
"\n",
"Authority filtering forms the set of actions whose permission predicate is true. The constrained choice further requires rho(a)<=b and maximizes reward within that intersection. An empty intersection returns no feasible choice. Abstention appears only if it is explicitly supplied with its own reward, risk, and permission.\n",
"\n",
"An expected-risk limit is a statement about a probability or average, not a pathwise guarantee. A permitted action with rho=0.05 can still realize an adverse event. Similarly, a penalty-optimal action can violate the declared limit when reward is large enough. The method accepts supplied risks; it does not estimate them or prove that a real system's risk is below the given number."
]
},
{
"cell_type": "markdown",
"id": "f56b7be6e7d2",
"metadata": {},
"source": [
"## A calculation you can run\n",
"\n",
"Provide action rows and declare the risk threshold and penalty separately. Validation checks finite rewards, probability domains, and exact permission booleans. The function returns unconstrained penalty choice, authorized penalty choice, constrained choice, and the number of feasible actions. Every alternative remains in the table with its flags and penalized value.\n",
"\n",
"Reward and risk plots use separate vertical units. Compare selected labels with those figures instead of reading a reward bar as safety evidence. Run the default case and identify the first filter that removes each rejected action. In the changed case raise the expected-risk limit to 0.25 while preserving rewards, risks, and permissions. Predict whether a forbidden action can become available through that change.\n",
"\n",
"The next cell finds the bundle and imports the same computation used by the chapter skill. It does not change your system Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "42ebefc0e5c1",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys, json\n",
"LAB_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"lab-manifest.json\").is_file()), None)\n",
"if LAB_ROOT is None:\n",
" raise RuntimeError(\"Open this notebook from the complete extracted laboratory folder.\")\n",
"sys.path.insert(0, str(LAB_ROOT / \"src\"))\n",
"from math_ai_agents.core import analyze, report_text\n",
"from math_ai_agents.plotting import figure_svg\n",
"from IPython.display import SVG, display\n"
]
},
{
"cell_type": "markdown",
"id": "7fffde8c0edb",
"metadata": {},
"source": [
"Set the declared inputs below. These are constructed teaching values, not measurements from a production agent. Change a value only after predicting what it should change."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "59a900dfd7ea",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 22: risk-authority-contract\n",
"Which choices satisfy both current authority and a declared expected-risk limit?\n",
"Evidence: constructed teaching example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"unconstrained_penalty_choice\": \"fast-release\",\n",
" \"authorized_penalty_choice\": \"risky-authorized\",\n",
" \"constrained_choice\": \"reviewed-release\",\n",
" \"feasible_count\": 2\n",
"}\n",
"\n",
"Interpretation:\n",
"A finite penalty can favor a forbidden or over-limit action. Authorization filters the action set; an expected-risk constraint filters authorized alternatives again.\n",
"\n",
"Assumptions:\n",
"- Risks are supplied expected probabilities for the same adverse event.\n",
"- Authorization is a current hard predicate.\n",
"\n",
"Limitations:\n",
"- Expected risk does not establish pathwise safety.\n",
"- Missing feasible actions requires abstention or a new authorized alternative.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"chapter = 22\n",
"inputs = {'risk_limit': 0.1,\n",
" 'risk_penalty': 5,\n",
" 'actions': [{'name': 'fast-release', 'reward': 10, 'risk': 0.3, 'authorized': False},\n",
" {'name': 'reviewed-release', 'reward': 5, 'risk': 0.05, 'authorized': True},\n",
" {'name': 'risky-authorized', 'reward': 8, 'risk': 0.2, 'authorized': True},\n",
" {'name': 'abstain', 'reward': 0, 'risk': 0, 'authorized': True}]}\n",
"report = analyze(chapter, inputs)\n",
"# This input was explicitly taken from the teaching fixture.\n",
"report['evidence_kind'] = 'constructed teaching example'\n",
"print(report_text(report))"
]
},
{
"cell_type": "markdown",
"id": "cf5a4bf5499d",
"metadata": {},
"source": [
"Fast-release has penalized value 10-5(0.3)=8.5 and wins unconstrained penalty choice, but it is unauthorized. Among authorized actions, risky-authorized has value 8-5(0.2)=7 and wins the penalty comparison. It exceeds the default risk limit 0.1.\n",
"\n",
"The hard constrained choice is reviewed-release, with reward 5 and risk 0.05. Abstention is also feasible but has reward 0. When the limit changes to 0.25, risky-authorized becomes feasible and wins at reward 8. Fast-release stays forbidden because a risk-threshold change cannot create permission. The three selected labels therefore answer different questions.\n",
"\n",
"The plot below uses the calculated quantities. Read each panel's units before comparing its values."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "590f57e74048",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Matplotlib is building the font cache; this may take a moment.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(SVG(figure_svg(report)))"
]
},
{
"cell_type": "markdown",
"id": "39b865af2c20",
"metadata": {},
"source": [
"**Figure 22.L1:** Calculated chapter experiment. Each panel labels its input and output units; interpret it under the assumptions printed in the report."
]
},
{
"cell_type": "markdown",
"id": "e03c523aa669",
"metadata": {},
"source": [
"## Change the assumption\n",
"\n",
"A large penalty can look like a safety rule while leaving prohibited behavior available. If reward grows or the penalty is miscalibrated, the controller may choose the forbidden row. The hard authorization predicate avoids that particular tradeoff by removing the action before optimization.\n",
"\n",
"Risk estimates also carry uncertainty. A supplied value 0.05 is not evidence of a bound unless an evaluation or formal process supports it. This notebook does not add confidence limits to supplied probabilities. When a limit must be conservative, provide a justified upper estimate or use a separate measurement design.\n",
"\n",
"Pathwise properties need another representation. A mean-risk constraint cannot ensure that every trajectory respects a temporal invariant or that a tool never receives an unauthorized command. Use an explicit monitor and effect trace for those questions. Keep refusal and escalation alternatives visible when no permitted action meets the risk contract."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "7a6e169b26b6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 22: risk-authority-contract\n",
"Which choices satisfy both current authority and a declared expected-risk limit?\n",
"Evidence: constructed changed-assumption example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"unconstrained_penalty_choice\": \"fast-release\",\n",
" \"authorized_penalty_choice\": \"risky-authorized\",\n",
" \"constrained_choice\": \"risky-authorized\",\n",
" \"feasible_count\": 3\n",
"}\n",
"\n",
"Interpretation:\n",
"A finite penalty can favor a forbidden or over-limit action. Authorization filters the action set; an expected-risk constraint filters authorized alternatives again.\n",
"\n",
"Assumptions:\n",
"- Risks are supplied expected probabilities for the same adverse event.\n",
"- Authorization is a current hard predicate.\n",
"\n",
"Limitations:\n",
"- Expected risk does not establish pathwise safety.\n",
"- Missing feasible actions requires abstention or a new authorized alternative.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"changed_inputs = {'risk_limit': 0.25,\n",
" 'risk_penalty': 5,\n",
" 'actions': [{'name': 'fast-release', 'reward': 10, 'risk': 0.3, 'authorized': False},\n",
" {'name': 'reviewed-release', 'reward': 5, 'risk': 0.05, 'authorized': True},\n",
" {'name': 'risky-authorized', 'reward': 8, 'risk': 0.2, 'authorized': True},\n",
" {'name': 'abstain', 'reward': 0, 'risk': 0, 'authorized': True}]}\n",
"changed = analyze(chapter, changed_inputs)\n",
"changed['evidence_kind'] = 'constructed changed-assumption example'\n",
"print(report_text(changed))\n",
"display(SVG(figure_svg(changed)))"
]
},
{
"cell_type": "markdown",
"id": "9f497405d75a",
"metadata": {},
"source": [
"**Figure 22.L2:** The changed-assumption result. Compare the printed quantities and the stated assumptions with the first run. A different input need not imply a causal effect in a deployed agent."
]
},
{
"cell_type": "markdown",
"id": "aada1953691a",
"metadata": {},
"source": [
"## Try a new case\n",
"\n",
"The transfer limit is zero. Act has risk 0.01 and is therefore infeasible, even though its penalized value is 19 under penalty 100. Wait has risk 0 and reward-1, so it is the only constrained choice. Negative reward does not invalidate a feasible refusal.\n",
"\n",
"For local use, identify who owns the permission predicate and who owns the risk threshold. State the adverse event precisely and keep its probability denominator consistent across alternatives. A practical report should explain which rows fail authority, which fail expected risk, and which remain available. It should not convert a low average into a guarantee about every run."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "15df4b46d041",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 22: risk-authority-contract\n",
"Which choices satisfy both current authority and a declared expected-risk limit?\n",
"Evidence: constructed transfer example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"unconstrained_penalty_choice\": \"act\",\n",
" \"authorized_penalty_choice\": \"act\",\n",
" \"constrained_choice\": \"wait\",\n",
" \"feasible_count\": 1\n",
"}\n",
"\n",
"Interpretation:\n",
"A finite penalty can favor a forbidden or over-limit action. Authorization filters the action set; an expected-risk constraint filters authorized alternatives again.\n",
"\n",
"Assumptions:\n",
"- Risks are supplied expected probabilities for the same adverse event.\n",
"- Authorization is a current hard predicate.\n",
"\n",
"Limitations:\n",
"- Expected risk does not establish pathwise safety.\n",
"- Missing feasible actions requires abstention or a new authorized alternative.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"transfer_inputs = {'risk_limit': 0,\n",
" 'risk_penalty': 100,\n",
" 'actions': [{'name': 'act', 'reward': 20, 'risk': 0.01, 'authorized': True},\n",
" {'name': 'wait', 'reward': -1, 'risk': 0, 'authorized': True}]}\n",
"transfer = analyze(chapter, transfer_inputs)\n",
"transfer['evidence_kind'] = 'constructed transfer example'\n",
"print(report_text(transfer))"
]
},
{
"cell_type": "markdown",
"id": "97c47e70e812",
"metadata": {},
"source": [
"## Apply the method to your inputs\n",
"\n",
"The example file below has the exact input shape the method accepts. Copy it to a new file, replace its values, then point `reader_file` at your copy. Run the cell again. Supplied inputs retain their stated provenance; the program cannot establish that they are representative observations.\n",
"\n",
"- **risk_limit:** Maximum allowed adverse-event probability in [0,1].\n",
"- **risk_penalty:** Nonnegative utility deduction per unit probability.\n",
"- **actions:** Each row has name,reward:finite number,risk:probability,authorized:exact boolean. Abstention must be an explicit row."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "5f9f598be151",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 22: risk-authority-contract\n",
"Which choices satisfy both current authority and a declared expected-risk limit?\n",
"Evidence: supplied local inputs; provenance not independently verified\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"unconstrained_penalty_choice\": \"act\",\n",
" \"authorized_penalty_choice\": \"act\",\n",
" \"constrained_choice\": \"wait\",\n",
" \"feasible_count\": 1\n",
"}\n",
"\n",
"Interpretation:\n",
"A finite penalty can favor a forbidden or over-limit action. Authorization filters the action set; an expected-risk constraint filters authorized alternatives again.\n",
"\n",
"Assumptions:\n",
"- Risks are supplied expected probabilities for the same adverse event.\n",
"- Authorization is a current hard predicate.\n",
"\n",
"Limitations:\n",
"- Expected risk does not establish pathwise safety.\n",
"- Missing feasible actions requires abstention or a new authorized alternative.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"reader_file = LAB_ROOT / 'data/examples/ch22.json'\n",
"reader_inputs = json.loads(reader_file.read_text())\n",
"reader_report = analyze(chapter, reader_inputs)\n",
"print(report_text(reader_report))"
]
},
{
"cell_type": "markdown",
"id": "1f02db73bee9",
"metadata": {},
"source": [
"## Questions\n",
"\n",
"1. Which default action wins the hard constraint?\n",
"\n",
"2. Can risk_limit = 0.25 authorize fast-release?\n",
"\n",
"3. Why does transfer choose wait despite negative reward?\n",
"\n",
"Answers: [separate solutions](../solutions/ch22.md). Try the calculation before opening them."
]
},
{
"cell_type": "markdown",
"id": "9b4e41b8f0b9",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"Reward penalties, expected-risk constraints, and hard authority are different decision rules. The notebook compares their choices explicitly and preserves every rejected alternative's reason. Relaxing risk can admit a previously over-limit action but cannot grant permission. The transfer case shows that a finite penalty can still prefer positive risk under a zero-risk contract. Use the constrained result within its declared probability model, while reserving pathwise and enforcement claims for separate evidence.\n",
"\n",
"Limits of this experiment:\n",
"\n",
"- This is a local calculation under declared inputs, not an empirical claim about a deployed agent.\n",
"- Read the returned assumptions and limitations before applying the numerical result.\n",
"\n",
"The assistant skill is [`maa-22-risk-authority-contract`](../skills/maa-22-risk-authority-contract/SKILL.md). It uses this notebook's tested computation and input contract."
]
},
{
"cell_type": "markdown",
"id": "9cc9f25ba898",
"metadata": {},
"source": [
"## Equations from the chapter\n",
"\n",
"These are the unchanged display equations and their explanations from the canonical chapter. They are a reference for the experiment, not a claim that every equation is numerically implemented by this one method."
]
},
{
"cell_type": "markdown",
"id": "30f137e06b54",
"metadata": {},
"source": [
"### Equation 22.1\n",
"\n",
"\n",
"\n",
"Separates goal-seeking from permission: reward is maximized only among policies that meet every declared auxiliary-cost limit.\n",
"\n",
"The same policy `\\pi` appears in both ledgers, while each `i` names a distinct measured hazard and `d_i` names its limit.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\begin{aligned}\n",
"\\text{maximize}_{\\pi}\\quad & \\operatorname{Ret}^{\\pi}\\\\\n",
"\\text{subject to}\\quad & \\operatorname{Cost}_i^{\\pi}\\le d_i \\quad \\text{for every declared } i.\n",
"\\end{aligned}\\tag{22.1}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "dcd953c28a5f",
"metadata": {},
"source": [
"### Equation 22.2\n",
"\n",
"\n",
"\n",
"Turns one auxiliary cost into a fixed reward penalty so ordinary reward optimization can rank policies by one blended score.\n",
"\n",
"`\\lambda_{\\mathrm{safe}}` sets a constant exchange rate between reward units and the first declared cost's units.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{Score}_{\\lambda_{\\mathrm{safe}}}(\\pi)\n",
"=\\operatorname{Ret}^{\\pi}-\\lambda_{\\mathrm{safe}}\\operatorname{Cost}_1^{\\pi}.\n",
"\\tag{22.2}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "4732702f982b",
"metadata": {},
"source": [
"### Equation 22.3\n",
"\n",
"\n",
"\n",
"Bounds one update's modeled excess over cost limit by a term controlled by policy-step size, discounting, and constraint advantage.\n",
"\n",
"The right side is `d_i` plus a nonnegative allowance, so the displayed result permits bounded violation rather than asserting satisfaction.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{Cost}_i^{\\pi_{k+1}}\n",
"\\le d_i+\n",
"\\frac{\\sqrt{2\\delta}\\,\\gamma\\,\\epsilon_i}{(1-\\gamma)^2}.\n",
"\\tag{22.3}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "25c14d0aeba0",
"metadata": {},
"source": [
"### Equation 22.4\n",
"\n",
"\n",
"\n",
"Reserves room below a deployment cost limit for known modeling, approximation, sampling, and update-bound exposure.\n",
"\n",
"Optimization targets a stricter internal ceiling, leaving `\\varepsilon_{\\mathrm{safe}}` between measured planning cost and external limit `d_i`.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{Cost}_i^{\\pi}\n",
"\\le d_i-\\varepsilon_{\\mathrm{safe}},\n",
"\\qquad \\varepsilon_{\\mathrm{safe}}>0.\n",
"\\tag{22.4}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "f94cf1516333",
"metadata": {},
"source": [
"### Equation 22.5\n",
"\n",
"\n",
"\n",
"Replaces a mean-cost target with a declared upper-tail severity measure for episode cost `C`.\n",
"\n",
"The threshold `z` searches tail cutoffs, while `1/(1-\\alpha)` scales severity beyond each cutoff.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{CVaR}_{\\alpha}(C)\n",
"= \\inf_{z\\in\\mathbb R}\\left[z+\\frac{1}{1-\\alpha}\\,\\mathbb E[(C-z)_+]\\right].\n",
"\\tag{22.5}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "a2c81c685a67",
"metadata": {},
"source": [
"### Equation 22.6\n",
"\n",
"\n",
"\n",
"Carries a declared finite-horizon risk budget forward, preventing one charged action from spending more authority than remains.\n",
"\n",
"`b_t` is an accounting state, while `\\widehat r_t(a_t)` needs its own calibration and threat-model evidence.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"b_{t+1}=b_t-\\widehat r_t(a_t),\\qquad b_0=B,\\qquad \\widehat r_t(a_t)\\le b_t.\n",
"\\tag{22.6}\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Mathematics of AI Agents",
"language": "python",
"name": "maa-lab"
},
"lab_chapter": 22,
"lab_execution": {
"code_cells": 6,
"created_utc": "2026-10-02T05:16:14.543217+00:00",
"elapsed_seconds": 8.98612824990414,
"method": "fresh process; new ipykernel InProcessKernelManager; cells submitted as Jupyter execute requests",
"network_transport_tested": false,
"python": "3.11.15",
"source_sha256": "8430099ed1d44d63b6fd55138c6d7b2df48a6226f0886688f220b5002faf1857"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}