{
"cells": [
{
"cell_type": "markdown",
"id": "9a8e6679e8a0",
"metadata": {},
"source": [
"# Chapter 25: Improving an Agent Without Trusting the Improvement\n",
"\n",
"An improvement search produces several promising procedures. One wins development evaluation, but its apparent gain may come from adaptation to that development set. Release needs a different question: how does the frozen winner perform on evidence that did not help select it?\n",
"\n",
"This notebook separates development selection from a guard comparison. The default development winner fails the release threshold, while another candidate looks better on guard. That temptation is deliberate. Switching winners after inspecting guard results would use the guard for selection and break the clean protocol. The transfer case makes contamination explicit rather than treating a favorable held-out number as certification.\n",
"\n",
"**Outcome:** Select on development and check only the frozen winner against a declared guard contract.\n",
"\n",
"- Inspect the declared input contract\n",
"- Predict the hand-checkable case\n",
"- Run the shared computation\n",
"- Change the critical assumption\n",
"- Apply the method to the transfer data\n",
"\n",
"**Guided route:** Run the worked calculation, inspect its figure, change the stated assumption, and try the transfer case. Read the explanations beside each result before opening the answers.\n",
"\n",
"**Deeper route:** First read the mathematics and canonical equation reference. Audit the input contract, predict the changed result, then inspect the shared chapter implementation and solve the questions independently. Both routes use the same calculations and preserve the equations."
]
},
{
"cell_type": "markdown",
"id": "a7781f3ee7dc",
"metadata": {},
"source": [
"## Technical Requirements\n",
"\n",
"Python 3.11 or later, the complete laboratory folder, and the notebook dependencies listed in `requirements-notebooks.txt` (the launcher's **Install notebook tools** choice installs them; see START-HERE). Standard-library chapter commands also support Python 3.10. No API key, model account or network call is used by this experiment.\n",
"\n",
"Prior knowledge:\n",
"\n",
"- Python lists and dictionaries\n",
"- The mapped chapter and its notation"
]
},
{
"cell_type": "markdown",
"id": "de16dbb0dd25",
"metadata": {},
"source": [
"## The question and its mathematics\n",
"\n",
"For candidate c, development rate is the mean of its binary development outcomes. The selected candidate maximizes that rate, with input order resolving ties. Guard outcomes are aligned by position with the baseline guard. The selected gain is mean_i[Y_c,i-Y_baseline,i].\n",
"\n",
"Acceptance requires gain at least the prespecified threshold and guard_reused=false. The threshold is a procedural rule, not a confidence bound. Finite sample noise can produce a pass even when the target-population gain is smaller. A statistical release rule would require additional assumptions, error control, and an appropriately reserved evaluation design.\n",
"\n",
"The function reports every candidate's descriptive guard rate for teaching inspection, but the acceptance decision uses only the development-selected row. Those displayed guard results must not feed later candidate selection if the same evidence is to remain a clean guard. Repeated adaptive reuse changes its role. A boolean flag records that declared contamination boundary; it does not independently audit the history of a real experiment."
]
},
{
"cell_type": "markdown",
"id": "31def12b72e6",
"metadata": {},
"source": [
"## A calculation you can run\n",
"\n",
"Supply binary baseline guard outcomes, candidate development and guard vectors, minimum gain, and reuse status. Guard lengths must match the baseline; every observed outcome must be zero or one. Development samples need not share guard length because they serve a different purpose.\n",
"\n",
"The function calculates development rates, freezes the winner, computes paired guard gains, and applies the acceptance contract. Separate figures show development selection and guard gains. In the default case, identify the winner before looking at the second plot. The changed case improves that winner's guard evidence without changing selection. The transfer case gives a favorable gain but declares prior reuse, so you can test whether contamination blocks a clean release claim.\n",
"\n",
"The next cell finds the bundle and imports the same computation used by the chapter skill. It does not change your system Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "42ebefc0e5c1",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys, json\n",
"LAB_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"lab-manifest.json\").is_file()), None)\n",
"if LAB_ROOT is None:\n",
" raise RuntimeError(\"Open this notebook from the complete extracted laboratory folder.\")\n",
"sys.path.insert(0, str(LAB_ROOT / \"src\"))\n",
"from math_ai_agents.core import analyze, report_text\n",
"from math_ai_agents.plotting import figure_svg\n",
"from IPython.display import SVG, display\n"
]
},
{
"cell_type": "markdown",
"id": "7fffde8c0edb",
"metadata": {},
"source": [
"Set the declared inputs below. These are constructed teaching values, not measurements from a production agent. Change a value only after predicting what it should change."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "ba309f5cf916",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 25: development-guard-gate\n",
"Should the development winner be accepted under a separate release guard?\n",
"Evidence: constructed teaching example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"selected_candidate\": \"flashy\",\n",
" \"selected_guard_gain\": 0.0,\n",
" \"release_accepted\": false,\n",
" \"guard_contaminated\": false,\n",
" \"baseline_guard_rate\": 0.5\n",
"}\n",
"\n",
"Interpretation:\n",
"Development selects one candidate. Only that frozen candidate meets the declared guard threshold, and a reused guard invalidates the clean-release claim.\n",
"\n",
"Assumptions:\n",
"- Guard rows are paired with baseline by their position.\n",
"- Acceptance threshold is declared before observing guard outcomes.\n",
"- Guard is reserved for this single decision.\n",
"\n",
"Limitations:\n",
"- Passing a finite gate is not certification or a confidence guarantee.\n",
"- Reporting every candidate's guard result requires keeping those results out of future selection.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"chapter = 25\n",
"inputs = {'baseline_guard': [True, False, True, False],\n",
" 'minimum_guard_gain': 0.2,\n",
" 'guard_reused': False,\n",
" 'candidates': [{'name': 'flashy',\n",
" 'development': [True, True, True, True],\n",
" 'guard': [True, False, True, False]},\n",
" {'name': 'steady',\n",
" 'development': [True, True, True, False],\n",
" 'guard': [True, True, True, False]}]}\n",
"report = analyze(chapter, inputs)\n",
"# This input was explicitly taken from the teaching fixture.\n",
"report['evidence_kind'] = 'constructed teaching example'\n",
"print(report_text(report))"
]
},
{
"cell_type": "markdown",
"id": "4a54da0330ae",
"metadata": {},
"source": [
"Flashy wins development at 1.0, above steady at 0.75. Flashy's default guard equals the baseline, so gain is 0 and release is rejected against threshold 0.2. Steady's guard gain is 0.25, but selecting it after seeing that result would turn guard into another development set.\n",
"\n",
"In the changed case, flashy repairs one baseline failure, giving guard gain 0.25. It now meets the declared threshold and is accepted because the guard is not reused. That pass means the finite procedural gate passed. It does not establish a confidence guarantee or remove the need for reversible deployment and monitoring.\n",
"\n",
"The plot below uses the calculated quantities. Read each panel's units before comparing its values."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "590f57e74048",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Matplotlib is building the font cache; this may take a moment.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(SVG(figure_svg(report)))"
]
},
{
"cell_type": "markdown",
"id": "b5895ad9d027",
"metadata": {},
"source": [
"**Figure 25.L1:** Calculated chapter experiment. Each panel labels its input and output units; interpret it under the assumptions printed in the report."
]
},
{
"cell_type": "markdown",
"id": "67cb57ef7cdc",
"metadata": {},
"source": [
"## Change the assumption\n",
"\n",
"The most direct contamination failure is choosing the candidate with the best guard score instead of the development winner. Another is repeatedly tuning a procedure after every failed guard result while continuing to call the same data untouched. Both make the guard participate in improvement.\n",
"\n",
"The transfer case has a large favorable gain but guard_reused=true. The method rejects a clean release claim because the evidence no longer has its reserved role. The gain remains a descriptive quantity; contamination does not make arithmetic disappear.\n",
"\n",
"A gate can also fail through undefined pairing or a post hoc threshold. Here positions define matched guard tasks, so the caller must preserve that meaning. Choose the threshold before inspecting outcomes and keep candidate, data, and procedure identifiers in the record. Passing this small deterministic rule is not certification of a generally improved agent."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "cb2b4493b968",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 25: development-guard-gate\n",
"Should the development winner be accepted under a separate release guard?\n",
"Evidence: constructed changed-assumption example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"selected_candidate\": \"flashy\",\n",
" \"selected_guard_gain\": 0.25,\n",
" \"release_accepted\": true,\n",
" \"guard_contaminated\": false,\n",
" \"baseline_guard_rate\": 0.5\n",
"}\n",
"\n",
"Interpretation:\n",
"Development selects one candidate. Only that frozen candidate meets the declared guard threshold, and a reused guard invalidates the clean-release claim.\n",
"\n",
"Assumptions:\n",
"- Guard rows are paired with baseline by their position.\n",
"- Acceptance threshold is declared before observing guard outcomes.\n",
"- Guard is reserved for this single decision.\n",
"\n",
"Limitations:\n",
"- Passing a finite gate is not certification or a confidence guarantee.\n",
"- Reporting every candidate's guard result requires keeping those results out of future selection.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"changed_inputs = {'baseline_guard': [True, False, True, False],\n",
" 'minimum_guard_gain': 0.2,\n",
" 'guard_reused': False,\n",
" 'candidates': [{'name': 'flashy',\n",
" 'development': [True, True, True, True],\n",
" 'guard': [True, True, True, False]},\n",
" {'name': 'steady',\n",
" 'development': [True, True, True, False],\n",
" 'guard': [True, True, True, False]}]}\n",
"changed = analyze(chapter, changed_inputs)\n",
"changed['evidence_kind'] = 'constructed changed-assumption example'\n",
"print(report_text(changed))\n",
"display(SVG(figure_svg(changed)))"
]
},
{
"cell_type": "markdown",
"id": "0570c1a725c0",
"metadata": {},
"source": [
"**Figure 25.L2:** The changed-assumption result. Compare the printed quantities and the stated assumptions with the first run. A different input need not imply a causal effect in a deployed agent."
]
},
{
"cell_type": "markdown",
"id": "e86f717a9fa1",
"metadata": {},
"source": [
"## Try a new case\n",
"\n",
"The transfer proposal has guard success 1 compared with baseline 0.5, for gain 0.5. It exceeds threshold 0.1, but release remains rejected because the guard was reused. This separates favorable observed evidence from a valid acceptance protocol.\n",
"\n",
"For local improvement work, freeze the candidate and its runtime before opening guard outcomes. Record the source of every prior evaluation and preserve a genuinely new guard after contamination. If uncertainty control is required, design that statistical rule explicitly rather than assigning a scientific meaning to a convenient numeric threshold. Keep rollback and monitoring outside the claim that the finite gate itself makes deployment safe."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a5c90877998f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 25: development-guard-gate\n",
"Should the development winner be accepted under a separate release guard?\n",
"Evidence: constructed transfer example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"selected_candidate\": \"proposal\",\n",
" \"selected_guard_gain\": 0.5,\n",
" \"release_accepted\": false,\n",
" \"guard_contaminated\": true,\n",
" \"baseline_guard_rate\": 0.5\n",
"}\n",
"\n",
"Interpretation:\n",
"Development selects one candidate. Only that frozen candidate meets the declared guard threshold, and a reused guard invalidates the clean-release claim.\n",
"\n",
"Assumptions:\n",
"- Guard rows are paired with baseline by their position.\n",
"- Acceptance threshold is declared before observing guard outcomes.\n",
"- Guard is reserved for this single decision.\n",
"\n",
"Limitations:\n",
"- Passing a finite gate is not certification or a confidence guarantee.\n",
"- Reporting every candidate's guard result requires keeping those results out of future selection.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"transfer_inputs = {'baseline_guard': [False, True],\n",
" 'minimum_guard_gain': 0.1,\n",
" 'guard_reused': True,\n",
" 'candidates': [{'name': 'proposal', 'development': [True, True], 'guard': [True, True]}]}\n",
"transfer = analyze(chapter, transfer_inputs)\n",
"transfer['evidence_kind'] = 'constructed transfer example'\n",
"print(report_text(transfer))"
]
},
{
"cell_type": "markdown",
"id": "f36d08300140",
"metadata": {},
"source": [
"## Apply the method to your inputs\n",
"\n",
"The example file below has the exact input shape the method accepts. Copy it to a new file, replace its values, then point `reader_file` at your copy. Run the cell again. Supplied inputs retain their stated provenance; the program cannot establish that they are representative observations.\n",
"\n",
"- **baseline_guard:** Nonempty binary baseline outcomes aligned with each candidate guard.\n",
"- **minimum_guard_gain:** Finite prespecified candidate-minus-baseline mean threshold.\n",
"- **guard_reused:** Exact boolean stating whether guard evidence influenced improvement or prior selection.\n",
"- **candidates:** Nonempty rows with name,development:binary list,guard:binary list matched to baseline_guard."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "faeb73f10ca3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 25: development-guard-gate\n",
"Should the development winner be accepted under a separate release guard?\n",
"Evidence: supplied local inputs; provenance not independently verified\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"selected_candidate\": \"proposal\",\n",
" \"selected_guard_gain\": 0.5,\n",
" \"release_accepted\": false,\n",
" \"guard_contaminated\": true,\n",
" \"baseline_guard_rate\": 0.5\n",
"}\n",
"\n",
"Interpretation:\n",
"Development selects one candidate. Only that frozen candidate meets the declared guard threshold, and a reused guard invalidates the clean-release claim.\n",
"\n",
"Assumptions:\n",
"- Guard rows are paired with baseline by their position.\n",
"- Acceptance threshold is declared before observing guard outcomes.\n",
"- Guard is reserved for this single decision.\n",
"\n",
"Limitations:\n",
"- Passing a finite gate is not certification or a confidence guarantee.\n",
"- Reporting every candidate's guard result requires keeping those results out of future selection.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"reader_file = LAB_ROOT / 'data/examples/ch25.json'\n",
"reader_inputs = json.loads(reader_file.read_text())\n",
"reader_report = analyze(chapter, reader_inputs)\n",
"print(report_text(reader_report))"
]
},
{
"cell_type": "markdown",
"id": "cd10944f624f",
"metadata": {},
"source": [
"## Questions\n",
"\n",
"1. Why not release steady by default?\n",
"\n",
"2. What changed gain allows flashy to pass?\n",
"\n",
"3. Does contamination erase observed gain?\n",
"\n",
"Answers: [separate solutions](../solutions/ch25.md). Try the calculation before opening them."
]
},
{
"cell_type": "markdown",
"id": "d532eacfc3b5",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"Development chooses the procedure; reserved guard evidence checks the frozen choice. The default winner fails, the changed winner passes a finite threshold, and the transfer gain cannot support clean acceptance after reuse. These distinctions prevent adaptation from masquerading as independent validation. The gate is an inspectable protocol under declared pairing and history, not certification. Preserve data roles, thresholds, and candidate identity before making a release claim.\n",
"\n",
"Limits of this experiment:\n",
"\n",
"- This is a local calculation under declared inputs, not an empirical claim about a deployed agent.\n",
"- Read the returned assumptions and limitations before applying the numerical result.\n",
"\n",
"The assistant skill is [`maa-25-development-guard-gate`](../skills/maa-25-development-guard-gate/SKILL.md). It uses this notebook's tested computation and input contract."
]
},
{
"cell_type": "markdown",
"id": "9cc9f25ba898",
"metadata": {},
"source": [
"## Equations from the chapter\n",
"\n",
"These are the unchanged display equations and their explanations from the canonical chapter. They are a reference for the experiment, not a claim that every equation is numerically implemented by this one method."
]
},
{
"cell_type": "markdown",
"id": "dc4a226c5e88",
"metadata": {},
"source": [
"### Equation 25.1\n",
"\n",
"\n",
"\n",
"Names a proposed procedure version while keeping the underlying model parameters frozen.\n",
"\n",
"`e_t` can lead to a change in `\\phi`; it does not make a claim about a new value of `\\theta`.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\phi'\\in\\mathcal M(\\phi_t,e_t), \\qquad\n",
"\\theta_{t+1}=\\theta_t.\n",
"\\tag{25.1}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "07c741226578",
"metadata": {},
"source": [
"### Equation 25.2\n",
"\n",
"\n",
"\n",
"Selects a development candidate, then defines its paired guard estimate.\n",
"\n",
"Development chooses `\\phi'`; the guard averages bounded parent-candidate differences after that choice is fixed.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\begin{aligned}\n",
"&\\widehat\\Delta_D(\\phi_j)=\\widehat V_D(\\phi_j)-\\widehat V_D(\\phi_t),\n",
"\\qquad \\phi'=\\arg\\max_{\\phi_j\\in\\Phi_t}\\widehat\\Delta_D(\\phi_j),\\\\\n",
"&Z_i(\\phi')\\in[-1,1], \\qquad\n",
"\\widehat\\Delta_G(\\phi')=\\frac{1}{m}\\sum_{i=1}^{m}Z_i(\\phi').\n",
"\\end{aligned}\n",
"\\tag{25.2}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "62cac796258d",
"metadata": {},
"source": [
"### Equation 25.3\n",
"\n",
"\n",
"\n",
"Bounds one fixed candidate's false acceptance and shows the stated ten-decision scale.\n",
"\n",
"Both values depend on the frozen-candidate, bounded-outcome, and independent-guard assumptions.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\Pr\\!\\left(\\widehat\\Delta_G(\\phi')\\geq 0.25\\right) \\leq\n",
"\\exp\\!\\left(-\\frac{200(0.25)^2}{2}\\right) \\approx 0.00193\n",
"\\quad\\text{for every fixed }\\phi'\\text{ with }\\Delta(\\phi')\\leq0,\n",
"\\qquad 10(0.00193)\\approx0.0193.\n",
"\\tag{25.3}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "157001f43774",
"metadata": {},
"source": [
"### Equation 25.4\n",
"\n",
"\n",
"\n",
"Admits a procedure version only when uplift, cost, authority, and rollback each pass their own test.\n",
"\n",
"A high uplift cannot compensate for missing authority or an untested return path.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{accept}(\\phi')=1 \\iff \\begin{cases}\n",
"\\widehat\\Delta_G(\\phi')\\geq\\tau,\\\\\n",
"\\widehat C_G(\\phi')\\leq c-\\varepsilon_{\\mathrm{safe}},\\\\\n",
"\\operatorname{Authorized}_{\\mathcal G}(\\phi')=1,\\\\\n",
"\\operatorname{Rollback}(\\phi',\\phi_t)=1.\n",
"\\end{cases}\n",
"\\tag{25.4}\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Mathematics of AI Agents",
"language": "python",
"name": "maa-lab"
},
"lab_chapter": 25,
"lab_execution": {
"code_cells": 6,
"created_utc": "2026-10-02T05:16:41.842749+00:00",
"elapsed_seconds": 8.890727207995951,
"method": "fresh process; new ipykernel InProcessKernelManager; cells submitted as Jupyter execute requests",
"network_transport_tested": false,
"python": "3.11.15",
"source_sha256": "da5839162a0a82eb1d9c5560c6bde819ccb53d774875158b28c3440d6c0b5559"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}