{
"cells": [
{
"cell_type": "markdown",
"id": "22ed7d0fc41e",
"metadata": {},
"source": [
"# Chapter 16: Thought as Search\n",
"\n",
"Generating another answer can improve the chance that some answer is correct. It does not ensure that the controller can recognize it. A selector that shares the same misconception can confidently choose the wrong candidate while total compute rises.\n",
"\n",
"This notebook separates coverage, selection, cost, and time. You provide several candidate allocations, a correctness construction, and a conditional selector accuracy. The method removes allocations that miss the budget or deadline, then chooses the feasible one with the greatest declared selected success. The changed case replaces independent candidate errors with a shared failure condition. Marginal correctness stays fixed while the benefit of additional sampling changes.\n",
"\n",
"**Outcome:** Separate candidate coverage from actual selector success and choose a feasible allocation.\n",
"\n",
"- Inspect the declared input contract\n",
"- Predict the hand-checkable case\n",
"- Run the shared computation\n",
"- Change the critical assumption\n",
"- Apply the method to the transfer data\n",
"\n",
"**Guided route:** Run the worked calculation, inspect its figure, change the stated assumption, and try the transfer case. Read the explanations beside each result before opening the answers.\n",
"\n",
"**Deeper route:** First read the mathematics and canonical equation reference. Audit the input contract, predict the changed result, then inspect the shared chapter implementation and solve the questions independently. Both routes use the same calculations and preserve the equations."
]
},
{
"cell_type": "markdown",
"id": "a7781f3ee7dc",
"metadata": {},
"source": [
"## Technical Requirements\n",
"\n",
"Python 3.11 or later, the complete laboratory folder, and the notebook dependencies listed in `requirements-notebooks.txt` (the launcher's **Install notebook tools** choice installs them; see START-HERE). Standard-library chapter commands also support Python 3.10. No API key, model account or network call is used by this experiment.\n",
"\n",
"Prior knowledge:\n",
"\n",
"- Python lists and dictionaries\n",
"- The mapped chapter and its notation"
]
},
{
"cell_type": "markdown",
"id": "97b0b4892f23",
"metadata": {},
"source": [
"## The question and its mathematics\n",
"\n",
"Under independent equal candidate correctness p, the probability that at least one of n candidates is correct is C_n=1-(1-p)^n. Under one shared good/bad condition, C_n=p for every positive n. Equal marginals do not distinguish those joint laws.\n",
"\n",
"For n>1 with independent errors, this implementation defines selected success as q C_n, where q is the probability that the selector chooses a correct candidate conditional on one existing. Under the shared condition a bank that holds a correct candidate holds only correct candidates, so any choice from it is correct and selected success equals C_n=p whatever q is. When n=1 the selector is bypassed, so success is p. This convention matters: with independent errors, paying an unreliable selector can make two samples worse than one.\n",
"\n",
"Expense is n c_sample+c_selector and serial latency is n t_sample+t_selector for multiple candidates. Single-candidate allocation omits selector overhead. Feasibility requires both quantities to lie within declared limits. The selected allocation maximizes success, breaking ties toward lower expense. A candidate count with high oracle coverage is not necessarily feasible or operationally best."
]
},
{
"cell_type": "markdown",
"id": "52969f26650d",
"metadata": {},
"source": [
"## A calculation you can run\n",
"\n",
"Provide candidate counts and all probability, cost, latency, deadline, and dependence inputs. Numeric validation rejects negative resource values, invalid probabilities, and unsupported counts. For each allocation the function calculates coverage, selected success, expense, latency, and feasibility, preserving every row for inspection.\n",
"\n",
"The two figures compare oracle coverage and selector completion by sample count. Their gap is the selector boundary, not an error bar. The selected allocation may skip the visually highest point if it violates a resource constraint. In the changed case keep every scalar fixed and enable shared_error. Predict which row now wins and explain why sampling no longer improves coverage. Export dependence and selector assumptions with the result.\n",
"\n",
"The next cell finds the bundle and imports the same computation used by the chapter skill. It does not change your system Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "42ebefc0e5c1",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys, json\n",
"LAB_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"lab-manifest.json\").is_file()), None)\n",
"if LAB_ROOT is None:\n",
" raise RuntimeError(\"Open this notebook from the complete extracted laboratory folder.\")\n",
"sys.path.insert(0, str(LAB_ROOT / \"src\"))\n",
"from math_ai_agents.core import analyze, report_text\n",
"from math_ai_agents.plotting import figure_svg\n",
"from IPython.display import SVG, display\n"
]
},
{
"cell_type": "markdown",
"id": "7fffde8c0edb",
"metadata": {},
"source": [
"Set the declared inputs below. These are constructed teaching values, not measurements from a production agent. Change a value only after predicting what it should change."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "8eb1bb66bc10",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 16: sample-allocation\n",
"How many candidates should the controller generate under cost and deadline constraints?\n",
"Evidence: constructed teaching example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"selected_allocation\": {\n",
" \"samples\": 5,\n",
" \"coverage\": 0.92224,\n",
" \"selected_success\": 0.830016,\n",
" \"cost\": 6.0,\n",
" \"latency\": 6.0,\n",
" \"feasible\": true\n",
" },\n",
" \"allocations\": [\n",
" {\n",
" \"samples\": 1,\n",
" \"coverage\": 0.4,\n",
" \"selected_success\": 0.4,\n",
" \"cost\": 1.0,\n",
" \"latency\": 1.0,\n",
" \"feasible\": true\n",
" },\n",
" {\n",
" \"samples\": 2,\n",
" \"coverage\": 0.64,\n",
" \"selected_success\": 0.576,\n",
" \"cost\": 3.0,\n",
" \"latency\": 3.0,\n",
" \"feasible\": true\n",
" },\n",
" {\n",
" \"samples\": 3,\n",
" \"coverage\": 0.784,\n",
" \"selected_success\": 0.7056,\n",
" \"cost\": 4.0,\n",
" \"latency\": 4.0,\n",
" \"feasible\": true\n",
" },\n",
" {\n",
" \"samples\": 5,\n",
" \"coverage\": 0.92224,\n",
" \"selected_success\": 0.830016,\n",
" \"cost\": 6.0,\n",
" \"latency\": 6.0,\n",
" \"feasible\": true\n",
" }\n",
" ],\n",
" \"shared_error\": false\n",
"}\n",
"\n",
"Interpretation:\n",
"Candidate coverage and selected success are different quantities. A correlated shared failure prevents independent-sampling gains; deadlines can remove the largest allocation.\n",
"\n",
"Assumptions:\n",
"- With independent errors, the selector succeeds with probability q conditional on at least one correct candidate.\n",
"- Under a shared failure a bank with a correct candidate holds only correct candidates, so selection succeeds whenever coverage does and q is not used.\n",
"- One candidate bypasses the selector.\n",
"- Latency is serial and all candidates have the same declared marginal success.\n",
"\n",
"Limitations:\n",
"- Selector accuracy can depend on candidate quality in real workflows.\n",
"- More samples do not establish correctness when errors or verifier agreement are shared.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"chapter = 16\n",
"inputs = {'candidate_success': 0.4,\n",
" 'selector_accuracy': 0.9,\n",
" 'sample_cost': 1,\n",
" 'sample_latency': 1,\n",
" 'selector_cost': 1,\n",
" 'selector_latency': 1,\n",
" 'deadline': 6,\n",
" 'budget': 6,\n",
" 'shared_error': False,\n",
" 'sample_counts': [1, 2, 3, 5]}\n",
"report = analyze(chapter, inputs)\n",
"# This input was explicitly taken from the teaching fixture.\n",
"report['evidence_kind'] = 'constructed teaching example'\n",
"print(report_text(report))"
]
},
{
"cell_type": "markdown",
"id": "5b19e73937e1",
"metadata": {},
"source": [
"Default coverage at n=5 is 1-0.6^5=0.92224. Selected success is 0.9 times that value, or 0.830016. Cost and serial latency are both 6, so this allocation exactly fits the limits and wins among the listed options.\n",
"\n",
"Under shared error, every multiple-candidate row has coverage 0.4 and selected success 0.4: a bank with a correct candidate holds only correct candidates, so the selector's 0.9 accuracy makes no difference. The single candidate bypasses selection and also delivers 0.4, with lower expense. It therefore wins the tie. More samples now add cost without improving justified completion, even though each candidate has the same marginal correctness as before.\n",
"\n",
"The plot below uses the calculated quantities. Read each panel's units before comparing its values."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "590f57e74048",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Matplotlib is building the font cache; this may take a moment.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(SVG(figure_svg(report)))"
]
},
{
"cell_type": "markdown",
"id": "e0e05321aebf",
"metadata": {},
"source": [
"**Figure 16.L1:** Calculated chapter experiment. Each panel labels its input and output units; interpret it under the assumptions printed in the report."
]
},
{
"cell_type": "markdown",
"id": "a98184c9c338",
"metadata": {},
"source": [
"## Change the assumption\n",
"\n",
"The formula q C_n for independent errors is a declared selector model. Real selector accuracy can depend on candidate count, task difficulty, disagreement, or the distribution of wrong answers. A constant q should not be adopted merely because it makes the curve easy to draw. Measure it on held-out candidate sets or state its uncertainty.\n",
"\n",
"Independence is another strong assumption. Repeated samples from one shared flawed context can agree for the wrong reason. The changed construction is deliberately extreme, showing why marginal correctness alone cannot justify the usual saturation curve.\n",
"\n",
"Resource models also need care. Parallel generation changes latency, and a deadline can alter selector behavior. This notebook uses serial time and fixed overhead. If no row is feasible, the selected allocation is unavailable; it does not invent a smaller budget or recommend exceeding a hard limit."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "3375d33d88bd",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 16: sample-allocation\n",
"How many candidates should the controller generate under cost and deadline constraints?\n",
"Evidence: constructed changed-assumption example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"selected_allocation\": {\n",
" \"samples\": 1,\n",
" \"coverage\": 0.4,\n",
" \"selected_success\": 0.4,\n",
" \"cost\": 1.0,\n",
" \"latency\": 1.0,\n",
" \"feasible\": true\n",
" },\n",
" \"allocations\": [\n",
" {\n",
" \"samples\": 1,\n",
" \"coverage\": 0.4,\n",
" \"selected_success\": 0.4,\n",
" \"cost\": 1.0,\n",
" \"latency\": 1.0,\n",
" \"feasible\": true\n",
" },\n",
" {\n",
" \"samples\": 2,\n",
" \"coverage\": 0.4,\n",
" \"selected_success\": 0.4,\n",
" \"cost\": 3.0,\n",
" \"latency\": 3.0,\n",
" \"feasible\": true\n",
" },\n",
" {\n",
" \"samples\": 3,\n",
" \"coverage\": 0.4,\n",
" \"selected_success\": 0.4,\n",
" \"cost\": 4.0,\n",
" \"latency\": 4.0,\n",
" \"feasible\": true\n",
" },\n",
" {\n",
" \"samples\": 5,\n",
" \"coverage\": 0.4,\n",
" \"selected_success\": 0.4,\n",
" \"cost\": 6.0,\n",
" \"latency\": 6.0,\n",
" \"feasible\": true\n",
" }\n",
" ],\n",
" \"shared_error\": true\n",
"}\n",
"\n",
"Interpretation:\n",
"Candidate coverage and selected success are different quantities. A correlated shared failure prevents independent-sampling gains; deadlines can remove the largest allocation.\n",
"\n",
"Assumptions:\n",
"- With independent errors, the selector succeeds with probability q conditional on at least one correct candidate.\n",
"- Under a shared failure a bank with a correct candidate holds only correct candidates, so selection succeeds whenever coverage does and q is not used.\n",
"- One candidate bypasses the selector.\n",
"- Latency is serial and all candidates have the same declared marginal success.\n",
"\n",
"Limitations:\n",
"- Selector accuracy can depend on candidate quality in real workflows.\n",
"- More samples do not establish correctness when errors or verifier agreement are shared.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"changed_inputs = {'candidate_success': 0.4,\n",
" 'selector_accuracy': 0.9,\n",
" 'sample_cost': 1,\n",
" 'sample_latency': 1,\n",
" 'selector_cost': 1,\n",
" 'selector_latency': 1,\n",
" 'deadline': 6,\n",
" 'budget': 6,\n",
" 'shared_error': True,\n",
" 'sample_counts': [1, 2, 3, 5]}\n",
"changed = analyze(chapter, changed_inputs)\n",
"changed['evidence_kind'] = 'constructed changed-assumption example'\n",
"print(report_text(changed))\n",
"display(SVG(figure_svg(changed)))"
]
},
{
"cell_type": "markdown",
"id": "b1e67d4df498",
"metadata": {},
"source": [
"**Figure 16.L2:** The changed-assumption result. Compare the printed quantities and the stated assumptions with the first run. A different input need not imply a causal effect in a deployed agent."
]
},
{
"cell_type": "markdown",
"id": "2e152387c7fc",
"metadata": {},
"source": [
"## Try a new case\n",
"\n",
"The transfer case gives single-candidate success 0.7. Multiple candidates face selector accuracy 0.5 and overhead that makes them miss the three-unit deadline or six-unit budget. The feasible recommendation is one candidate, even though an oracle would see broader coverage in a larger bank.\n",
"\n",
"For local allocation data, distinguish candidate success evidence from selector evidence. Record whether failures are independent, shared, or simply unknown. Unknown dependence calls for a measurement design or sensitivity constructions, not an automatic independence assumption. Use run-level evaluation after executing the chosen allocation to test actual authorized confirmed completion."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "71222a267fb7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 16: sample-allocation\n",
"How many candidates should the controller generate under cost and deadline constraints?\n",
"Evidence: constructed transfer example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"selected_allocation\": {\n",
" \"samples\": 1,\n",
" \"coverage\": 0.7,\n",
" \"selected_success\": 0.7,\n",
" \"cost\": 2.0,\n",
" \"latency\": 1.0,\n",
" \"feasible\": true\n",
" },\n",
" \"allocations\": [\n",
" {\n",
" \"samples\": 1,\n",
" \"coverage\": 0.7,\n",
" \"selected_success\": 0.7,\n",
" \"cost\": 2.0,\n",
" \"latency\": 1.0,\n",
" \"feasible\": true\n",
" },\n",
" {\n",
" \"samples\": 2,\n",
" \"coverage\": 0.91,\n",
" \"selected_success\": 0.455,\n",
" \"cost\": 7.0,\n",
" \"latency\": 4.0,\n",
" \"feasible\": false\n",
" },\n",
" {\n",
" \"samples\": 4,\n",
" \"coverage\": 0.9919,\n",
" \"selected_success\": 0.49595,\n",
" \"cost\": 11.0,\n",
" \"latency\": 6.0,\n",
" \"feasible\": false\n",
" }\n",
" ],\n",
" \"shared_error\": false\n",
"}\n",
"\n",
"Interpretation:\n",
"Candidate coverage and selected success are different quantities. A correlated shared failure prevents independent-sampling gains; deadlines can remove the largest allocation.\n",
"\n",
"Assumptions:\n",
"- With independent errors, the selector succeeds with probability q conditional on at least one correct candidate.\n",
"- Under a shared failure a bank with a correct candidate holds only correct candidates, so selection succeeds whenever coverage does and q is not used.\n",
"- One candidate bypasses the selector.\n",
"- Latency is serial and all candidates have the same declared marginal success.\n",
"\n",
"Limitations:\n",
"- Selector accuracy can depend on candidate quality in real workflows.\n",
"- More samples do not establish correctness when errors or verifier agreement are shared.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"transfer_inputs = {'candidate_success': 0.7,\n",
" 'selector_accuracy': 0.5,\n",
" 'sample_cost': 2,\n",
" 'sample_latency': 1,\n",
" 'selector_cost': 3,\n",
" 'selector_latency': 2,\n",
" 'deadline': 3,\n",
" 'budget': 6,\n",
" 'shared_error': False,\n",
" 'sample_counts': [1, 2, 4]}\n",
"transfer = analyze(chapter, transfer_inputs)\n",
"transfer['evidence_kind'] = 'constructed transfer example'\n",
"print(report_text(transfer))"
]
},
{
"cell_type": "markdown",
"id": "dd1ea098f2d1",
"metadata": {},
"source": [
"## Apply the method to your inputs\n",
"\n",
"The example file below has the exact input shape the method accepts. Copy it to a new file, replace its values, then point `reader_file` at your copy. Run the cell again. Supplied inputs retain their stated provenance; the program cannot establish that they are representative observations.\n",
"\n",
"- **candidate_success:** Marginal correctness probability in [0,1].\n",
"- **selector_accuracy:** Conditional probability of choosing a correct candidate when one exists.\n",
"- **sample_cost:** Nonnegative cost per generated candidate.\n",
"- **sample_latency:** Nonnegative serial time per candidate.\n",
"- **selector_cost:** Nonnegative selector expense for n>1.\n",
"- **selector_latency:** Nonnegative selector time for n>1.\n",
"- **deadline:** Nonnegative total time limit.\n",
"- **budget:** Nonnegative total expense limit.\n",
"- **shared_error:** Exact boolean choosing shared versus independent correctness construction.\n",
"- **sample_counts:** Nonempty list of positive integer candidate counts up to 1000."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "9d9e0d5e8743",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 16: sample-allocation\n",
"How many candidates should the controller generate under cost and deadline constraints?\n",
"Evidence: supplied local inputs; provenance not independently verified\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"selected_allocation\": {\n",
" \"samples\": 1,\n",
" \"coverage\": 0.7,\n",
" \"selected_success\": 0.7,\n",
" \"cost\": 2.0,\n",
" \"latency\": 1.0,\n",
" \"feasible\": true\n",
" },\n",
" \"allocations\": [\n",
" {\n",
" \"samples\": 1,\n",
" \"coverage\": 0.7,\n",
" \"selected_success\": 0.7,\n",
" \"cost\": 2.0,\n",
" \"latency\": 1.0,\n",
" \"feasible\": true\n",
" },\n",
" {\n",
" \"samples\": 2,\n",
" \"coverage\": 0.91,\n",
" \"selected_success\": 0.455,\n",
" \"cost\": 7.0,\n",
" \"latency\": 4.0,\n",
" \"feasible\": false\n",
" },\n",
" {\n",
" \"samples\": 4,\n",
" \"coverage\": 0.9919,\n",
" \"selected_success\": 0.49595,\n",
" \"cost\": 11.0,\n",
" \"latency\": 6.0,\n",
" \"feasible\": false\n",
" }\n",
" ],\n",
" \"shared_error\": false\n",
"}\n",
"\n",
"Interpretation:\n",
"Candidate coverage and selected success are different quantities. A correlated shared failure prevents independent-sampling gains; deadlines can remove the largest allocation.\n",
"\n",
"Assumptions:\n",
"- With independent errors, the selector succeeds with probability q conditional on at least one correct candidate.\n",
"- Under a shared failure a bank with a correct candidate holds only correct candidates, so selection succeeds whenever coverage does and q is not used.\n",
"- One candidate bypasses the selector.\n",
"- Latency is serial and all candidates have the same declared marginal success.\n",
"\n",
"Limitations:\n",
"- Selector accuracy can depend on candidate quality in real workflows.\n",
"- More samples do not establish correctness when errors or verifier agreement are shared.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"reader_file = LAB_ROOT / 'data/examples/ch16.json'\n",
"reader_inputs = json.loads(reader_file.read_text())\n",
"reader_report = analyze(chapter, reader_inputs)\n",
"print(report_text(reader_report))"
]
},
{
"cell_type": "markdown",
"id": "3038128e14f2",
"metadata": {},
"source": [
"## Questions\n",
"\n",
"1. Compute default n=5 coverage and selection.\n",
"\n",
"2. Why prefer n1 under shared error?\n",
"\n",
"3. What if no allocation is feasible?\n",
"\n",
"Answers: [separate solutions](../solutions/ch16.md). Try the calculation before opening them."
]
},
{
"cell_type": "markdown",
"id": "f5686d9d6069",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"Sampling, selection, and feasible completion are separate layers. Independent coverage can rise with candidate count, while shared failures prevent that gain and selector mistakes consume it. Cost and deadlines filter allocations before choice. The default uses five samples, the shared-error case prefers one, and the transfer checks overhead limits. The result is conditional on declared probabilities and serial resource accounting, without turning repeated agreement into correctness evidence.\n",
"\n",
"Limits of this experiment:\n",
"\n",
"- This is a local calculation under declared inputs, not an empirical claim about a deployed agent.\n",
"- Read the returned assumptions and limitations before applying the numerical result.\n",
"\n",
"The assistant skill is [`maa-16-sample-allocation`](../skills/maa-16-sample-allocation/SKILL.md). It uses this notebook's tested computation and input contract."
]
},
{
"cell_type": "markdown",
"id": "9cc9f25ba898",
"metadata": {},
"source": [
"## Equations from the chapter\n",
"\n",
"These are the unchanged display equations and their explanations from the canonical chapter. They are a reference for the experiment, not a claim that every equation is numerically implemented by this one method."
]
},
{
"cell_type": "markdown",
"id": "d5cef1725a54",
"metadata": {},
"source": [
"### Equation 16.1\n",
"\n",
"\n",
"\n",
"Equation (16.1) gives the chance that at least one of `k` samples is correct.\n",
"\n",
"One minus the chance that every single sample is wrong.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{Cov}(k) \\;=\\; 1 - (1-p)^{k}.\n",
"\\tag{16.1}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "4d9e54dd4879",
"metadata": {},
"source": [
"### Equation 16.2\n",
"\n",
"\n",
"\n",
"Equation (16.2) gives the chance the returned sample is correct when the selector is blind.\n",
"\n",
"The expected fraction of correct samples among the `k` drawn is `p`, whatever `k` is.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{Sel}(k) \\;=\\; \\mathrm{E}\\!\\left[\\frac{m}{k}\\right] \\;=\\; p .\n",
"\\tag{16.2}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "8aad8a3910c1",
"metadata": {},
"source": [
"### Equation 16.3\n",
"\n",
"\n",
"\n",
"Equation (16.3) splits the returned answer's correctness into whether a correct sample was present and whether the selector found it.\n",
"\n",
"Multiply the chance that a correct sample exists among the `k` by the chance the selector ranks one of them top.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{Sel}(k) \\;=\\; \\operatorname{Cov}(k)\\cdot \\pi(k) \\;+\\; \\big(1-\\operatorname{Cov}(k)\\big)\\cdot 0 ,\n",
"\\tag{16.3}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "8777858c8832",
"metadata": {},
"source": [
"### Equation 16.4\n",
"\n",
"\n",
"\n",
"Equation (16.4) gives the value of drawing one more sample, before any selector is applied.\n",
"\n",
"The gain from sample number `k` plus one is the chance it is correct times the chance all the previous ones were not.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{Cov}(k+1) - \\operatorname{Cov}(k) \\;=\\; p\\,(1-p)^{k}.\n",
"\\tag{16.4}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "bc71d060a344",
"metadata": {},
"source": [
"### Equation 16.5\n",
"\n",
"\n",
"\n",
"Equation (16.5) allocates all observed attempt costs across the observed authorized completions.\n",
"\n",
"Add every run's cost, then divide by the number of runs that actually satisfied the declared completion contract.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\widehat c_{\\mathrm{success}}\n",
"=\n",
"\\frac{\\sum_{i=1}^{N}c_i}\n",
"{\\sum_{i=1}^{N}\\mathbf 1\\{\\text{authorized confirmed completion in run }i\\}}.\n",
"\\tag{16.5}\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Mathematics of AI Agents",
"language": "python",
"name": "maa-lab"
},
"lab_chapter": 16,
"lab_execution": {
"code_cells": 6,
"created_utc": "2026-10-02T23:09:41.226879+00:00",
"elapsed_seconds": 8.639831667067483,
"method": "fresh process; new ipykernel InProcessKernelManager; cells submitted as Jupyter execute requests",
"network_transport_tested": false,
"python": "3.11.15",
"source_sha256": "b319ce088b5161f1b648729f464370b00343f9982dbefdf739549e4349b84c01"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}