{
"cells": [
{
"cell_type": "markdown",
"id": "933b04ae5aaa",
"metadata": {},
"source": [
"# Chapter 26: How Much More Could the System Become?\n",
"\n",
"A candidate bank contains a correct solution to every task. The controller still fails most tasks because its selector repeatedly chooses the wrong candidate. An oracle view of the bank has discovered available capability, not executable capability.\n",
"\n",
"This notebook places three layers side by side. Oracle coverage asks whether any candidate succeeds on each task. Actual selection checks the chosen candidate. Deployment filtering then removes successes that current authority does not permit. A separate independent-sampling curve illustrates saturation under a declared law. It is deliberately not fitted to the observed bank, because finite candidate overlap and independent future sampling are different assumptions.\n",
"\n",
"**Outcome:** Separate bank oracle coverage, actual choice, and permission-filtered success.\n",
"\n",
"- Inspect the declared input contract\n",
"- Predict the hand-checkable case\n",
"- Run the shared computation\n",
"- Change the critical assumption\n",
"- Apply the method to the transfer data\n",
"\n",
"**Guided route:** Run the worked calculation, inspect its figure, change the stated assumption, and try the transfer case. Read the explanations beside each result before opening the answers.\n",
"\n",
"**Deeper route:** First read the mathematics and canonical equation reference. Audit the input contract, predict the changed result, then inspect the shared chapter implementation and solve the questions independently. Both routes use the same calculations and preserve the equations."
]
},
{
"cell_type": "markdown",
"id": "a7781f3ee7dc",
"metadata": {},
"source": [
"## Technical Requirements\n",
"\n",
"Python 3.11 or later, the complete laboratory folder, and the notebook dependencies listed in `requirements-notebooks.txt` (the launcher's **Install notebook tools** choice installs them; see START-HERE). Standard-library chapter commands also support Python 3.10. No API key, model account or network call is used by this experiment.\n",
"\n",
"Prior knowledge:\n",
"\n",
"- Python lists and dictionaries\n",
"- The mapped chapter and its notation"
]
},
{
"cell_type": "markdown",
"id": "947facbfd177",
"metadata": {},
"source": [
"## The question and its mathematics\n",
"\n",
"Let S_ct be the binary success of candidate c on task t and w_t the normalized task weight. Oracle task coverage is O_t=max_c S_ct, giving weighted bank coverage sum_t w_t O_t. A supplied selector chooses c(t), so actual success is A_t=S_(c(t),t). With deployment permission D_t, deployed success is A_t D_t.\n",
"\n",
"Pointwise, deployed success<=actual success<=oracle coverage. Weighted values inherit that ordering. The inequalities condition on this bank, task distribution, interface, and permission contract. They are not a universal maximum over possible models or future procedures.\n",
"\n",
"The prefix coverage curve adds candidates in their supplied order and is nondecreasing. Candidate order changes that curve but not final full-bank coverage. The independent construction gives 1-(1-p)^n for equal independent candidate correctness p. Its diminishing increments are a property of that joint law, not evidence that actual candidate diversity is independent or that the bank predicts future progress."
]
},
{
"cell_type": "markdown",
"id": "2d629aa0f7d3",
"metadata": {},
"source": [
"## A calculation you can run\n",
"\n",
"Provide a rectangular candidate-by-task boolean matrix, normalized task weights, one selected candidate index per task, and one deployment permission flag per task. The code validates exact booleans and index ranges. It forms oracle, selected, and deployed task flags, then aggregates them under the target weights.\n",
"\n",
"The first figure shows prefix coverage of the supplied bank. The second shows a separate independent constructed saturation law. Their labels distinguish observed finite structure from a hypothetical sampling model. In the changed case improve selection while denying deployment on the heaviest task. Predict all three aggregate values before execution. Export the bank identity and task boundary whenever reporting a ceiling.\n",
"\n",
"The next cell finds the bundle and imports the same computation used by the chapter skill. It does not change your system Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "42ebefc0e5c1",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys, json\n",
"LAB_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"lab-manifest.json\").is_file()), None)\n",
"if LAB_ROOT is None:\n",
" raise RuntimeError(\"Open this notebook from the complete extracted laboratory folder.\")\n",
"sys.path.insert(0, str(LAB_ROOT / \"src\"))\n",
"from math_ai_agents.core import analyze, report_text\n",
"from math_ai_agents.plotting import figure_svg\n",
"from IPython.display import SVG, display\n"
]
},
{
"cell_type": "markdown",
"id": "7fffde8c0edb",
"metadata": {},
"source": [
"Set the declared inputs below. These are constructed teaching values, not measurements from a production agent. Change a value only after predicting what it should change."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "081c3bd44450",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 26: conditional-bank-ceiling\n",
"How much capability exists in this finite candidate bank, and how much can selection deploy?\n",
"Evidence: constructed teaching example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"bank_oracle_coverage\": 1.0,\n",
" \"actual_selection_success\": 0.2,\n",
" \"deployment_success\": 0.2,\n",
" \"oracle_task_flags\": [\n",
" true,\n",
" true,\n",
" true\n",
" ],\n",
" \"selection_task_flags\": [\n",
" true,\n",
" false,\n",
" false\n",
" ],\n",
" \"deployment_task_flags\": [\n",
" true,\n",
" false,\n",
" false\n",
" ]\n",
"}\n",
"\n",
"Interpretation:\n",
"Oracle coverage asks whether some candidate solves each task. Actual selection and deployment filtering can only reduce that finite-bank ceiling.\n",
"\n",
"Assumptions:\n",
"- Bank contents and task distribution are fixed.\n",
"- Selection is supplied per task; oracle selection is not assumed achievable.\n",
"- Independent saturation is a separate constructed sampling law.\n",
"\n",
"Limitations:\n",
"- The bank ceiling is not a universal agent limit or a forecast of future models.\n",
"- Unobserved tasks and candidate classes remain outside the bound.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"chapter = 26\n",
"inputs = {'success_matrix': [[True, False, False], [False, True, False], [False, False, True]],\n",
" 'task_weights': [0.2, 0.3, 0.5],\n",
" 'selected_candidates': [0, 0, 0],\n",
" 'deployment_allowed': [True, True, True],\n",
" 'independent_candidate_success': 0.4}\n",
"report = analyze(chapter, inputs)\n",
"# This input was explicitly taken from the teaching fixture.\n",
"report['evidence_kind'] = 'constructed teaching example'\n",
"print(report_text(report))"
]
},
{
"cell_type": "markdown",
"id": "6a35e7457901",
"metadata": {},
"source": [
"The default bank has one specialist for each of three tasks. Oracle coverage is 1. The selector chooses candidate 0 for every task, so actual success is only task 0's weight 0.2. All tasks are deployable, leaving deployed success 0.2.\n",
"\n",
"The changed selector chooses the correct specialist for every task, raising actual success to 1. Deployment denies task 2, whose weight is 0.5, so deployed success is 0.5. Oracle coverage remains 1. Better selection extracted available capability, but the authority boundary still limits what the system may realize. None of these numbers forecasts a future model's maximum.\n",
"\n",
"The plot below uses the calculated quantities. Read each panel's units before comparing its values."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "590f57e74048",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Matplotlib is building the font cache; this may take a moment.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(SVG(figure_svg(report)))"
]
},
{
"cell_type": "markdown",
"id": "7f437653674f",
"metadata": {},
"source": [
"**Figure 26.L1:** Calculated chapter experiment. Each panel labels its input and output units; interpret it under the assumptions printed in the report."
]
},
{
"cell_type": "markdown",
"id": "5ada7186d259",
"metadata": {},
"source": [
"## Change the assumption\n",
"\n",
"An oracle ceiling can be mistaken for a deployable system when a report omits the selector. The gap may be large even if every task has a correct candidate somewhere. Evaluating only bank coverage rewards availability without measuring the controller's ability to identify the right answer.\n",
"\n",
"Another failure transports the finite ceiling to unseen tasks or candidate classes. The stated maximum is conditional on the supplied bank and distribution. Adding a new candidate can raise it; changing task weights can change it; a new interface can change feasibility.\n",
"\n",
"The independent curve also becomes misleading under shared errors. Candidates may agree because they share a failure mechanism, not because repeated sampling created independent evidence. Use the finite success matrix to study actual overlap, and treat the independent curve as a labeled construction rather than an empirical extrapolation."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "ca523a79ab8f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 26: conditional-bank-ceiling\n",
"How much capability exists in this finite candidate bank, and how much can selection deploy?\n",
"Evidence: constructed changed-assumption example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"bank_oracle_coverage\": 1.0,\n",
" \"actual_selection_success\": 1.0,\n",
" \"deployment_success\": 0.5,\n",
" \"oracle_task_flags\": [\n",
" true,\n",
" true,\n",
" true\n",
" ],\n",
" \"selection_task_flags\": [\n",
" true,\n",
" true,\n",
" true\n",
" ],\n",
" \"deployment_task_flags\": [\n",
" true,\n",
" true,\n",
" false\n",
" ]\n",
"}\n",
"\n",
"Interpretation:\n",
"Oracle coverage asks whether some candidate solves each task. Actual selection and deployment filtering can only reduce that finite-bank ceiling.\n",
"\n",
"Assumptions:\n",
"- Bank contents and task distribution are fixed.\n",
"- Selection is supplied per task; oracle selection is not assumed achievable.\n",
"- Independent saturation is a separate constructed sampling law.\n",
"\n",
"Limitations:\n",
"- The bank ceiling is not a universal agent limit or a forecast of future models.\n",
"- Unobserved tasks and candidate classes remain outside the bound.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"changed_inputs = {'success_matrix': [[True, False, False], [False, True, False], [False, False, True]],\n",
" 'task_weights': [0.2, 0.3, 0.5],\n",
" 'selected_candidates': [0, 1, 2],\n",
" 'deployment_allowed': [True, True, False],\n",
" 'independent_candidate_success': 0.4}\n",
"changed = analyze(chapter, changed_inputs)\n",
"changed['evidence_kind'] = 'constructed changed-assumption example'\n",
"print(report_text(changed))\n",
"display(SVG(figure_svg(changed)))"
]
},
{
"cell_type": "markdown",
"id": "1cd6df369e71",
"metadata": {},
"source": [
"**Figure 26.L2:** The changed-assumption result. Compare the printed quantities and the stated assumptions with the first run. A different input need not imply a causal effect in a deployed agent."
]
},
{
"cell_type": "markdown",
"id": "483a03772c67",
"metadata": {},
"source": [
"## Try a new case\n",
"\n",
"The transfer bank has oracle coverage 1 because candidate 0 solves both tasks. Selection chooses candidate 1, which solves only task 0, so actual success is 0.5. Task 1 is denied, but it was already a selected failure, leaving deployed success 0.5.\n",
"\n",
"For local use, define the candidate bank and task success predicate before inspecting selection. Keep permission filtering outside correctness labels so a denied correct answer is not counted as a model error. Report the supported conditional ceiling together with actual selector performance and the missing measurements needed for any proposed expansion. Do not rename a finite bank result an open-ended capability maximum."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "6d57cb2872db",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 26: conditional-bank-ceiling\n",
"How much capability exists in this finite candidate bank, and how much can selection deploy?\n",
"Evidence: constructed transfer example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"bank_oracle_coverage\": 1.0,\n",
" \"actual_selection_success\": 0.5,\n",
" \"deployment_success\": 0.5,\n",
" \"oracle_task_flags\": [\n",
" true,\n",
" true\n",
" ],\n",
" \"selection_task_flags\": [\n",
" true,\n",
" false\n",
" ],\n",
" \"deployment_task_flags\": [\n",
" true,\n",
" false\n",
" ]\n",
"}\n",
"\n",
"Interpretation:\n",
"Oracle coverage asks whether some candidate solves each task. Actual selection and deployment filtering can only reduce that finite-bank ceiling.\n",
"\n",
"Assumptions:\n",
"- Bank contents and task distribution are fixed.\n",
"- Selection is supplied per task; oracle selection is not assumed achievable.\n",
"- Independent saturation is a separate constructed sampling law.\n",
"\n",
"Limitations:\n",
"- The bank ceiling is not a universal agent limit or a forecast of future models.\n",
"- Unobserved tasks and candidate classes remain outside the bound.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"transfer_inputs = {'success_matrix': [[True, True], [True, False]],\n",
" 'task_weights': [0.5, 0.5],\n",
" 'selected_candidates': [1, 1],\n",
" 'deployment_allowed': [True, False],\n",
" 'independent_candidate_success': 0.6}\n",
"transfer = analyze(chapter, transfer_inputs)\n",
"transfer['evidence_kind'] = 'constructed transfer example'\n",
"print(report_text(transfer))"
]
},
{
"cell_type": "markdown",
"id": "8bc05b438f5b",
"metadata": {},
"source": [
"## Apply the method to your inputs\n",
"\n",
"The example file below has the exact input shape the method accepts. Copy it to a new file, replace its values, then point `reader_file` at your copy. Run the cell again. Supplied inputs retain their stated provenance; the program cannot establish that they are representative observations.\n",
"\n",
"- **success_matrix:** Candidate by task matrix of exact boolean task success.\n",
"- **task_weights:** Normalized task-distribution weights.\n",
"- **selected_candidates:** Zero-based candidate index chosen for each task.\n",
"- **deployment_allowed:** Exact boolean deployment permission for each task.\n",
"- **independent_candidate_success:** Probability for a separate constructed independent-candidate saturation curve."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "12a4ce52df94",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 26: conditional-bank-ceiling\n",
"How much capability exists in this finite candidate bank, and how much can selection deploy?\n",
"Evidence: supplied local inputs; provenance not independently verified\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"bank_oracle_coverage\": 1.0,\n",
" \"actual_selection_success\": 0.5,\n",
" \"deployment_success\": 0.5,\n",
" \"oracle_task_flags\": [\n",
" true,\n",
" true\n",
" ],\n",
" \"selection_task_flags\": [\n",
" true,\n",
" false\n",
" ],\n",
" \"deployment_task_flags\": [\n",
" true,\n",
" false\n",
" ]\n",
"}\n",
"\n",
"Interpretation:\n",
"Oracle coverage asks whether some candidate solves each task. Actual selection and deployment filtering can only reduce that finite-bank ceiling.\n",
"\n",
"Assumptions:\n",
"- Bank contents and task distribution are fixed.\n",
"- Selection is supplied per task; oracle selection is not assumed achievable.\n",
"- Independent saturation is a separate constructed sampling law.\n",
"\n",
"Limitations:\n",
"- The bank ceiling is not a universal agent limit or a forecast of future models.\n",
"- Unobserved tasks and candidate classes remain outside the bound.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"reader_file = LAB_ROOT / 'data/examples/ch26.json'\n",
"reader_inputs = json.loads(reader_file.read_text())\n",
"reader_report = analyze(chapter, reader_inputs)\n",
"print(report_text(reader_report))"
]
},
{
"cell_type": "markdown",
"id": "13f42c685347",
"metadata": {},
"source": [
"## Questions\n",
"\n",
"1. What are default oracle,actual,deployed values?\n",
"\n",
"2. Why is changed deployment 0.5 despite perfect selection?\n",
"\n",
"3. Can adding a candidate reduce oracle prefix coverage?\n",
"\n",
"Answers: [separate solutions](../solutions/ch26.md). Try the calculation before opening them."
]
},
{
"cell_type": "markdown",
"id": "6aea7b7a35fb",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"Available capability, selected capability, and deployed capability are different quantities. This notebook verifies their ordering and exposes selector and authority gaps under a fixed task distribution. The changed case improves selection while limiting deployment; the transfer checks overlap. Prefix coverage is finite-bank evidence, while independent saturation is a separate construction. Any ceiling remains conditional on bank, distribution, interface, and permissions, without forecasting universal future capability.\n",
"\n",
"Limits of this experiment:\n",
"\n",
"- This is a local calculation under declared inputs, not an empirical claim about a deployed agent.\n",
"- Read the returned assumptions and limitations before applying the numerical result.\n",
"\n",
"The assistant skill is [`maa-26-conditional-bank-ceiling`](../skills/maa-26-conditional-bank-ceiling/SKILL.md). It uses this notebook's tested computation and input contract."
]
},
{
"cell_type": "markdown",
"id": "9cc9f25ba898",
"metadata": {},
"source": [
"## Equations from the chapter\n",
"\n",
"These are the unchanged display equations and their explanations from the canonical chapter. They are a reference for the experiment, not a claim that every equation is numerically implemented by this one method."
]
},
{
"cell_type": "markdown",
"id": "87b235336433",
"metadata": {},
"source": [
"### Equation 26.1\n",
"\n",
"\n",
"\n",
"Averages each task's finite-pool chance that a size-`k` subset contains a unit-test-passing candidate.\n",
"\n",
"The shared `k` stays within every `n_i`, and each term removes the all-failing-subset probability.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\widehat{\\operatorname{Cov}}_{\\mathcal K}(k)\n",
"= \\frac{1}{|\\mathcal K|}\\sum_{i\\in\\mathcal K}\n",
"\\left(1 - \\frac{\\binom{n_i-c_i}{k}}{\\binom{n_i}{k}}\\right),\n",
"\\qquad 1\\le k\\le \\min_{i\\in\\mathcal K}n_i.\n",
"\\tag{26.1}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "6c3bdd79df72",
"metadata": {},
"source": [
"### Equation 26.2\n",
"\n",
"\n",
"\n",
"States that a selector cannot return a correct candidate more often than correct candidates appear in its bank.\n",
"\n",
"Selection can lose available successes; it cannot create a correct candidate absent from generation.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{Sel}(k) \\le \\operatorname{Cov}(k).\n",
"\\tag{26.2}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "3e5251c67f8d",
"metadata": {},
"source": [
"### Equation 26.3\n",
"\n",
"\n",
"\n",
"Names available success left undispatched by the actual selector under one matched candidate bank and evaluator.\n",
"\n",
"A large gap points toward selection or evidence; a small gap points back toward generation or budget.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{Cov}(k)-\\operatorname{Sel}(k) \\ge 0.\n",
"\\tag{26.3}\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Mathematics of AI Agents",
"language": "python",
"name": "maa-lab"
},
"lab_chapter": 26,
"lab_execution": {
"code_cells": 6,
"created_utc": "2026-10-02T05:16:50.935890+00:00",
"elapsed_seconds": 8.851196041912772,
"method": "fresh process; new ipykernel InProcessKernelManager; cells submitted as Jupyter execute requests",
"network_transport_tested": false,
"python": "3.11.15",
"source_sha256": "2fceec3a4d38cbe547b4220f349e580d62a8e48d3ec195d86607f198f465b07f"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}