{
"cells": [
{
"cell_type": "markdown",
"id": "6676ceaa4fc9",
"metadata": {},
"source": [
"# Chapter 19: When Another Mind Becomes Part of the World\n",
"\n",
"Two controllers coordinate well with copies of themselves and badly with each other. Their private conventions are useful locally but do not travel. A deployment that changes partners can therefore change the effective environment without changing either controller's underlying model.\n",
"\n",
"This notebook makes the distinction visible in a cross-play matrix. Rows are candidate policies and columns are partners. Diagonal cells measure matching-policy coordination, while off-diagonal cells reveal transfer. Weighted row values answer a deployment question: which policy works best against the partners expected here? A changed supervisor distribution can reverse the preferred policy, so diagonal success alone is an incomplete selection rule.\n",
"\n",
"**Outcome:** Compute diagonal, cross-play, and deployment-mixture performance.\n",
"\n",
"- Inspect the declared input contract\n",
"- Predict the hand-checkable case\n",
"- Run the shared computation\n",
"- Change the critical assumption\n",
"- Apply the method to the transfer data\n",
"\n",
"**Guided route:** Run the worked calculation, inspect its figure, change the stated assumption, and try the transfer case. Read the explanations beside each result before opening the answers.\n",
"\n",
"**Deeper route:** First read the mathematics and canonical equation reference. Audit the input contract, predict the changed result, then inspect the shared chapter implementation and solve the questions independently. Both routes use the same calculations and preserve the equations."
]
},
{
"cell_type": "markdown",
"id": "a7781f3ee7dc",
"metadata": {},
"source": [
"## Technical Requirements\n",
"\n",
"Python 3.11 or later, the complete laboratory folder, and the notebook dependencies listed in `requirements-notebooks.txt` (the launcher's **Install notebook tools** choice installs them; see START-HERE). Standard-library chapter commands also support Python 3.10. No API key, model account or network call is used by this experiment.\n",
"\n",
"Prior knowledge:\n",
"\n",
"- Python lists and dictionaries\n",
"- The mapped chapter and its notation"
]
},
{
"cell_type": "markdown",
"id": "db11a1590622",
"metadata": {},
"source": [
"## The question and its mathematics\n",
"\n",
"For matrix M_ij, mean self-play is the average diagonal value. Mean cross-play averages off-diagonal values when at least two policies exist. These descriptive summaries weight their cells uniformly; they are not automatically the deployment estimand.\n",
"\n",
"With target partner weights w_j summing to one, policy i has value V_i=sum_j w_j M_ij. A second weight vector computes another environment mixture over the same columns. Selecting argmax_i V_i conditions the recommendation on that distribution. Changing weights can reverse the winner even when every matrix entry is fixed.\n",
"\n",
"The supplied entries are probabilities or compatible normalized performance values under a shared success definition. The runtime does not estimate them from counts. Matching task distributions, budgets, and measurement rules is an assumption of the comparison. If partners adapt to the selected policy, a frozen matrix may cease to describe their behavior; the report should then become a dynamic-game measurement design rather than a static guarantee."
]
},
{
"cell_type": "markdown",
"id": "4e6de78f2909",
"metadata": {},
"source": [
"## A calculation you can run\n",
"\n",
"Provide a square matrix and two normalized column-weight vectors. The function validates probabilities and dimensions, computes diagonal and off-diagonal summaries, then evaluates every row under both mixtures. It returns selected policy indices and their expected mixture values.\n",
"\n",
"The figures show self-play diagonal values, partner-transfer values, and changed-supervisor values separately. Policy index identifies each row; retain a name mapping in reader notes when policies have operational names. In the changed case the primary partner mixture becomes concentrated on column 0. Predict whether that favors the first policy's strong diagonal or the second policy's more even transfer. No matrix observations change.\n",
"\n",
"The next cell finds the bundle and imports the same computation used by the chapter skill. It does not change your system Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "42ebefc0e5c1",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys, json\n",
"LAB_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"lab-manifest.json\").is_file()), None)\n",
"if LAB_ROOT is None:\n",
" raise RuntimeError(\"Open this notebook from the complete extracted laboratory folder.\")\n",
"sys.path.insert(0, str(LAB_ROOT / \"src\"))\n",
"from math_ai_agents.core import analyze, report_text\n",
"from math_ai_agents.plotting import figure_svg\n",
"from IPython.display import SVG, display\n"
]
},
{
"cell_type": "markdown",
"id": "7fffde8c0edb",
"metadata": {},
"source": [
"Set the declared inputs below. These are constructed teaching values, not measurements from a production agent. Change a value only after predicting what it should change."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "13eebe0a4642",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 19: cross-play-transfer\n",
"Does successful self-play transfer to the partners the controller will actually meet?\n",
"Evidence: constructed teaching example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"mean_self_play\": 0.925,\n",
" \"mean_cross_play\": 0.3,\n",
" \"joint_policy_correlation_loss\": 0.6756756757,\n",
" \"partner_mixture_values\": [\n",
" 0.575,\n",
" 0.65\n",
" ],\n",
" \"supervisor_mixture_values\": [\n",
" 0.875,\n",
" 0.45\n",
" ],\n",
" \"selected_policy\": 1,\n",
" \"supervisor_selected_policy\": 0\n",
"}\n",
"\n",
"Interpretation:\n",
"joint_policy_correlation_loss is Equation 19.2: (mean diagonal minus mean off-diagonal) divided by mean diagonal, computed here from the supplied matrix. Diagonal success measures coordination with matching partners. Off-diagonal entries and target weights determine transfer to other partners.\n",
"\n",
"Assumptions:\n",
"- Matrix cells use matched tasks, budgets, and success definitions.\n",
"- Partner and supervisor weights are declared deployment mixtures.\n",
"\n",
"Limitations:\n",
"- Changing counterparties can change the environment itself.\n",
"- A supplied matrix does not establish stationarity or sampling precision.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"chapter = 19\n",
"inputs = {'matrix': [[0.95, 0.2], [0.4, 0.9]],\n",
" 'partner_weights': [0.5, 0.5],\n",
" 'supervisor_weights': [0.9, 0.1]}\n",
"report = analyze(chapter, inputs)\n",
"# This input was explicitly taken from the teaching fixture.\n",
"report['evidence_kind'] = 'constructed teaching example'\n",
"print(report_text(report))"
]
},
{
"cell_type": "markdown",
"id": "aac714fc30a8",
"metadata": {},
"source": [
"Default mean self-play is (0.95+0.9)/2=0.925. Mean cross-play is (0.2+0.4)/2=0.3, so the proportional loss of Equation 19.2 is (0.925-0.3)/0.925, about 0.676, which the report returns as joint_policy_correlation_loss. Under equal partner weights, policy 0 scores 0.575 and policy 1 scores 0.65, so policy 1 wins despite policy 0's stronger diagonal.\n",
"\n",
"With weights [0.9,0.1], policy 0 scores 0.875 and policy 1 scores 0.45. The preferred policy reverses. The second supervisor mixture already demonstrates that reversal in the default report. Interpret it as conditional transfer across partner distributions, not proof that one policy is intrinsically more cooperative.\n",
"\n",
"The plot below uses the calculated quantities. Read each panel's units before comparing its values."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "590f57e74048",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Matplotlib is building the font cache; this may take a moment.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(SVG(figure_svg(report)))"
]
},
{
"cell_type": "markdown",
"id": "101cdde92631",
"metadata": {},
"source": [
"**Figure 19.L1:** Calculated chapter experiment. Each panel labels its input and output units; interpret it under the assumptions printed in the report."
]
},
{
"cell_type": "markdown",
"id": "1863c93da2da",
"metadata": {},
"source": [
"## Change the assumption\n",
"\n",
"A cross-play matrix can conceal unmatched conditions. If one row receives easier tasks, more time, or a different supervisor, weighted averages compare more than policy behavior. Preserve those conditions before treating a cell difference as transfer evidence.\n",
"\n",
"Uniform off-diagonal averaging is also easy to misuse. The deployed system may almost never encounter some partners, while frequently meeting an omitted partner. A high average across the supplied bank says little about an unsupported population.\n",
"\n",
"Another failure is strategic adaptation. A partner that reacts to the deployed policy creates a changing environment. Static matrix values may describe the initial interaction but not later equilibria. This laboratory calculates a declared finite comparison and does not infer common knowledge, incentives, or an equilibrium from success entries."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a6bdf9c068ae",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 19: cross-play-transfer\n",
"Does successful self-play transfer to the partners the controller will actually meet?\n",
"Evidence: constructed changed-assumption example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"mean_self_play\": 0.925,\n",
" \"mean_cross_play\": 0.3,\n",
" \"joint_policy_correlation_loss\": 0.6756756757,\n",
" \"partner_mixture_values\": [\n",
" 0.875,\n",
" 0.45\n",
" ],\n",
" \"supervisor_mixture_values\": [\n",
" 0.875,\n",
" 0.45\n",
" ],\n",
" \"selected_policy\": 0,\n",
" \"supervisor_selected_policy\": 0\n",
"}\n",
"\n",
"Interpretation:\n",
"joint_policy_correlation_loss is Equation 19.2: (mean diagonal minus mean off-diagonal) divided by mean diagonal, computed here from the supplied matrix. Diagonal success measures coordination with matching partners. Off-diagonal entries and target weights determine transfer to other partners.\n",
"\n",
"Assumptions:\n",
"- Matrix cells use matched tasks, budgets, and success definitions.\n",
"- Partner and supervisor weights are declared deployment mixtures.\n",
"\n",
"Limitations:\n",
"- Changing counterparties can change the environment itself.\n",
"- A supplied matrix does not establish stationarity or sampling precision.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"changed_inputs = {'matrix': [[0.95, 0.2], [0.4, 0.9]],\n",
" 'partner_weights': [0.9, 0.1],\n",
" 'supervisor_weights': [0.9, 0.1]}\n",
"changed = analyze(chapter, changed_inputs)\n",
"changed['evidence_kind'] = 'constructed changed-assumption example'\n",
"print(report_text(changed))\n",
"display(SVG(figure_svg(changed)))"
]
},
{
"cell_type": "markdown",
"id": "51663185ca4d",
"metadata": {},
"source": [
"**Figure 19.L2:** The changed-assumption result. Compare the printed quantities and the stated assumptions with the first run. A different input need not imply a causal effect in a deployed agent."
]
},
{
"cell_type": "markdown",
"id": "5b31390b7e36",
"metadata": {},
"source": [
"## Try a new case\n",
"\n",
"The transfer matrix includes a specialist for column 0, a generalist with success 0.6 everywhere, and a specialist for column 2. Under weights [0.2,0.6,0.2], the generalist wins at 0.6 while each specialist scores 0.2. Concentrating the supervisor on the two extreme columns gives the specialists 0.5 each, still below 0.6.\n",
"\n",
"For local data, collect off-diagonal cells deliberately rather than testing only familiar pairings. Record counts and uncertainty separately if entries are estimates. Define the partner mixture relevant to deployment and inspect plausible shifts before choosing a policy. Self-play remains a useful diagnostic, but it answers a narrower question."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "b10a0ce64a12",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 19: cross-play-transfer\n",
"Does successful self-play transfer to the partners the controller will actually meet?\n",
"Evidence: constructed transfer example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"mean_self_play\": 0.8666666667,\n",
" \"mean_cross_play\": 0.2,\n",
" \"joint_policy_correlation_loss\": 0.7692307692,\n",
" \"partner_mixture_values\": [\n",
" 0.2,\n",
" 0.6,\n",
" 0.2\n",
" ],\n",
" \"supervisor_mixture_values\": [\n",
" 0.5,\n",
" 0.6,\n",
" 0.5\n",
" ],\n",
" \"selected_policy\": 1,\n",
" \"supervisor_selected_policy\": 1\n",
"}\n",
"\n",
"Interpretation:\n",
"joint_policy_correlation_loss is Equation 19.2: (mean diagonal minus mean off-diagonal) divided by mean diagonal, computed here from the supplied matrix. Diagonal success measures coordination with matching partners. Off-diagonal entries and target weights determine transfer to other partners.\n",
"\n",
"Assumptions:\n",
"- Matrix cells use matched tasks, budgets, and success definitions.\n",
"- Partner and supervisor weights are declared deployment mixtures.\n",
"\n",
"Limitations:\n",
"- Changing counterparties can change the environment itself.\n",
"- A supplied matrix does not establish stationarity or sampling precision.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"transfer_inputs = {'matrix': [[1, 0, 0], [0.6, 0.6, 0.6], [0, 0, 1]],\n",
" 'partner_weights': [0.2, 0.6, 0.2],\n",
" 'supervisor_weights': [0.5, 0, 0.5]}\n",
"transfer = analyze(chapter, transfer_inputs)\n",
"transfer['evidence_kind'] = 'constructed transfer example'\n",
"print(report_text(transfer))"
]
},
{
"cell_type": "markdown",
"id": "a098605b03db",
"metadata": {},
"source": [
"## Apply the method to your inputs\n",
"\n",
"The example file below has the exact input shape the method accepts. Copy it to a new file, replace its values, then point `reader_file` at your copy. Run the cell again. Supplied inputs retain their stated provenance; the program cannot establish that they are representative observations.\n",
"\n",
"- **matrix:** Square policy by partner success-probability matrix, entries in [0,1].\n",
"- **partner_weights:** Normalized target partner distribution.\n",
"- **supervisor_weights:** Normalized changed partner or supervisor distribution over the same columns."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "9e70ee026cb5",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 19: cross-play-transfer\n",
"Does successful self-play transfer to the partners the controller will actually meet?\n",
"Evidence: supplied local inputs; provenance not independently verified\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"mean_self_play\": 0.8666666667,\n",
" \"mean_cross_play\": 0.2,\n",
" \"joint_policy_correlation_loss\": 0.7692307692,\n",
" \"partner_mixture_values\": [\n",
" 0.2,\n",
" 0.6,\n",
" 0.2\n",
" ],\n",
" \"supervisor_mixture_values\": [\n",
" 0.5,\n",
" 0.6,\n",
" 0.5\n",
" ],\n",
" \"selected_policy\": 1,\n",
" \"supervisor_selected_policy\": 1\n",
"}\n",
"\n",
"Interpretation:\n",
"joint_policy_correlation_loss is Equation 19.2: (mean diagonal minus mean off-diagonal) divided by mean diagonal, computed here from the supplied matrix. Diagonal success measures coordination with matching partners. Off-diagonal entries and target weights determine transfer to other partners.\n",
"\n",
"Assumptions:\n",
"- Matrix cells use matched tasks, budgets, and success definitions.\n",
"- Partner and supervisor weights are declared deployment mixtures.\n",
"\n",
"Limitations:\n",
"- Changing counterparties can change the environment itself.\n",
"- A supplied matrix does not establish stationarity or sampling precision.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"reader_file = LAB_ROOT / 'data/examples/ch19.json'\n",
"reader_inputs = json.loads(reader_file.read_text())\n",
"reader_report = analyze(chapter, reader_inputs)\n",
"print(report_text(reader_report))"
]
},
{
"cell_type": "markdown",
"id": "0b3fa46fcacc",
"metadata": {},
"source": [
"## Questions\n",
"\n",
"1. Compute default policy 1 mixture value.\n",
"\n",
"2. Why does the changed mixture favor policy 0?\n",
"\n",
"3. Which transfer policy wins?\n",
"\n",
"Answers: [separate solutions](../solutions/ch19.md). Try the calculation before opening them."
]
},
{
"cell_type": "markdown",
"id": "2549c6f18f6a",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"Self-play measures coordination with a matching partner; cross-play tests transfer. Deployment choice depends on partner weights, and changed counterparties can reverse rankings without changing policies. The notebook separates these quantities and checks a specialist/generalist transfer case. Its matrix is a finite declared environment, not a universal cooperation score or dynamic equilibrium model. Preserve matched conditions and the intended partner population beside the result.\n",
"\n",
"Limits of this experiment:\n",
"\n",
"- This is a local calculation under declared inputs, not an empirical claim about a deployed agent.\n",
"- Read the returned assumptions and limitations before applying the numerical result.\n",
"\n",
"The assistant skill is [`maa-19-cross-play-transfer`](../skills/maa-19-cross-play-transfer/SKILL.md). It uses this notebook's tested computation and input contract."
]
},
{
"cell_type": "markdown",
"id": "9cc9f25ba898",
"metadata": {},
"source": [
"## Equations from the chapter\n",
"\n",
"These are the unchanged display equations and their explanations from the canonical chapter. They are a reference for the experiment, not a claim that every equation is numerically implemented by this one method."
]
},
{
"cell_type": "markdown",
"id": "716b8fa98185",
"metadata": {},
"source": [
"### Equation 19.1\n",
"\n",
"\n",
"\n",
"Equation (19.1) records what happens when a policy from one training run meets a policy from another.\n",
"\n",
"Take party one's policy from run `s`, party two's from run `t`, play them together, and average the return.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"M_{st} \\;=\\; \\mathrm{E}\\Big[\\,u\\big(\\pi_1^{(s)},\\,\\pi_2^{(t)}\\big)\\Big].\n",
"\\tag{19.1}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "c3e64eaecbc4",
"metadata": {},
"source": [
"### Equation 19.2\n",
"\n",
"\n",
"\n",
"Equation (19.2) normalizes the measured matched-partner gap under the declared procedure.\n",
"\n",
"Subtract the average score against strangers from the average score against training partners, and divide by the latter.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{JPC}(M) \\;=\\; \\frac{\\operatorname{diag}(M) - \\operatorname{off}(M)}{\\operatorname{diag}(M)}.\n",
"\\tag{19.2}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "505f36302f22",
"metadata": {},
"source": [
"### Equation 19.3\n",
"\n",
"\n",
"\n",
"Equation (19.3) exposes a policy dependence that must be checked before applying a stationary-system guarantee.\n",
"\n",
"The distribution over next states now depends on the other party's policy as well as on the state and the action.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"P\\big(x' \\mid x, a\\big) \\;\\longrightarrow\\; P\\big(x' \\mid x, a,\\, \\pi_{-i}\\big).\n",
"\\tag{19.3}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "f1e5ff692623",
"metadata": {},
"source": [
"### Equation 19.4\n",
"\n",
"\n",
"\n",
"Equation (19.4) gives the best an agent can do against a stated opponent policy.\n",
"\n",
"Choose the policy maximizing this party's payoff, holding the other party's policy fixed.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{BR}_i(\\pi_{-i}) \\;=\\; \\arg\\max_{\\pi_i}\\; u_i\\big(\\pi_i, \\pi_{-i}\\big).\n",
"\\tag{19.4}\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Mathematics of AI Agents",
"language": "python",
"name": "maa-lab"
},
"lab_chapter": 19,
"lab_execution": {
"code_cells": 6,
"created_utc": "2026-10-02T05:15:46.913995+00:00",
"elapsed_seconds": 8.98502224998083,
"method": "fresh process; new ipykernel InProcessKernelManager; cells submitted as Jupyter execute requests",
"network_transport_tested": false,
"python": "3.11.15",
"source_sha256": "bb744341f130230fc5ae881448480b083848ca714da8eb3e8195aaa368ceffb5"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}