{
"cells": [
{
"cell_type": "markdown",
"id": "6c1b841945ac",
"metadata": {},
"source": [
"# Chapter 3: How to Predict Emergence\n",
"\n",
"A curve can explain yesterday's observations beautifully and still fail at tomorrow's scale. The difference between explanation and forecasting lies partly in chronology: which observations were available when the family and coefficients were selected? If a test outcome helps choose the model, it is no longer independent evidence for the prediction it influenced.\n",
"\n",
"This notebook gives development and test observations separate fields. The coefficients are fitted from development data alone. The test data then scores that frozen forecast. The changed case leaves the fit untouched while changing unseen outcomes, making it impossible to confuse a stable fitted line with a successful prediction. This is a narrow implemented forecasting protocol, not a universal emergence predictor.\n",
"\n",
"**Outcome:** Fit a declared curve on development data and score a separate test.\n",
"\n",
"- Inspect the declared input contract\n",
"- Predict the hand-checkable case\n",
"- Run the shared computation\n",
"- Change the critical assumption\n",
"- Apply the method to the transfer data\n",
"\n",
"**Guided route:** Run the worked calculation, inspect its figure, change the stated assumption, and try the transfer case. Read the explanations beside each result before opening the answers.\n",
"\n",
"**Deeper route:** First read the mathematics and canonical equation reference. Audit the input contract, predict the changed result, then inspect the shared chapter implementation and solve the questions independently. Both routes use the same calculations and preserve the equations."
]
},
{
"cell_type": "markdown",
"id": "a7781f3ee7dc",
"metadata": {},
"source": [
"## Technical Requirements\n",
"\n",
"Python 3.11 or later, the complete laboratory folder, and the notebook dependencies listed in `requirements-notebooks.txt` (the launcher's **Install notebook tools** choice installs them; see START-HERE). Standard-library chapter commands also support Python 3.10. No API key, model account or network call is used by this experiment.\n",
"\n",
"Prior knowledge:\n",
"\n",
"- Python lists and dictionaries\n",
"- The mapped chapter and its notation"
]
},
{
"cell_type": "markdown",
"id": "639828127f88",
"metadata": {},
"source": [
"## The question and its mathematics\n",
"\n",
"The declared family is y=alpha+beta f(x), where f(x)=x for a linear fit or f(x)=log(x) for a log fit. Let z_i=f(x_i). Least squares gives beta=sum_i(z_i-z_bar)(y_i-y_bar)/sum_i(z_i-z_bar)^2 and alpha=y_bar-beta z_bar. Variation in the development scales is necessary because a zero denominator leaves the slope unidentified.\n",
"\n",
"At a test scale x_j, the frozen prediction is y_hat_j=alpha+beta f(x_j). Residuals are y_hat_j-y_j. Test RMSE is sqrt(mean residual squared), while bias is the signed mean residual. RMSE measures magnitude; bias distinguishes systematic overprediction from underprediction. Neither is a probability of being correct.\n",
"\n",
"The log family requires positive scales and interprets equal multiplicative scale changes as equal increments in its transformed coordinate. Family selection is part of the prospective protocol and must occur before seeing test outcomes. The software cannot verify the historical order of an external experiment, but it rejects overlapping development and test scales and never uses test y values in the fit."
]
},
{
"cell_type": "markdown",
"id": "f3557405796b",
"metadata": {},
"source": [
"## A calculation you can run\n",
"\n",
"Supply two aligned development vectors and two aligned test vectors. State the family explicitly. The function validates finite values, dimensional alignment, development variation, and separation of scale sets. It computes coefficients, frozen predictions, residuals, RMSE, and bias.\n",
"\n",
"The forecast and unseen-observation figures share test scale and score units. The residual figure makes errors visible rather than hiding them beneath a fitted line. Run the default case, save its coefficients, and predict what will happen when only test_y changes. The coefficients must remain identical. If they move, test evidence leaked into fitting. For real data, keep a timestamped forecast artifact or preregistered protocol beside the exported result to support the prospective claim.\n",
"\n",
"The next cell finds the bundle and imports the same computation used by the chapter skill. It does not change your system Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "42ebefc0e5c1",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys, json\n",
"LAB_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"lab-manifest.json\").is_file()), None)\n",
"if LAB_ROOT is None:\n",
" raise RuntimeError(\"Open this notebook from the complete extracted laboratory folder.\")\n",
"sys.path.insert(0, str(LAB_ROOT / \"src\"))\n",
"from math_ai_agents.core import analyze, report_text\n",
"from math_ai_agents.plotting import figure_svg\n",
"from IPython.display import SVG, display\n"
]
},
{
"cell_type": "markdown",
"id": "7fffde8c0edb",
"metadata": {},
"source": [
"Set the declared inputs below. These are constructed teaching values, not measurements from a production agent. Change a value only after predicting what it should change."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "0f41e36ebac7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 3: prospective-forecast\n",
"Can a frozen forecast predict unseen capability observations?\n",
"Evidence: constructed teaching example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"family\": \"linear\",\n",
" \"slope\": 0.1,\n",
" \"intercept\": 0.1,\n",
" \"predictions\": [\n",
" 0.5,\n",
" 0.6\n",
" ],\n",
" \"test_rmse\": 0.0,\n",
" \"test_bias\": 0.0\n",
"}\n",
"\n",
"Interpretation:\n",
"Only development observations fit the declared family. Test outcomes score the frozen forecast and never update its coefficients.\n",
"\n",
"Assumptions:\n",
"- Family selection is declared before the test is inspected.\n",
"- Least squares describes the supplied development observations.\n",
"\n",
"Limitations:\n",
"- Extrapolation validity and mechanism are not established by a good fit.\n",
"- Protocol metadata is needed to establish real prospective separation.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"chapter = 3\n",
"inputs = {'development_x': [1, 2, 3],\n",
" 'development_y': [0.2, 0.3, 0.4],\n",
" 'test_x': [4, 5],\n",
" 'test_y': [0.5, 0.6],\n",
" 'family': 'linear'}\n",
"report = analyze(chapter, inputs)\n",
"# This input was explicitly taken from the teaching fixture.\n",
"report['evidence_kind'] = 'constructed teaching example'\n",
"print(report_text(report))"
]
},
{
"cell_type": "markdown",
"id": "815ce3953def",
"metadata": {},
"source": [
"The default development line has slope 0.1 and intercept 0.1. It predicts 0.5 and 0.6 at unseen scales 4 and 5, matching the supplied test outcomes. RMSE and bias are zero up to numerical rounding.\n",
"\n",
"The changed test outcomes are 0.8 and 0.95. The forecast remains 0.5 and 0.6 because the development data did not change. Residuals become -0.3 and -0.35, with negative bias and a positive RMSE. That failure is informative. Refitting after seeing it might improve the retrospective graph, but it would not repair the original prospective forecast.\n",
"\n",
"The plot below uses the calculated quantities. Read each panel's units before comparing its values."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "590f57e74048",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Matplotlib is building the font cache; this may take a moment.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(SVG(figure_svg(report)))"
]
},
{
"cell_type": "markdown",
"id": "0adc732fdff9",
"metadata": {},
"source": [
"**Figure 3.L1:** Calculated chapter experiment. Each panel labels its input and output units; interpret it under the assumptions printed in the report."
]
},
{
"cell_type": "markdown",
"id": "644ce3385e21",
"metadata": {},
"source": [
"## Change the assumption\n",
"\n",
"The simplest leakage pattern is to move a disappointing test observation into the development set and continue calling the remaining assessment unseen. Another is to inspect several families against the same test outcomes and report only the best one. Both use the test to shape the prediction, so the reported errors cease to measure the original prospective protocol.\n",
"\n",
"The implementation checks explicit scale overlap, but it cannot detect hidden reuse, shared tasks, or a family chosen after reading test results. Those are evidence-management problems. Record the family, data boundary, fit command, and forecast before opening the withheld outcome file.\n",
"\n",
"A successful linear or log extrapolation also does not reveal an emergence mechanism. It is conditional performance of one declared family over one supplied range. New environments or measurement rules can break that relationship even if the development residuals were tiny."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "bf2f5b422c33",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 3: prospective-forecast\n",
"Can a frozen forecast predict unseen capability observations?\n",
"Evidence: constructed changed-assumption example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"family\": \"linear\",\n",
" \"slope\": 0.1,\n",
" \"intercept\": 0.1,\n",
" \"predictions\": [\n",
" 0.5,\n",
" 0.6\n",
" ],\n",
" \"test_rmse\": 0.3259601203,\n",
" \"test_bias\": -0.325\n",
"}\n",
"\n",
"Interpretation:\n",
"Only development observations fit the declared family. Test outcomes score the frozen forecast and never update its coefficients.\n",
"\n",
"Assumptions:\n",
"- Family selection is declared before the test is inspected.\n",
"- Least squares describes the supplied development observations.\n",
"\n",
"Limitations:\n",
"- Extrapolation validity and mechanism are not established by a good fit.\n",
"- Protocol metadata is needed to establish real prospective separation.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"changed_inputs = {'development_x': [1, 2, 3],\n",
" 'development_y': [0.2, 0.3, 0.4],\n",
" 'test_x': [4, 5],\n",
" 'test_y': [0.8, 0.95],\n",
" 'family': 'linear'}\n",
"changed = analyze(chapter, changed_inputs)\n",
"changed['evidence_kind'] = 'constructed changed-assumption example'\n",
"print(report_text(changed))\n",
"display(SVG(figure_svg(changed)))"
]
},
{
"cell_type": "markdown",
"id": "6a30d83d1cd8",
"metadata": {},
"source": [
"**Figure 3.L2:** The changed-assumption result. Compare the printed quantities and the stated assumptions with the first run. A different input need not imply a causal effect in a deployed agent."
]
},
{
"cell_type": "markdown",
"id": "1fadc04869c0",
"metadata": {},
"source": [
"## Try a new case\n",
"\n",
"The transfer data follows a logarithmic family. Doubling scale adds approximately log(2) to its outcome. Fit in log coordinates, then predict at scales 8 and 16. The slope is approximately 1 and the intercept approximately zero, so the predictions are approximately 2.0794 and 2.7726.\n",
"\n",
"When applying the method locally, define the forecast target before choosing the family. It might be a continuous score, a completion probability, or another declared quantity. The runtime allows finite targets without pretending all are probabilities. Explain the target units and score transformation in the accompanying report. If a bounded probability is required, this unconstrained regression may be the wrong family."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "f7068a2bc071",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 3: prospective-forecast\n",
"Can a frozen forecast predict unseen capability observations?\n",
"Evidence: constructed transfer example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"family\": \"log\",\n",
" \"slope\": 1.0,\n",
" \"intercept\": 0.0,\n",
" \"predictions\": [\n",
" 2.079441542,\n",
" 2.772588722\n",
" ],\n",
" \"test_rmse\": 0.0,\n",
" \"test_bias\": 0.0\n",
"}\n",
"\n",
"Interpretation:\n",
"Only development observations fit the declared family. Test outcomes score the frozen forecast and never update its coefficients.\n",
"\n",
"Assumptions:\n",
"- Family selection is declared before the test is inspected.\n",
"- Least squares describes the supplied development observations.\n",
"\n",
"Limitations:\n",
"- Extrapolation validity and mechanism are not established by a good fit.\n",
"- Protocol metadata is needed to establish real prospective separation.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"transfer_inputs = {'development_x': [1, 2, 4],\n",
" 'development_y': [0, 0.69314718056, 1.38629436112],\n",
" 'test_x': [8, 16],\n",
" 'test_y': [2.07944154168, 2.77258872224],\n",
" 'family': 'log'}\n",
"transfer = analyze(chapter, transfer_inputs)\n",
"transfer['evidence_kind'] = 'constructed transfer example'\n",
"print(report_text(transfer))"
]
},
{
"cell_type": "markdown",
"id": "896f04cca377",
"metadata": {},
"source": [
"## Apply the method to your inputs\n",
"\n",
"The example file below has the exact input shape the method accepts. Copy it to a new file, replace its values, then point `reader_file` at your copy. Run the cell again. Supplied inputs retain their stated provenance; the program cannot establish that they are representative observations.\n",
"\n",
"- **development_x:** At least two development scales with variation.\n",
"- **development_y:** Aligned finite development observations.\n",
"- **test_x:** Separate unseen scales, with no development overlap.\n",
"- **test_y:** Aligned withheld observations used only for scoring.\n",
"- **family:** linear or log, declared before inspecting test outcomes."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "a4f5029afee1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 3: prospective-forecast\n",
"Can a frozen forecast predict unseen capability observations?\n",
"Evidence: supplied local inputs; provenance not independently verified\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"family\": \"log\",\n",
" \"slope\": 1.0,\n",
" \"intercept\": 0.0,\n",
" \"predictions\": [\n",
" 2.079441542,\n",
" 2.772588722\n",
" ],\n",
" \"test_rmse\": 0.0,\n",
" \"test_bias\": 0.0\n",
"}\n",
"\n",
"Interpretation:\n",
"Only development observations fit the declared family. Test outcomes score the frozen forecast and never update its coefficients.\n",
"\n",
"Assumptions:\n",
"- Family selection is declared before the test is inspected.\n",
"- Least squares describes the supplied development observations.\n",
"\n",
"Limitations:\n",
"- Extrapolation validity and mechanism are not established by a good fit.\n",
"- Protocol metadata is needed to establish real prospective separation.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"reader_file = LAB_ROOT / 'data/examples/ch03.json'\n",
"reader_inputs = json.loads(reader_file.read_text())\n",
"reader_report = analyze(chapter, reader_inputs)\n",
"print(report_text(reader_report))"
]
},
{
"cell_type": "markdown",
"id": "8d2c4a4b1bc7",
"metadata": {},
"source": [
"## Questions\n",
"\n",
"1. What are default fit coefficients?\n",
"\n",
"2. Does the changed test update the fit?\n",
"\n",
"3. Why must log-family scales be positive?\n",
"\n",
"Answers: [separate solutions](../solutions/ch03.md). Try the calculation before opening them."
]
},
{
"cell_type": "markdown",
"id": "4d523bab3ed7",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"A forecast is prospective only when its choices are frozen before the outcomes used to evaluate it. This computation separates development fitting from test scoring and reports both magnitude and direction of error. The changed case preserves coefficients while exposing forecast failure; the transfer case changes the declared family. Good fit does not establish mechanism, universality, or calibration. Preserve the chronology and input boundary so the numerical result can be interpreted as evidence for the protocol that actually ran.\n",
"\n",
"Limits of this experiment:\n",
"\n",
"- This is a local calculation under declared inputs, not an empirical claim about a deployed agent.\n",
"- Read the returned assumptions and limitations before applying the numerical result.\n",
"\n",
"The assistant skill is [`maa-03-prospective-forecast`](../skills/maa-03-prospective-forecast/SKILL.md). It uses this notebook's tested computation and input contract."
]
},
{
"cell_type": "markdown",
"id": "9cc9f25ba898",
"metadata": {},
"source": [
"## Equations from the chapter\n",
"\n",
"These are the unchanged display equations and their explanations from the canonical chapter. They are a reference for the experiment, not a claim that every equation is numerically implemented by this one method."
]
},
{
"cell_type": "markdown",
"id": "631ab8b2fda7",
"metadata": {},
"source": [
"### Equation 3.1\n",
"\n",
"\n",
"\n",
"Equation (3.1) is a fitted curve that lets a handful of completed training runs predict the loss of a much larger run that has not finished.\n",
"\n",
"`C` is training compute and `a`, `b`, `c` are constants chosen to fit the smaller runs, so a larger `C` reads off a prediction.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"L(C)=aC^b+c.\n",
"\\tag{3.1}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "9838278203c9",
"metadata": {},
"source": [
"### Equation 3.2\n",
"\n",
"\n",
"\n",
"Equation (3.2) proposes that a member's measured interaction is explained by declared features plus leftover error, which is a modeling choice rather than an existing result.\n",
"\n",
"The measured interaction for member `m` equals a fitted function of the declared features `z_m`, plus the part that fit misses.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\Gamma_m=f_\\psi(z_m)+\\epsilon_m.\n",
"\\tag{3.2}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "926755fa1af8",
"metadata": {},
"source": [
"### Equation 3.3\n",
"\n",
"\n",
"\n",
"Equation (3.3) turns a continuous success probability into a single pass-or-fail bit using a cutoff chosen in advance.\n",
"\n",
"Compare the member's success probability against the threshold and report one when it reaches the threshold, zero otherwise.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\operatorname{pass}_\\tau(q_m)=\\mathbf{1}\\{q_m\\geq\\tau\\}.\n",
"\\tag{3.3}\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Mathematics of AI Agents",
"language": "python",
"name": "maa-lab"
},
"lab_chapter": 3,
"lab_execution": {
"code_cells": 6,
"created_utc": "2026-10-02T05:13:20.559049+00:00",
"elapsed_seconds": 8.833646249957383,
"method": "fresh process; new ipykernel InProcessKernelManager; cells submitted as Jupyter execute requests",
"network_transport_tested": false,
"python": "3.11.15",
"source_sha256": "0e314ae86fd181231f38f1e12b81255a7edfc5866811624370ddfea73c33b7db"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}