{
"cells": [
{
"cell_type": "markdown",
"id": "dbbe1fc435c2",
"metadata": {},
"source": [
"# Chapter 1: When a Score Becomes a Skill\n",
"\n",
"A controller receives a new model version and its benchmark suddenly turns green. That visual change can feel like the arrival of a new skill. Yet the evaluator might be doing something much simpler: turning a gradual score increase into a pass flag at a fixed cutoff. The same outputs can support a smooth curve and a sudden-looking staircase.\n",
"\n",
"This notebook holds the supplied continuous scores fixed while changing the threshold. The experiment gives measurement its own place in the system boundary. Model change, measurement change, and environment change are different hypotheses, and a green cell cannot distinguish them. Begin by drawing the raw scores before looking at the binary result. Ask which evidence would remain if the cutoff moved.\n",
"\n",
"**Outcome:** Compare continuous evidence and thresholded scores without attributing a mechanism.\n",
"\n",
"- Inspect the declared input contract\n",
"- Predict the hand-checkable case\n",
"- Run the shared computation\n",
"- Change the critical assumption\n",
"- Apply the method to the transfer data\n",
"\n",
"**Guided route:** Run the worked calculation, inspect its figure, change the stated assumption, and try the transfer case. Read the explanations beside each result before opening the answers.\n",
"\n",
"**Deeper route:** First read the mathematics and canonical equation reference. Audit the input contract, predict the changed result, then inspect the shared chapter implementation and solve the questions independently. Both routes use the same calculations and preserve the equations."
]
},
{
"cell_type": "markdown",
"id": "a7781f3ee7dc",
"metadata": {},
"source": [
"## Technical Requirements\n",
"\n",
"Python 3.11 or later, the complete laboratory folder, and the notebook dependencies listed in `requirements-notebooks.txt` (the launcher's **Install notebook tools** choice installs them; see START-HERE). Standard-library chapter commands also support Python 3.10. No API key, model account or network call is used by this experiment.\n",
"\n",
"Prior knowledge:\n",
"\n",
"- Python lists and dictionaries\n",
"- The mapped chapter and its notation"
]
},
{
"cell_type": "markdown",
"id": "dab0fbe434bd",
"metadata": {},
"source": [
"## The question and its mathematics\n",
"\n",
"Let s(x) be a supplied continuous score at scale x. The thresholded score is B_tau(x)=1{s(x)>=tau}. A discontinuity in B_tau can occur whenever s crosses tau, even if s itself has no sharp change. This construction is a property of the measurement rule. It does not require a discontinuity inside the model.\n",
"\n",
"Adjacent raw-score slopes are [s(x_i)-s(x_(i-1))]/[x_i-x_(i-1)]. They have score-per-scale units, so the scale definition matters. Replacing model size with its logarithm would change their numerical meaning. The implementation requires strictly increasing scales to keep these slopes defined and aligned.\n",
"\n",
"Crossing count measures adjacent changes in the binary score, including downward crossings. It is not a statistical test for emergence. With only a handful of supplied points, many different underlying functions could interpolate the observations. A useful evaluation contract therefore declares the task, scoring rule, scale variable, environment, and uncertainty before attaching a mechanism label."
]
},
{
"cell_type": "markdown",
"id": "35ab50c1ab2e",
"metadata": {},
"source": [
"## A calculation you can run\n",
"\n",
"Provide aligned lists of increasing scales and scores between zero and one. Set the threshold separately. Validation rejects a repeated scale, mismatched vector lengths, nonfinite values, and a cutoff outside the score domain. The function calculates the pass flags, adjacent raw slopes, and number of binary crossings.\n",
"\n",
"The two figures share scale on the horizontal axis. The continuous chart shows score movement; the binary chart shows the measurement transformation. Compare their shapes before making any claim about new ability. In the changed case only the threshold moves from 0.5 to 0.55. The scores and scale values remain fixed. That choice isolates a measurement intervention from a model intervention. The returned interpretation keeps this distinction explicit.\n",
"\n",
"The next cell finds the bundle and imports the same computation used by the chapter skill. It does not change your system Python."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "42ebefc0e5c1",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys, json\n",
"LAB_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"lab-manifest.json\").is_file()), None)\n",
"if LAB_ROOT is None:\n",
" raise RuntimeError(\"Open this notebook from the complete extracted laboratory folder.\")\n",
"sys.path.insert(0, str(LAB_ROOT / \"src\"))\n",
"from math_ai_agents.core import analyze, report_text\n",
"from math_ai_agents.plotting import figure_svg\n",
"from IPython.display import SVG, display\n"
]
},
{
"cell_type": "markdown",
"id": "7fffde8c0edb",
"metadata": {},
"source": [
"Set the declared inputs below. These are constructed teaching values, not measurements from a production agent. Change a value only after predicting what it should change."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "9e401a38321d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 1: score-threshold\n",
"Did capability jump, or did the scoring rule create a cliff?\n",
"Evidence: constructed teaching example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"thresholded_scores\": [\n",
" 0,\n",
" 0,\n",
" 0,\n",
" 1,\n",
" 1\n",
" ],\n",
" \"crossing_count\": 1,\n",
" \"max_raw_slope\": 0.04\n",
"}\n",
"\n",
"Interpretation:\n",
"A jump in the thresholded curve may come from the scoring rule even when the supplied continuous scores change gradually.\n",
"\n",
"Assumptions:\n",
"- The same task, output boundary, and environment apply across scales.\n",
"\n",
"Limitations:\n",
"- This diagnostic neither identifies a neural mechanism nor proves a phase transition.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"chapter = 1\n",
"inputs = {'scales': [1, 2, 3, 4, 5], 'scores': [0.42, 0.46, 0.49, 0.52, 0.56], 'threshold': 0.5}\n",
"report = analyze(chapter, inputs)\n",
"# This input was explicitly taken from the teaching fixture.\n",
"report['evidence_kind'] = 'constructed teaching example'\n",
"print(report_text(report))"
]
},
{
"cell_type": "markdown",
"id": "6fb53e7552a8",
"metadata": {},
"source": [
"At threshold 0.5, the default flags are 0,0,0,1,1. There is one crossing between scales 3 and 4. The raw scores rise by only a few hundredths at each interval, and the largest adjacent slope is 0.04 score units per scale unit.\n",
"\n",
"At threshold 0.55, the flags become 0,0,0,0,1. The location of the apparent skill arrival moves to the last observation without changing a single score. The binary jump is real as a reported measurement. The unsupported inference would be that it identifies a sudden internal mechanism or a universal capability boundary.\n",
"\n",
"The plot below uses the calculated quantities. Read each panel's units before comparing its values."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "590f57e74048",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Matplotlib is building the font cache; this may take a moment.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"display(SVG(figure_svg(report)))"
]
},
{
"cell_type": "markdown",
"id": "a5146acf01e5",
"metadata": {},
"source": [
"**Figure 1.L1:** Calculated chapter experiment. Each panel labels its input and output units; interpret it under the assumptions printed in the report."
]
},
{
"cell_type": "markdown",
"id": "64ae07c66250",
"metadata": {},
"source": [
"## Change the assumption\n",
"\n",
"A threshold diagnostic can fail through overinterpretation even when its arithmetic is correct. The raw curve might conceal a genuinely sharp change between sampled scales, or the score might be a noisy estimate with a broad uncertainty interval. This notebook does not settle either possibility because it receives no replicate measurements and fits no mechanism model.\n",
"\n",
"A second failure comes from comparing scores produced under different tasks or environments. A larger score can result from an easier prompt, a changed tool interface, or a more forgiving evaluator. Those changes need separate records. The computation validates numeric alignment, but it cannot verify that the rows share a scientific contract. Keep the source of every score beside the figure and use a matched experiment to isolate the part of the assembled system being changed."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "9bf7f2a49ec3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 1: score-threshold\n",
"Did capability jump, or did the scoring rule create a cliff?\n",
"Evidence: constructed changed-assumption example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"thresholded_scores\": [\n",
" 0,\n",
" 0,\n",
" 0,\n",
" 0,\n",
" 1\n",
" ],\n",
" \"crossing_count\": 1,\n",
" \"max_raw_slope\": 0.04\n",
"}\n",
"\n",
"Interpretation:\n",
"A jump in the thresholded curve may come from the scoring rule even when the supplied continuous scores change gradually.\n",
"\n",
"Assumptions:\n",
"- The same task, output boundary, and environment apply across scales.\n",
"\n",
"Limitations:\n",
"- This diagnostic neither identifies a neural mechanism nor proves a phase transition.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
},
{
"data": {
"image/svg+xml": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"changed_inputs = {'scales': [1, 2, 3, 4, 5], 'scores': [0.42, 0.46, 0.49, 0.52, 0.56], 'threshold': 0.55}\n",
"changed = analyze(chapter, changed_inputs)\n",
"changed['evidence_kind'] = 'constructed changed-assumption example'\n",
"print(report_text(changed))\n",
"display(SVG(figure_svg(changed)))"
]
},
{
"cell_type": "markdown",
"id": "18ab29fff092",
"metadata": {},
"source": [
"**Figure 1.L2:** The changed-assumption result. Compare the printed quantities and the stated assumptions with the first run. A different input need not imply a causal effect in a deployed agent."
]
},
{
"cell_type": "markdown",
"id": "503f4117e9b7",
"metadata": {},
"source": [
"## Try a new case\n",
"\n",
"The transfer data uses a wider scale range and stronger raw-score movement. Its cutoff of 0.7 produces a pass only at the final point. Calculate the adjacent slopes with their unequal scale intervals before describing the curve. A score increase of 0.3 over 20 scale units is not the same local rate as an increase of 0.2 over 40 units.\n",
"\n",
"For local data, retain the original continuous outcome wherever possible. If only pass/fail results were saved, record that loss of information rather than reconstructing an invented smooth curve. The next useful measurement is the pre-threshold score or a set of outcomes under several declared cutoffs."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "0ef12867ee60",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 1: score-threshold\n",
"Did capability jump, or did the scoring rule create a cliff?\n",
"Evidence: constructed transfer example\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"thresholded_scores\": [\n",
" 0,\n",
" 0,\n",
" 0,\n",
" 1\n",
" ],\n",
" \"crossing_count\": 1,\n",
" \"max_raw_slope\": 0.02\n",
"}\n",
"\n",
"Interpretation:\n",
"A jump in the thresholded curve may come from the scoring rule even when the supplied continuous scores change gradually.\n",
"\n",
"Assumptions:\n",
"- The same task, output boundary, and environment apply across scales.\n",
"\n",
"Limitations:\n",
"- This diagnostic neither identifies a neural mechanism nor proves a phase transition.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"transfer_inputs = {'scales': [10, 20, 40, 80], 'scores': [0.1, 0.3, 0.6, 0.8], 'threshold': 0.7}\n",
"transfer = analyze(chapter, transfer_inputs)\n",
"transfer['evidence_kind'] = 'constructed transfer example'\n",
"print(report_text(transfer))"
]
},
{
"cell_type": "markdown",
"id": "2b9d3e4d458d",
"metadata": {},
"source": [
"## Apply the method to your inputs\n",
"\n",
"The example file below has the exact input shape the method accepts. Copy it to a new file, replace its values, then point `reader_file` at your copy. Run the cell again. Supplied inputs retain their stated provenance; the program cannot establish that they are representative observations.\n",
"\n",
"- **scales:** Finite strictly increasing numeric scale values.\n",
"- **scores:** Aligned continuous task scores in [0,1].\n",
"- **threshold:** Pass cutoff in [0,1]."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "f670416592d3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chapter 1: score-threshold\n",
"Did capability jump, or did the scoring rule create a cliff?\n",
"Evidence: supplied local inputs; provenance not independently verified\n",
"\n",
"Calculated quantities:\n",
"{\n",
" \"thresholded_scores\": [\n",
" 0,\n",
" 0,\n",
" 0,\n",
" 1\n",
" ],\n",
" \"crossing_count\": 1,\n",
" \"max_raw_slope\": 0.02\n",
"}\n",
"\n",
"Interpretation:\n",
"A jump in the thresholded curve may come from the scoring rule even when the supplied continuous scores change gradually.\n",
"\n",
"Assumptions:\n",
"- The same task, output boundary, and environment apply across scales.\n",
"\n",
"Limitations:\n",
"- This diagnostic neither identifies a neural mechanism nor proves a phase transition.\n",
"\n",
"Execution: completed locally; constructed inputs are not deployment measurements.\n"
]
}
],
"source": [
"reader_file = LAB_ROOT / 'data/examples/ch01.json'\n",
"reader_inputs = json.loads(reader_file.read_text())\n",
"reader_report = analyze(chapter, reader_inputs)\n",
"print(report_text(reader_report))"
]
},
{
"cell_type": "markdown",
"id": "9e5b903a7e46",
"metadata": {},
"source": [
"## Questions\n",
"\n",
"1. List default pass flags.\n",
"\n",
"2. Does changing only the cutoff prove model change?\n",
"\n",
"3. What is the first transfer slope?\n",
"\n",
"Answers: [separate solutions](../solutions/ch01.md). Try the calculation before opening them."
]
},
{
"cell_type": "markdown",
"id": "4d9309859e85",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"Thresholding can turn gradual score movement into an abrupt reported jump. Compare raw and transformed scores under a fixed evaluation contract, then test whether the conclusion survives a cutoff change. The figures describe supplied observations, not an internal mechanism. Preserve scale units, task definitions, and measurement provenance. A pass flag is an outcome of the scoring rule; calling it a skill requires a separate operational definition and evidence that the declared ability transfers to the tasks that matter.\n",
"\n",
"Limits of this experiment:\n",
"\n",
"- This is a local calculation under declared inputs, not an empirical claim about a deployed agent.\n",
"- Read the returned assumptions and limitations before applying the numerical result.\n",
"\n",
"The assistant skill is [`maa-01-score-threshold`](../skills/maa-01-score-threshold/SKILL.md). It uses this notebook's tested computation and input contract."
]
},
{
"cell_type": "markdown",
"id": "9cc9f25ba898",
"metadata": {},
"source": [
"## Equations from the chapter\n",
"\n",
"These are the unchanged display equations and their explanations from the canonical chapter. They are a reference for the experiment, not a claim that every equation is numerically implemented by this one method."
]
},
{
"cell_type": "markdown",
"id": "00be00292e0e",
"metadata": {},
"source": [
"### Equation 1.1\n",
"\n",
"\n",
"\n",
"Equation (1.1) draws the book's first system boundary by naming the model as a probability law over outputs given context, and nothing more.\n",
"\n",
"Given a context $c$, this returns the model's whole distribution over possible next tokens $z$, one number for every token, rather than a single answer.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"K_\\theta(z\\mid c)=\\Pr_\\theta(Z=z\\mid C=c).\n",
"\\tag{1.1}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "c49b3a091e5f",
"metadata": {},
"source": [
"### Equation 1.2\n",
"\n",
"\n",
"\n",
"Equation (1.2) compresses everything a model did on a test set into one number: the fraction of prompts a fixed rule was willing to call a success.\n",
"\n",
"For each prompt, generate an output, run the declared evaluator, and score one or zero; the reported number is the average of those verdicts.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"\\widehat U_{\\mathcal V}(\\{\\theta\\})=\\frac{1}{n}\\sum_{i=1}^{n}\n",
"\\mathbf{1}\\{\\operatorname{eval}_\\tau(\\hat z_i)=1\\}.\n",
"\\tag{1.2}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "1c60b73feb58",
"metadata": {},
"source": [
"### Equation 1.3\n",
"\n",
"\n",
"\n",
"Equation (1.3) separates the three places a task result can move: the model's distribution, the rule that picks one output, and the rule that judges it.\n",
"\n",
"Start at the left with the model's law, apply the decoding rule to get one output, then apply the evaluator to get the judged result; each arrow is a place a change can enter.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"K_\\theta(z\\mid c)\n",
"\\longrightarrow\n",
"\\hat z=\\operatorname{dec}\\bigl(K_\\theta(\\cdot\\mid c)\\bigr)\n",
"\\longrightarrow\n",
"y=\\operatorname{eval}_\\tau(\\hat z,c).\n",
"\\tag{1.3}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "0e2aa394b295",
"metadata": {},
"source": [
"### Equation 1.4\n",
"\n",
"\n",
"\n",
"Equation (1.4) converts a smooth per-token improvement into a task score that moves at a completely different rate, using nothing but multiplication.\n",
"\n",
"Raise the per-token success probability to the power of the target length; in logarithms the same statement says the target length multiplies whatever the model gained per token.\n",
"\n",
"LaTeX source, preserved for inspection:\n",
"\n",
"```latex\n",
"U=q^{n},\n",
"\\qquad\\text{so}\\qquad\n",
"\\log U = n\\log q.\n",
"\\tag{1.4}\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Mathematics of AI Agents",
"language": "python",
"name": "maa-lab"
},
"lab_chapter": 1,
"lab_execution": {
"code_cells": 6,
"created_utc": "2026-10-02T05:13:02.400484+00:00",
"elapsed_seconds": 8.927125833928585,
"method": "fresh process; new ipykernel InProcessKernelManager; cells submitted as Jupyter execute requests",
"network_transport_tested": false,
"python": "3.11.15",
"source_sha256": "472665951a16318b704b7ef0a6e3cc9a07375b5a85eb9e9d8702da9beac7f75a"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}