Chapter 5: What the Model Becomes Inside an Agent. The Mathematics of AI Agents, laboratory reader

Demonstration 1 of 4

Six assemblies around one frozen model

If the model never changes, how can six assemblies of it succeed at different rates, and what does the budget have to do with it?

A blind answer is right about as often as it is wrong, whatever the hidden bit. Rows 2 and 3 add a memory and a tool, but no later call reads their output, so they cannot move the score; the stacked bars show that no part of their runs is rescued after the request. Row 4 adds a second blind call and row 5 an informed one, so both gain only through the requests they rescue. Row 6 is row 5 with one grant revoked. Memory is what row 2 supplies, the world is what the tool reads in row 5, and the budget is what pays for row 4's second call: with one call allowed, every row falls to the blind rate.

$0.90(0.85)+0.10(0.10)=0.775$

$x_t=(c_t,m_t,w_t,b_t)$

Scroll sideways for the whole equation

A blind call is a model call that has not seen the tool's result. Its request probability is the share of blind calls that output request instead of an answer; the rest is split equally between the two answers. The hidden bit Y is 0 or 1 with equal chance. The tool reports Y correctly with probability 0.90. In a context that contains the report, the model answers with the reported bit with probability 0.85, with the other bit with probability 0.10, and outputs request with probability 0.05. A success is a correct final answer. In the state x_t = (c_t, m_t, w_t, b_t), c is the context the model sees, m the retained memory, w the world and b the remaining budget; here b is the number of calls left.

Predict first. Leave the request probability at 0.40 and change denial to a second blind call. Which bar moves, and to what value?

Your prediction Row 6 rises from 0.300 to 0.420, the value of row 4 Row 5 falls from 0.610 to 0.420 No bar moves, because denial removes the tool either way

Share of blind calls that output request

What a denied request does

Calls allowed in a run

Figure: Six assemblies around one frozen model. Six horizontal stacked bars, one per assembly, split into correct on the first call, correct after the request, wrong answer and stopped with no answer. Rows 1 to 3 end at 0.300 correct, row 4 at 0.420, row 5 at 0.610 and row 6 at 0.300. A dashed line marks the model-only rate. — Share of blind calls that output request: 0.4, What a denied request does: Stops, no answer (the book's rule), Calls allowed in a run: Two (the book's budget)
Constructed example: the chapter's six-assembly experiment; the default 0.40 request probability gives the book's values 0.30, 0.42 and 0.61.

Model only: 0.300
Memory, no reader: 0.300
Tool, no recurrence: 0.300
Recurrence, blind: 0.420
Connected, granted: 0.610
Connected, denied: 0.300
Row 5 minus row 1: 0.310
Row 5 wrong answer: 0.370
Row 5 stops with no answer: 0.020
Calls allowed: 2

A blind answer is correct with probability (1 - 0.40) / 2 = 0.300, whatever the hidden bit is. Row 4 = 0.300 + 0.40 x 0.300 = 0.420. An informed answer is correct with probability 0.90 x 0.85 + 0.10 x 0.10 = 0.775, so row 5 = 0.300 + 0.40 x 0.775 = 0.610. Rows 2 and 3 hold real components that no later call reads, so they stay at 0.300. Row 6 stops at the rejection, so it scores the blind rate 0.300, exactly row 1. The tool was never reached, so one revoked grant removes the whole gain. Check for row 5: correct 0.610 + wrong 0.370 + stopped 0.020 = 1.000.

Worked steps

Blind call: answer 0 and answer 1 each have probability (1 - 0.40) / 2 = 0.300; request has 0.40.
Either answer is right half the time, so a blind answer is correct with probability 0.300.
Row 4: the request starts a second blind call: 0.300 + 0.40 x 0.300 = 0.420.
Informed answer: 0.90 x 0.85 + 0.10 x 0.10 = 0.775; row 5 = 0.300 + 0.40 x 0.775 = 0.610.
Rows 1 to 3 cannot rescue a request, so each scores 0.300; row 6 is row 5 with the grant revoked.

Use the idea

When someone says an agent is better because it has memory or tools, ask which later decision reads their output. A component that sits on no path from the model's output to the outcome contributes exactly zero.

Where the conclusion applies

At most two model calls, a tool that is either granted or denied, and the response laws above, which hold in every row. The six bars are assembly comparisons, not a matched factorial design: row 5 also adds recurrence, so the rows do not identify a memory and tool interaction. The rejection rule matters: if denial sends the run to a second blind call, row 6 equals row 4, not row 1. Nothing here says deployed effect sizes look like these. The one-call state is the chapter's own rule (a request on the last call terminates) applied to a smaller budget.

Common wrong turn: A component that is present must be contributing

Storage nobody reads adds nothing, a tool nobody calls adds nothing, and a tool whose result never reaches a decision adds nothing. Rows 2 and 3 contain real, correctly built components that contribute exactly zero because they sit on no path from the model's output to the outcome.

What this does not settle

The six assemblies are teaching numbers, illustrative rather than a measurement of any deployed system. The size of the jump in row 5 is a modeling choice, made large because the argument is easier to see at full strength.

Chapter 5 source: "Reading the table honestly".

Check your understanding: If a blind call requested 0.60 of the time, what would the fully connected assembly (row 5) score?

Blind rate = (1 - 0.60) / 2 = 0.20. Row 5 = 0.20 + 0.60 x 0.775 = 0.20 + 0.465 = 0.665.

Chapter 5 source: section "The fair experiment". Demonstration C05-D01.

Demonstration 2 of 4

A fixed chooser, a different tool, a different system

If the chooser's probabilities never change, can changing only the tool change where the system spends its time?

Equation (5.2) averages the tool's next-state law over the chooser's proposals. With two states the sum is small enough to do by hand: each row of K is the chooser's weights times the tool's rows. Repeating that one-step kernel is recurrence, and it settles at a long-run share that depends on both factors. The two states are a coarse version of x_t = (c_t, m_t, w_t, b_t) from Equation (5.1): the kernel is a law over those states, not over prompts.

$Equation (5.2), written in LaTeX: \begin{aligned}P_{\mathrm{env}}(x'\mid x,u) &=\sum_{g,o}\operatorname{Grant}(g\mid x,u)\\ & \cdot \operatorname{Env}(w',o\mid x,u,g)\\ & \cdot \operatorname{Upd}(c',m',b'\mid x,u,g,w',o),\\[3pt]P_\theta(x'\mid x) &=\sum_u\pi_\theta(u\mid x)P_{\mathrm{env}}(x'\mid x,u).\end{aligned}$

Scroll sideways for the whole equation

A state is one of two constructed states, 0 and 1, standing for coarsened values of the agent's full state. The chooser row for state i gives the probability of each proposed action (the role of pi in the equation). The tool row for action a gives the probability of each next state (the role of P_env). Their composition K(i,j) is the chance of moving from state i to state j in one transition. The probability of state 0 after n transitions starts at the chosen starting value and is updated by K each step. The transfer case has an identity chooser, so K equals the tool, and starts with probability one half in each state.

Predict first. Switch to the changed tool and keep two transitions. Does the chance of being in state 0 go up or down, and by about how much?

Your prediction Up, above 0.68 Down, from about 0.68 to about 0.38 Unchanged, because the chooser's rows are unchanged

System

Number of transitions

Composite row 0: [0.76, 0.24]
Composite row 1: [0.41, 0.59]
State 0 after 2 transitions: 0.676
Long-run state 0: 0.631
Chooser rows: [0.80, 0.20] and [0.30, 0.70] (unchanged)

Each row of K mixes the tool's rows, weighted by the chooser's probabilities: K(0,0) = 0.8 x 0.9 + 0.2 x 0.2 = 0.76 and K(0,1) = 0.8 x 0.1 + 0.2 x 0.8 = 0.24; K(1,0) = 0.3 x 0.9 + 0.7 x 0.2 = 0.41. Starting with probability 1.00 in state 0, the chance of state 0 after one transition is 1 x 0.76 + 0 x 0.41 = 0.76, after two it is 0.76 x 0.76 + 0.24 x 0.41 = 0.676. After 2 it is 0.676, and it settles near 0.631. Only the tool differs between the default and changed systems; the chooser's probabilities are identical, yet the recurrent system the chooser lives in is different.

Worked steps

Chooser row 0 = [0.80, 0.20]; tool rows [0.9, 0.1] and [0.2, 0.8].
K(0,0) = 0.8 x 0.9 + 0.2 x 0.2 = 0.76 and K(0,1) = 0.8 x 0.1 + 0.2 x 0.8 = 0.24.
K(1,0) = 0.3 x 0.9 + 0.7 x 0.2 = 0.41; the second entry of row 1 is 1 - 0.41 = 0.59.
After one transition: 1 x 0.76 + 0 x 0.41 = 0.76.
After two transitions: 0.76 x 0.76 + 0.24 x 0.41 = 0.676.
After 2: 0.676; the long-run share is K(1,0) / (1 - K(0,0) + K(1,0)) = 0.631.

Use the idea

When a model is blamed or credited for a change in behavior, ask which factor changed. Here the chooser rows are identical in the default and changed systems, so every difference in the curves is a consequence of the tool, not of the model.

Where the conclusion applies

Two states, two proposals, no permission or budget coordinate, and a tool law that depends on the state only through the proposal. If the next state also depends on something the state leaves out, such as a memory version, the composite kernel is not a valid description and more transitions will not repair it.

Common wrong turn: Crediting the visible component

When an agent behaves differently, the most visible component need not be the cause. The frozen chooser's rows are identical in the default and changed systems; everything that differs lies in the tool and in the recurrence it sits in.

What this does not settle

Equation (5.2) describes one factorization of one constructed agent; a different architecture factors differently.

Chapter 5 source: "What this does not settle".

Check your understanding: With the default tool, what is the chance of state 0 after two transitions when the system starts in state 0?

K(0,0) = 0.76 and K(1,0) = 0.3 x 0.9 + 0.7 x 0.2 = 0.41. After two steps: 0.76 x 0.76 + 0.24 x 0.41 = 0.5776 + 0.0984 = 0.676.

Chapter 5 source: section "The law the model does not contain". Demonstration C05-D02.

Demonstration 3 of 4

What an observation is worth, and what a memory may drop

How many bits does the tool's report deliver, does delivering them change the outcome, and what happens if the memory drops the report?

With a report that is right with probability a, the posterior on the reported value is a, so H(Y | O) is the entropy of a and I = 1 - H(Y | O). That number depends only on the tool. Whether it matters for the task depends on a different question: does a later choice read it? A memory that keeps the report separates the two states and lets the model answer with it. A memory that drops it merges them, and Equation (5.3) fails because the next answer's chances differ between them, so the connected assembly degrades to the blind second call of row 4.

$Equation (5.4), written in LaTeX: I(Y;O)=H(Y)-H(Y\mid O)$

$Equation (5.3), written in LaTeX: \sum_{y:\kappa(y)=\bar y}P(y\mid x_1,a)=\sum_{y:\kappa(y)=\bar y}P(y\mid x_2,a)$

Scroll sideways for the whole equation

Y is the hidden bit and O the tool's report. H(Y) is the uncertainty about Y in bits, 1 bit when both values are equally likely. H(Y | O) is the uncertainty left after seeing O. I(Y;O) is their difference: the bits the report delivers. The accuracy is the chance the report equals Y. Success is the chance of a correct final answer. kappa merges detailed states into abstract classes; dropping the report merges the state 'report says 0' with the state 'report says 1'. Equation (5.3) asks each merged state to send equal probability into every abstract class, here the classes 'answer 0' and 'answer 1'.

Predict first. At accuracy 0.90, does the information change if the run stops right after the observation? Does the success rate change?

Your prediction The information stays 0.531 bits and success stays at the model-only 0.300 The information stays 0.531 bits and success rises to 0.610 The information falls to 0 bits and success stays at 0.300

Chance the tool reports the bit correctly

What happens to the report

Figure: What an observation is worth, and what a memory may drop. Left: the information a report delivers against the tool's accuracy, with 0.531 bits marked at accuracy 0.90. Right: four horizontal bars of the chance of a correct answer: model only 0.300, report then stop 0.300, report dropped by memory 0.420 and report kept and read 0.610; the chosen bar is highlighted. — Chance the tool reports the bit correctly: 0.9, What happens to the report: The run stops (tool, no recurrence)
Constructed example: the chapter's tool with accuracy 0.90 (about 0.531 bits) and the blind and informed response laws it declares; other accuracies are values defined for the reader.

Uncertainty after, H(Y | O): 0.469 bits
Information, I(Y;O): 0.531 bits
Success with this assembly: 0.300
Success gain over model only: 0.000

H(Y | O) = 0.1 x 3.322 + 0.9 x 0.152 = 0.469 bits, so I(Y;O) = 1 - 0.469 = 0.531 bits. The run stops after the observation, so no choice consumes the bits: success = 0.30 + 0.40 x 0 = 0.30, the model-only rate. The information is real and its decision value here is zero.

Worked steps

The tool reports the bit correctly with probability 0.9, so after a report the posterior on the reported value is 0.9.
H(Y | O) = 0.1 x 3.322 + 0.9 x 0.152 = 0.469 bits.
I(Y;O) = H(Y) - H(Y | O) = 1 - 0.469 = 0.531 bits.
The run stops, so no later call reads the report: success = 0.30 + 0.40 x 0 = 0.30.

Use the idea

For every piece of information a system collects, name the decision it changes. If no decision changes, the collection is decoration, however many bits it carries. Before a summary replaces a record, check that the states it merges send the same probabilities into the same classes.

Where the conclusion applies

One hidden bit with equal prior, a symmetric tool, and the informed response law held at the book's values for every accuracy. That includes 0.50, where the report carries no information but the declared law still responds to it, because the law is held fixed by construction. Mutual information is symmetric and not causal, and it measures uncertainty removed, not usefulness. Dropping the report entirely is the extreme case of a lossy memory; a partial summary would sit between the two connected rows.

Common wrong turn: Counting bits as value

Mutual information measures reduction in uncertainty rather than usefulness: an observation can be highly informative about something the agent has no way to act on, in which case its information value is real and its decision value is zero. It is also symmetric and carries no direction, so it is not a causal statement.

What this does not settle

Mutual information measures uncertainty reduction, not causation and not usefulness.

Chapter 5 source: "What this does not settle".

Check your understanding: A tool is right with probability 0.75. How many bits does it deliver, and what is H(Y | O)?

H(Y | O) = 0.25 x 2.000 + 0.75 x 0.415 = 0.500 + 0.311 = 0.811 bits, so I = 1 - 0.811 = 0.189 bits.

Chapter 5 source: section "What an observation is worth". Demonstration C05-D03.

Demonstration 4 of 4

Recurrence multiplies

If every step is very likely to succeed, how long a task can still survive, and what if the steps share a cause?

Equation (5.5) says a run's probability is a product of one-step probabilities. With a constant conditional success q, n steps give q multiplied by itself n times, which falls fast even when q is close to 1. A repair step raises q itself, which is why retries and verifiers are the main response to the exponent. A shared cause breaks the product altogether: the marginal rate per step is the same, but the steps are no longer independent, so completion depends on the joint law and not on p.

$Equation (5.5), written in LaTeX: \Pr(x_{0:T})=\Pr(x_0)\prod_{t=0}^{T-1}P(x_{t+1}\mid x_t)$

$Equation, written in LaTeX: n = \log 0.5 / \log p$

Scroll sideways for the whole equation

A run is a sequence of states x_0 to x_T. P(x_{t+1} | x_t) is the probability of one step. Here every required step succeeds with the same conditional probability q given that all earlier steps succeeded, and any unrepaired failure ends the task. With one retry per step, q is the chance that at least one of two independent tries succeeds. With a shared cause, one condition that is good with the marginal probability decides every step together. The half-success horizon is the number of steps n at which q^n = 0.5, n = log 0.5 / log q. The transfer case lists a different conditional rate for each of three steps.

Predict first. At per-step success 0.95 with no retry, is the chance of finishing 40 steps closer to 0.9, 0.5 or 0.1?

Your prediction Closer to 0.9 Closer to 0.5 Closer to 0.1

Per-step success

How steps relate

Figure: Recurrence multiplies. Success probability against the number of required steps up to 100 for independent steps, one retry per step and one shared cause, at per-step success 0.990; the chosen curve is highlighted and its value 0.669 at 40 steps is marked. — Per-step success: 0.99 per step, How steps relate: Independent, no retry
Constructed example: the chapter's compounding values (0.99 over 40 steps is 0.669, 0.95 is 0.129) and half-success horizons (about 14, 69 and 693 steps), computed with the laboratory's chain function; the retry rule is a construction defined for the reader, and the transfer case is the notebook's (rates 0.9, 0.8, 0.7).

Conditional success per step used: 0.9900
After 10 steps: 0.904
After 40 steps: 0.669
After 100 steps: 0.366
Steps until success falls below 0.5: 69.0 steps

Any failed step ends the task, so each step must succeed, q = 0.99. Equation (5.5) multiplies one-step probabilities along the run: two steps give 0.99 x 0.99 = 0.9801, and forty steps give 0.99 x 0.99 x ... x 0.99 (40 factors) = 0.669. The success curve falls below one half after about 69.0 steps, which is n = log 0.5 / log 0.99. This is the limit of this declared chain, not of agents in general.

Worked steps

Any failed step ends the task, so each step must succeed, q = 0.99.
Two steps: 0.99 x 0.99 = 0.9801.
Forty steps: 0.99^40 = 0.669.
Half-success horizon: n = log 0.5 / log 0.99 = 69.0 steps.

Use the idea

Before promising a long unattended task, ask what the per-step success is after the controller's own detection and recovery, not for the model's average accuracy on isolated questions.

Where the conclusion applies

Constant conditional success and independent retries that use the same budget coordinate (each retry costs budget the chain here does not track). If failures share a cause, such as bad evidence, a retry fails for the same reason and q improves less than shown. The shared-cause line is one constructed joint law with the same marginals, not a model of any real agent. Marginal step accuracies alone do not justify this product.

Common wrong turn: Using average accuracy as the per-step rate

You cannot substitute the average accuracy of isolated model answers for the conditional per-step rate: marginal step accuracies alone do not justify the product. Shared bad evidence can make failures cluster, and a verifier or recovery action changes which failures end the task.

What this does not settle

A task limit derived from this simple chain is a limit of that declared chain, not a universal ceiling on agents.

Chapter 5 source: "A failed step need not be a failed task".

Check your understanding: With per-step success 0.90 and one retry per step, what is the chance of finishing 2 steps?

q = 1 - 0.10 x 0.10 = 0.99. Two steps: 0.99 x 0.99 = 0.9801.

Chapter 5 source: section "Recurrence multiplies". Demonstration C05-D04.

Six assemblies around one frozen model

Choose an example

Calculated values

Worked steps

Use the idea

Where the conclusion applies

A fixed chooser, a different tool, a different system

Choose an example

Calculated values

Worked steps

Use the idea

Where the conclusion applies

What an observation is worth, and what a memory may drop

Choose an example

Calculated values

Worked steps

Use the idea

Where the conclusion applies

Recurrence multiplies

Choose an example

Calculated values

Worked steps

Use the idea

Where the conclusion applies