{ "cells": [ { "cell_type": "markdown", "id": "a5ace40c266e", "metadata": {}, "source": [ "# Chapter 13: Learning to Choose: Training an Agent from Trajectories\n", "\n", "A controller learns to choose the action its verifier rewards most. Training reward rises, and the result looks successful. Yet the verifier might favor a shortcut that seldom completes the external task. Optimization can amplify that mismatch rather than correct it.\n", "\n", "This notebook trains an actual finite softmax policy from sampled actions. It keeps verifier reward and external success probability in separate vectors and uses a separate random stream for evaluation. Repeated seeds expose stochastic learning variation. The changed case aligns the reward ranking with the task ranking. The transfer case changes terminal shaping, showing how a seemingly helpful addition can alter the objective itself.\n", "\n", "**Outcome:** Train a seeded softmax policy with REINFORCE and evaluate a separate success predicate.\n", "\n", "- Inspect the declared input contract\n", "- Predict the hand-checkable case\n", "- Run the shared computation\n", "- Change the critical assumption\n", "- Apply the method to the transfer data\n", "\n", "**Guided route:** Run the worked calculation, inspect its figure, change the stated assumption, and try the transfer case. Read the explanations beside each result before opening the answers.\n", "\n", "**Deeper route:** First read the mathematics and canonical equation reference. Audit the input contract, predict the changed result, then inspect the shared chapter implementation and solve the questions independently. Both routes use the same calculations and preserve the equations." ] }, { "cell_type": "markdown", "id": "a7781f3ee7dc", "metadata": {}, "source": [ "## Technical Requirements\n", "\n", "Python 3.11 or later, the complete laboratory folder, and the notebook dependencies listed in `requirements-notebooks.txt` (the launcher's **Install notebook tools** choice installs them; see START-HERE). Standard-library chapter commands also support Python 3.10. No API key, model account or network call is used by this experiment.\n", "\n", "Prior knowledge:\n", "\n", "- Python lists and dictionaries\n", "- The mapped chapter and its notation" ] }, { "cell_type": "markdown", "id": "fef374187866", "metadata": {}, "source": [ "## The question and its mathematics\n", "\n", "For logits theta_i, softmax gives pi_i=exp(theta_i)/sum_j exp(theta_j). REINFORCE uses the score gradient d log pi_A/d theta_i=1{i=A}-pi_i. The update here is theta_i <- theta_i+alpha(R_A-b)(1{i=A}-pi_i), where b is the current expected shaped reward. A baseline independent of the sampled action's randomness reduces variability without changing the expected score-gradient direction.\n", "\n", "This one-step experiment uses deterministic verifier reward after action selection. Shaped reward adds gamma Phi(terminal) with initial potential zero. In a properly terminated episodic construction, zero terminal potential preserves the original objective. A nonzero action-dependent terminal potential can change rankings, so the report exposes that condition.\n", "\n", "Expected verifier reward is sum_i pi_i r_i; expected external success is sum_i pi_i p_i. They are different functions of the same policy. Independent evaluation draws actions and Bernoulli success from a fresh seeded stream. Its empirical rate has finite-sample noise and is not identical to the known constructed expectation." ] }, { "cell_type": "markdown", "id": "5352ba9356c3", "metadata": {}, "source": [ "## A calculation you can run\n", "\n", "Supply aligned verifier rewards, external success probabilities, and terminal potentials. Declare episodes, learning rate, discount, seeds, and evaluation count. Each seed begins with equal logits. Stable softmax subtracts the maximum logit before exponentiation, avoiding avoidable overflow.\n", "\n", "At every training episode, the code samples one action, computes its centered shaped reward, updates all logits, and records expected unshaped verifier reward. After training, it reports the policy, expected external success, and a separate empirical evaluation rate. Curves are labeled by seed. Compare their direction with the external metrics rather than calling any upward reward curve task improvement. The changed reward vector is the only altered mechanism.\n", "\n", "The next cell finds the bundle and imports the same computation used by the chapter skill. It does not change your system Python." ] }, { "cell_type": "code", "execution_count": 1, "id": "42ebefc0e5c1", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "import sys, json\n", "LAB_ROOT = next((p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"lab-manifest.json\").is_file()), None)\n", "if LAB_ROOT is None:\n", " raise RuntimeError(\"Open this notebook from the complete extracted laboratory folder.\")\n", "sys.path.insert(0, str(LAB_ROOT / \"src\"))\n", "from math_ai_agents.core import analyze, report_text\n", "from math_ai_agents.plotting import figure_svg\n", "from IPython.display import SVG, display\n" ] }, { "cell_type": "markdown", "id": "7fffde8c0edb", "metadata": {}, "source": [ "Set the declared inputs below. These are constructed teaching values, not measurements from a production agent. Change a value only after predicting what it should change." ] }, { "cell_type": "code", "execution_count": 2, "id": "5f18e1d37a37", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Chapter 13: finite-policy-learning\n", "Does optimizing the verifier reward improve the external task?\n", "Evidence: constructed teaching example\n", "\n", "Calculated quantities:\n", "{\n", " \"runs\": [\n", " {\n", " \"seed\": 3,\n", " \"policy\": [\n", " 0.05660390942,\n", " 0.9433960906\n", " ],\n", " \"expected_training_reward\": 1.943396091,\n", " \"expected_shaped_reward\": 1.943396091,\n", " \"expected_external_success\": 0.2396227366,\n", " \"independent_evaluation_rate\": 0.245,\n", " \"evaluation_n\": 200\n", " },\n", " {\n", " \"seed\": 11,\n", " \"policy\": [\n", " 0.08134967477,\n", " 0.9186503252\n", " ],\n", " \"expected_training_reward\": 1.918650325,\n", " \"expected_shaped_reward\": 1.918650325,\n", " \"expected_external_success\": 0.2569447723,\n", " \"independent_evaluation_rate\": 0.3,\n", " \"evaluation_n\": 200\n", " }\n", " ],\n", " \"policy_invariant_shaping_condition\": true,\n", " \"verifier_optimal_action\": 1,\n", " \"shaped_rewards\": [\n", " 1.0,\n", " 2.0\n", " ],\n", " \"shaped_optimal_action\": 1,\n", " \"training_objective_conflicts_with_task\": true,\n", " \"task_optimal_action\": 0\n", "}\n", "\n", "Interpretation:\n", "REINFORCE updates softmax logits from sampled actions and a baseline, using the shaped reward (verifier reward plus discount times terminal potential). verifier_optimal_action ranks the unshaped rewards, shaped_optimal_action ranks the rewards actually trained on, and expected_training_reward is the unshaped expectation while expected_shaped_reward is the trained one. Independent evaluation uses a separate seeded stream and external success probabilities.\n", "\n", "Assumptions:\n", "- One-step finite policy; rewards are deterministic after each chosen action.\n", "- Potential shaping is gamma times the supplied terminal potential with zero initial potential.\n", "- Zero terminal potential preserves the episodic objective.\n", "\n", "Limitations:\n", "- Nonzero terminal potentials can change the objective.\n", "- Verifier reward improvement need not improve external task success.\n", "- Constructed success probabilities are not learned from actual deployments.\n", "\n", "Execution: completed locally; constructed inputs are not deployment measurements.\n" ] } ], "source": [ "chapter = 13\n", "inputs = {'rewards': [1, 2],\n", " 'success_probabilities': [0.9, 0.2],\n", " 'terminal_potentials': [0, 0],\n", " 'episodes': 120,\n", " 'seeds': [3, 11],\n", " 'learning_rate': 0.08,\n", " 'discount': 1,\n", " 'evaluation_runs': 200}\n", "report = analyze(chapter, inputs)\n", "# This input was explicitly taken from the teaching fixture.\n", "report['evidence_kind'] = 'constructed teaching example'\n", "print(report_text(report))" ] }, { "cell_type": "markdown", "id": "2d864eb55e8c", "metadata": {}, "source": [ "Initially the default policy is [0.5,0.5], with verifier reward 1.5 and external success 0.55. The verifier prefers action 1 with reward 2, while the task prefers action 0 with success probability 0.9. Learning toward the verifier can therefore raise reward while lowering expected task success.\n", "\n", "The changed reward vector [2,1] aligns both rankings. Reward-directed learning can now increase external success. Exact final policies differ by seed, so the hand check focuses on rankings, normalization, and expected metrics rather than a selected flattering run. Separate evaluation remains finite evidence even when the constructed expectation is known.\n", "\n", "The plot below uses the calculated quantities. Read each panel's units before comparing its values." ] }, { "cell_type": "code", "execution_count": 3, "id": "590f57e74048", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Matplotlib is building the font cache; this may take a moment.\n" ] }, { "data": { "image/svg+xml": [ "Calculated chapter experimentLabeled plot of the explicitly supplied chapter inputs. See the adjacent explanation for assumptions.\n", " \n", " \n", " \n", " \n", " image/svg+xml\n", " \n", " \n", " Mathematics of AI Agents Laboratory\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 0\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 20\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 40\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 60\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 80\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 100\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 120\n", " \n", " \n", " \n", " episode\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.5\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.6\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.7\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.8\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.9\n", " \n", " \n", " \n", " expected shaped reward (trained on)\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " shaped training reward seed 3\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 0\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 20\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 40\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 60\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 80\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 100\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 120\n", " \n", " \n", " \n", " episode\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.5\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.6\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.7\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.8\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.9\n", " \n", " \n", " \n", " expected shaped reward (trained on)\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " shaped training reward seed 11\n", " \n", " \n", " \n", " Chapter 13: finite policy learning\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(SVG(figure_svg(report)))" ] }, { "cell_type": "markdown", "id": "72314f82d092", "metadata": {}, "source": [ "**Figure 13.L1:** Calculated chapter experiment. Each panel labels its input and output units; interpret it under the assumptions printed in the report." ] }, { "cell_type": "markdown", "id": "270b0fb72aee", "metadata": {}, "source": [ "## Change the assumption\n", "\n", "Verifier agreement is not an independent correctness guarantee. If reward and external success share the same mistaken criterion, evaluation can repeat the training error. This experiment keeps them separate precisely to make that mismatch inspectable.\n", "\n", "Reward shaping is another boundary. Potential differences can preserve optimal behavior under their assumptions, but arbitrary terminal bonuses do not inherit that guarantee. The transfer case assigns potential 2 to the zero-reward action, giving it shaped reward 2 against the other action's reward 1. Training can then favor external failure.\n", "\n", "This runtime trains only a one-step finite policy. It does not train a language model or claim convergence of a general agent. Multi-step return estimation, function approximation, safety constraints, and deployment shifts need additional models and evidence. Record all seeds and evaluation counts rather than presenting one policy snapshot as a universal result." ] }, { "cell_type": "code", "execution_count": 4, "id": "46c8a7106654", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Chapter 13: finite-policy-learning\n", "Does optimizing the verifier reward improve the external task?\n", "Evidence: constructed changed-assumption example\n", "\n", "Calculated quantities:\n", "{\n", " \"runs\": [\n", " {\n", " \"seed\": 3,\n", " \"policy\": [\n", " 0.956006131,\n", " 0.04399386903\n", " ],\n", " \"expected_training_reward\": 1.956006131,\n", " \"expected_shaped_reward\": 1.956006131,\n", " \"expected_external_success\": 0.8692042917,\n", " \"independent_evaluation_rate\": 0.885,\n", " \"evaluation_n\": 200\n", " },\n", " {\n", " \"seed\": 11,\n", " \"policy\": [\n", " 0.9454970276,\n", " 0.0545029724\n", " ],\n", " \"expected_training_reward\": 1.945497028,\n", " \"expected_shaped_reward\": 1.945497028,\n", " \"expected_external_success\": 0.8618479193,\n", " \"independent_evaluation_rate\": 0.88,\n", " \"evaluation_n\": 200\n", " }\n", " ],\n", " \"policy_invariant_shaping_condition\": true,\n", " \"verifier_optimal_action\": 0,\n", " \"shaped_rewards\": [\n", " 2.0,\n", " 1.0\n", " ],\n", " \"shaped_optimal_action\": 0,\n", " \"training_objective_conflicts_with_task\": false,\n", " \"task_optimal_action\": 0\n", "}\n", "\n", "Interpretation:\n", "REINFORCE updates softmax logits from sampled actions and a baseline, using the shaped reward (verifier reward plus discount times terminal potential). verifier_optimal_action ranks the unshaped rewards, shaped_optimal_action ranks the rewards actually trained on, and expected_training_reward is the unshaped expectation while expected_shaped_reward is the trained one. Independent evaluation uses a separate seeded stream and external success probabilities.\n", "\n", "Assumptions:\n", "- One-step finite policy; rewards are deterministic after each chosen action.\n", "- Potential shaping is gamma times the supplied terminal potential with zero initial potential.\n", "- Zero terminal potential preserves the episodic objective.\n", "\n", "Limitations:\n", "- Nonzero terminal potentials can change the objective.\n", "- Verifier reward improvement need not improve external task success.\n", "- Constructed success probabilities are not learned from actual deployments.\n", "\n", "Execution: completed locally; constructed inputs are not deployment measurements.\n" ] }, { "data": { "image/svg+xml": [ "Calculated chapter experimentLabeled plot of the explicitly supplied chapter inputs. See the adjacent explanation for assumptions.\n", " \n", " \n", " \n", " \n", " image/svg+xml\n", " \n", " \n", " Mathematics of AI Agents Laboratory\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 0\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 20\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 40\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 60\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 80\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 100\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 120\n", " \n", " \n", " \n", " episode\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.5\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.6\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.7\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.8\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.9\n", " \n", " \n", " \n", " expected shaped reward (trained on)\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " shaped training reward seed 3\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 0\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 20\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 40\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 60\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 80\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 100\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 120\n", " \n", " \n", " \n", " episode\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.5\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.6\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.7\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.8\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " 1.9\n", " \n", " \n", " \n", " expected shaped reward (trained on)\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " shaped training reward seed 11\n", " \n", " \n", " \n", " Chapter 13: finite policy learning\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "changed_inputs = {'rewards': [2, 1],\n", " 'success_probabilities': [0.9, 0.2],\n", " 'terminal_potentials': [0, 0],\n", " 'episodes': 120,\n", " 'seeds': [3, 11],\n", " 'learning_rate': 0.08,\n", " 'discount': 1,\n", " 'evaluation_runs': 200}\n", "changed = analyze(chapter, changed_inputs)\n", "changed['evidence_kind'] = 'constructed changed-assumption example'\n", "print(report_text(changed))\n", "display(SVG(figure_svg(changed)))" ] }, { "cell_type": "markdown", "id": "109a6eecfc8f", "metadata": {}, "source": [ "**Figure 13.L2:** The changed-assumption result. Compare the printed quantities and the stated assumptions with the first run. A different input need not imply a causal effect in a deployed agent." ] }, { "cell_type": "markdown", "id": "ec7192ac3c6f", "metadata": {}, "source": [ "## Try a new case\n", "\n", "In the transfer case, action 0 has unshaped reward 0 and external success 0, but terminal potential 2 makes its shaped reward 2. Action 1 has reward 1 and success 1. The shaping condition is false, so the learner is trained on shaped rewards [2, 1] while the external task prefers action 1. Read the fields by name: verifier_optimal_action ranks the unshaped rewards and still says action 1, but shaped_optimal_action ranks the rewards the optimizer actually uses and says action 0, and training_objective_conflicts_with_task is true. expected_training_reward is the unshaped expectation, while expected_shaped_reward is the quantity the plotted curves track and the one that rises. Predict that conflict before inspecting the seeded curves.\n", "\n", "For a compatible local experiment, use a finite declared action set and an independently defined success predicate. Keep evaluation evidence outside reward adaptation. If the real question concerns a trajectory policy, treat this laboratory as a mechanics check and specify the missing return, state, and evaluation contracts before claiming broader learning capability." ] }, { "cell_type": "code", "execution_count": 5, "id": "8831fa0063ec", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Chapter 13: finite-policy-learning\n", "Does optimizing the verifier reward improve the external task?\n", "Evidence: constructed transfer example\n", "\n", "Calculated quantities:\n", "{\n", " \"runs\": [\n", " {\n", " \"seed\": 7,\n", " \"policy\": [\n", " 0.922620852,\n", " 0.077379148\n", " ],\n", " \"expected_training_reward\": 0.077379148,\n", " \"expected_shaped_reward\": 1.922620852,\n", " \"expected_external_success\": 0.077379148,\n", " \"independent_evaluation_rate\": 0.09,\n", " \"evaluation_n\": 100\n", " },\n", " {\n", " \"seed\": 17,\n", " \"policy\": [\n", " 0.9433388176,\n", " 0.05666118237\n", " ],\n", " \"expected_training_reward\": 0.05666118237,\n", " \"expected_shaped_reward\": 1.943338818,\n", " \"expected_external_success\": 0.05666118237,\n", " \"independent_evaluation_rate\": 0.09,\n", " \"evaluation_n\": 100\n", " }\n", " ],\n", " \"policy_invariant_shaping_condition\": false,\n", " \"verifier_optimal_action\": 1,\n", " \"shaped_rewards\": [\n", " 2.0,\n", " 1.0\n", " ],\n", " \"shaped_optimal_action\": 0,\n", " \"training_objective_conflicts_with_task\": true,\n", " \"task_optimal_action\": 1\n", "}\n", "\n", "Interpretation:\n", "REINFORCE updates softmax logits from sampled actions and a baseline, using the shaped reward (verifier reward plus discount times terminal potential). verifier_optimal_action ranks the unshaped rewards, shaped_optimal_action ranks the rewards actually trained on, and expected_training_reward is the unshaped expectation while expected_shaped_reward is the trained one. Independent evaluation uses a separate seeded stream and external success probabilities.\n", "\n", "Assumptions:\n", "- One-step finite policy; rewards are deterministic after each chosen action.\n", "- Potential shaping is gamma times the supplied terminal potential with zero initial potential.\n", "- Zero terminal potential preserves the episodic objective.\n", "\n", "Limitations:\n", "- Nonzero terminal potentials can change the objective.\n", "- Verifier reward improvement need not improve external task success.\n", "- Constructed success probabilities are not learned from actual deployments.\n", "\n", "Execution: completed locally; constructed inputs are not deployment measurements.\n" ] } ], "source": [ "transfer_inputs = {'rewards': [0, 1],\n", " 'success_probabilities': [0, 1],\n", " 'terminal_potentials': [2, 0],\n", " 'episodes': 100,\n", " 'seeds': [7, 17],\n", " 'learning_rate': 0.1,\n", " 'discount': 1,\n", " 'evaluation_runs': 100}\n", "transfer = analyze(chapter, transfer_inputs)\n", "transfer['evidence_kind'] = 'constructed transfer example'\n", "print(report_text(transfer))" ] }, { "cell_type": "markdown", "id": "bd6de8082213", "metadata": {}, "source": [ "## Apply the method to your inputs\n", "\n", "The example file below has the exact input shape the method accepts. Copy it to a new file, replace its values, then point `reader_file` at your copy. Run the cell again. Supplied inputs retain their stated provenance; the program cannot establish that they are representative observations.\n", "\n", "- **rewards:** Finite verifier reward per one-step action.\n", "- **success_probabilities:** External success chance per action in [0,1].\n", "- **terminal_potentials:** Potential value of each terminal action outcome; zero is the episodic invariant condition.\n", "- **episodes:** Bounded training sample count 1..10000.\n", "- **seeds:** Nonempty integer seed list.\n", "- **learning_rate:** Logit step size in [0,1].\n", "- **discount:** Gamma in [0,1].\n", "- **evaluation_runs:** Positive separate evaluation count." ] }, { "cell_type": "code", "execution_count": 6, "id": "609238b2eef4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Chapter 13: finite-policy-learning\n", "Does optimizing the verifier reward improve the external task?\n", "Evidence: supplied local inputs; provenance not independently verified\n", "\n", "Calculated quantities:\n", "{\n", " \"runs\": [\n", " {\n", " \"seed\": 7,\n", " \"policy\": [\n", " 0.922620852,\n", " 0.077379148\n", " ],\n", " \"expected_training_reward\": 0.077379148,\n", " \"expected_shaped_reward\": 1.922620852,\n", " \"expected_external_success\": 0.077379148,\n", " \"independent_evaluation_rate\": 0.09,\n", " \"evaluation_n\": 100\n", " },\n", " {\n", " \"seed\": 17,\n", " \"policy\": [\n", " 0.9433388176,\n", " 0.05666118237\n", " ],\n", " \"expected_training_reward\": 0.05666118237,\n", " \"expected_shaped_reward\": 1.943338818,\n", " \"expected_external_success\": 0.05666118237,\n", " \"independent_evaluation_rate\": 0.09,\n", " \"evaluation_n\": 100\n", " }\n", " ],\n", " \"policy_invariant_shaping_condition\": false,\n", " \"verifier_optimal_action\": 1,\n", " \"shaped_rewards\": [\n", " 2.0,\n", " 1.0\n", " ],\n", " \"shaped_optimal_action\": 0,\n", " \"training_objective_conflicts_with_task\": true,\n", " \"task_optimal_action\": 1\n", "}\n", "\n", "Interpretation:\n", "REINFORCE updates softmax logits from sampled actions and a baseline, using the shaped reward (verifier reward plus discount times terminal potential). verifier_optimal_action ranks the unshaped rewards, shaped_optimal_action ranks the rewards actually trained on, and expected_training_reward is the unshaped expectation while expected_shaped_reward is the trained one. Independent evaluation uses a separate seeded stream and external success probabilities.\n", "\n", "Assumptions:\n", "- One-step finite policy; rewards are deterministic after each chosen action.\n", "- Potential shaping is gamma times the supplied terminal potential with zero initial potential.\n", "- Zero terminal potential preserves the episodic objective.\n", "\n", "Limitations:\n", "- Nonzero terminal potentials can change the objective.\n", "- Verifier reward improvement need not improve external task success.\n", "- Constructed success probabilities are not learned from actual deployments.\n", "\n", "Execution: completed locally; constructed inputs are not deployment measurements.\n" ] } ], "source": [ "reader_file = LAB_ROOT / 'data/examples/ch13.json'\n", "reader_inputs = json.loads(reader_file.read_text())\n", "reader_report = analyze(chapter, reader_inputs)\n", "print(report_text(reader_report))" ] }, { "cell_type": "markdown", "id": "6c0d1b44695d", "metadata": {}, "source": [ "## Questions\n", "\n", "1. Compute initial default external success.\n", "\n", "2. Which action has the highest default verifier reward?\n", "\n", "3. What are transfer shaped rewards?\n", "\n", "Answers: [separate solutions](../solutions/ch13.md). Try the calculation before opening them." ] }, { "cell_type": "markdown", "id": "7d239820f09c", "metadata": {}, "source": [ "## Summary\n", "\n", "REINFORCE can optimize a declared finite reward while worsening the external task. This notebook performs real sampled logit updates, repeated seeds, and separate evaluation. The changed case aligns objectives; the transfer case breaks an episodic shaping condition. Track policy normalization, verifier reward, and external success separately. Improvement means improvement of a named objective under a stated experiment, not general agent competence.\n", "\n", "Limits of this experiment:\n", "\n", "- This is a local calculation under declared inputs, not an empirical claim about a deployed agent.\n", "- Read the returned assumptions and limitations before applying the numerical result.\n", "\n", "The assistant skill is [`maa-13-finite-policy-learning`](../skills/maa-13-finite-policy-learning/SKILL.md). It uses this notebook's tested computation and input contract." ] }, { "cell_type": "markdown", "id": "9cc9f25ba898", "metadata": {}, "source": [ "## Equations from the chapter\n", "\n", "These are the unchanged display equations and their explanations from the canonical chapter. They are a reference for the experiment, not a claim that every equation is numerically implemented by this one method." ] }, { "cell_type": "markdown", "id": "3577925da331", "metadata": {}, "source": [ "### Equation 13.1\n", "\n", "![Equation 13.1](../assets/math/e8b63b917b0cd273f2b0.svg)\n", "\n", "Equation (13.1) separates observations, sampled actions, rewards, terminal condition, and return-to-go.\n", "\n", "Observe, choose action, receive reward and next observation; stop at terminal `o_T`.\n", "\n", "LaTeX source, preserved for inspection:\n", "\n", "```latex\n", "\\tau=(o_0,a_0,r_0,o_1,a_1,r_1,\\ldots,o_T),\n", "\\qquad G_t=\\sum_{k=t}^{T-1}\\gamma^{k-t}r_k.\n", "\\tag{13.1}\n", "```" ] }, { "cell_type": "markdown", "id": "f08f2eb8f111", "metadata": {}, "source": [ "### Equation 13.2\n", "\n", "![Equation 13.2](../assets/math/5057a181c49bf1e087f0.svg)\n", "\n", "Equation (13.2) estimates how changing action probabilities changes expected trajectory return.\n", "\n", "Raise probability of sampled actions in proportion to discounted return-to-go, averaged over trajectories.\n", "\n", "LaTeX source, preserved for inspection:\n", "\n", "```latex\n", "\\nabla_\\phi J(\\phi)=\\operatorname E_{\\tau\\sim\\pi_\\phi}\\!\\left[\n", "\\sum_{t=0}^{T-1}\\gamma^t\\nabla_\\phi\\log \\pi_\\phi(a_t\\mid o_t)\\,G_t\n", "\\right].\n", "\\tag{13.2}\n", "```" ] }, { "cell_type": "markdown", "id": "5b12bf01a045", "metadata": {}, "source": [ "### Equation 13.3\n", "\n", "![Equation 13.3](../assets/math/fa10db71513b077b1f44.svg)\n", "\n", "Equation (13.3) moves some feedback between stages through a stated state potential.\n", "\n", "Add the discounted potential of the next state and subtract the potential of the state just left.\n", "\n", "LaTeX source, preserved for inspection:\n", "\n", "```latex\n", "r'_t=r_t+\\gamma\\operatorname{Pot}(x_{t+1})-\\operatorname{Pot}(x_t).\n", "\\tag{13.3}\n", "```" ] }, { "cell_type": "markdown", "id": "c78a41a46aa6", "metadata": {}, "source": [ "### Equation 13.4\n", "\n", "![Equation 13.4](../assets/math/c2a7371dff0313500054.svg)\n", "\n", "Equation (13.4) shows exactly which terms remain after intermediate potential changes cancel.\n", "\n", "Interior potentials telescope away; the initial and discounted terminal potentials survive.\n", "\n", "LaTeX source, preserved for inspection:\n", "\n", "```latex\n", "G'_0\n", "=\n", "G_0-\\operatorname{Pot}(x_0)\n", "+\\gamma^T\\operatorname{Pot}(x_T).\n", "\\tag{13.4}\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Mathematics of AI Agents", "language": "python", "name": "maa-lab" }, "lab_chapter": 13, "lab_execution": { "code_cells": 6, "created_utc": "2026-10-02T05:14:51.649301+00:00", "elapsed_seconds": 8.835897417040542, "method": "fresh process; new ipykernel InProcessKernelManager; cells submitted as Jupyter execute requests", "network_transport_tested": false, "python": "3.11.15", "source_sha256": "ef9b6b719a7c33419d5b76452cb81051ad38f82edc8a41bad84e14acc897641d" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }