--- name: math-thumb-information-theory description: "Apply Chapter 15 (Information Theory: Measuring Information and Its Limits) of Mathematical Rules of Thumb to solve, check, or teach problems. Use it to compute information quantities on a declared scale and identify the operational limit they express." --- # Information Theory: Measuring Information and Its Limits Use this chapter to help the reader make a checked mathematical decision. All 12 numbered rules are available in [the chapter source](references/chapter.md). [The workbook](references/notebook.md) contains a lab, exercises, solutions, and the full rule checklist. [The local rule index](references/rules.json) supplies discovery metadata. ## Start from the reader's task Infer solve, learn, or audit mode from the request. In solve mode, use their supplied numbers and target; in learn mode, use the workbook or their chosen rule; in audit mode, inspect their actual calculation before replacing it. Gather only missing information that changes the choice: Probability distribution; log base and units; alphabet; conditioning or side information; channel/source assumptions; common evaluation data. If the question falls outside this chapter, say which mathematical operation is missing and suggest a relevant chapter. If the whole-book skill is available, it can carry the task onward, but this chapter works independently. ## Select and apply a rule - **Is the question about one outcome?** Use surprisal. If it asks for the average over a source, use entropy or cross-entropy instead. - **Is the alphabet finite?** Apply the log-alphabet ceiling before accepting a larger uncertainty or mutual-information claim. - **Is a Bernoulli event rare?** Use the small-\(p\) entropy approximation only after checking the regime and whether temporal dependence changes the entropy rate. - **Are predictive models being compared?** Keep the tokenization, evaluation data, conditioning information, and log base fixed before comparing perplexity. - **Is dependence the target?** Mutual information is broader than correlation, but finite-sample estimation and causal interpretation remain separate issues. - **Has information passed through a transformation?** Draw the Markov diagram and identify any side information before invoking data processing. - **Must an information discrepancy become an event error?** State the KL direction and units, then use Pinsker. - **Is the question operational?** Match the exact source, channel, code, or probabilistic-data-structure assumptions before using the corresponding limit or tuning formula. Read the selected complete profile, including its equation, “How to read it,” and “How to use it.” The compact graph assumptions are search cues, not a substitute for the profile. Preserve the numbered citation and role. **Independent**, **Workflow**, and **Specialized** describe the relationship to a calculation; exactness, approximation, bound, diagnostic, and heuristic describe a different dimension. Use verified inputs, show the substitution and units, and interpret the result in the reader's decision. Verify by an appropriate bound, alternative computation, limiting case, residual with conditioning, or sensitivity check. If a required condition fails, reject that use and give the specific missing information or alternative method; do not calculate a plausible-looking answer from an invalid formula. **Essential boundary:** Bits and nats are different units. Equal-looking perplexities from different tokenizations are not direct evidence of equal predictive quality. For a sufficient independent result, stop with the decision it supports. For a workflow or specialized rule, name the downstream calculation still needed. A numerical demonstration is evidence for that instance, not a universal proof. ## Teach and check understanding Use the [workbook](references/notebook.md) for guided practice. Start with 15.1.1, 15.1.2, 15.1.4 when the reader wants a starting exercise. Ask for an attempt, offer a relevant hint, and reveal the answer when requested or when teaching requires it. Do not force a quiz when the reader asked for a worked solution. Check whether the reader can explain the controlling quantity, apply the rule to a changed input, identify an invalid use, and distinguish a final answer from a preparatory step. Track only demonstrated work. Give a short prerequisite explanation when needed; avoid requiring completion of earlier chapters. ## Return a usable result Include the chosen rule numbers, assumptions that matter, calculation, verification, and next action. For ongoing work, offer this compact record: question; inputs and units; rules; claim type; book role; assumption status; result and error; check; decision; unresolved next step. Write a progress file only when asked or within an already authorized notebook-editing task. The source chapter is a fixed book snapshot. Preserve its mathematical qualifications and historical evidence gaps. Use outside material only when the reader's task needs it, verify material facts appropriately, and identify that material separately from the book. ## Illustrated exploration Open [the browser reader](assets/reader.html) or [the saved illustrated notebook](assets/notebook.ipynb). [Equation cards](references/equations.json) record the book rule, formula, fixed inputs, supported choices, assumptions and executed default results. - **C15-D01: Separate rare-event surprisal from average entropy**: rule 15.1.3. - **C15-D02: Price a probability model mismatch**: rule 15.1.4. - **C15-D03: Read a channel limit as a limit**: rule 15.3.3. - **C15-D04: Price signal power in bits**: rule 15.3.2. - **C15-D05: Pick the number of hashes for a Bloom filter**: rule 15.3.5. Use a saved illustration only when its conditions fit. Browser controls select finite precomputed choices; they do not calculate arbitrary reader inputs. For different inputs, make a checked calculation using the selected rule. Explain what changes, and never claim the notebook ran or the browser was viewed unless it did. Offer prediction questions for learning; answer direct requests without a mandatory quiz.