← Illustrated chapter

Chapter 12: Probability: Fast Reasoning Under Uncertainty

One hundred independent components each have a failure probability of $0.01$. The chance that at least one fails is not $0.01$, and the linear estimate $100(0.01)=1$ has already saturated. The exact calculation is

$$1-0.99^{100}\approx0.634.$$

That quick example contains most of probability’s recurring hazards. A tiny individual risk can accumulate into a substantial system risk. Independence can make a product legitimate, but only if it is actually present. An approximation can reveal the right scale while still failing as a probability. A polished decimal is useful only after its assumptions have been named.

The thirty rules in this chapter form an uncertainty ladder. The first group starts with broad bounds and recurring scales that need little distributional detail. The second group updates beliefs and replaces exact distributions with simpler approximations, but only after earning those approximations through independence, moment, shape, or rarity conditions. The third group controls tails and audits simulation precision.

The governing habit is simple: use the weakest assumptions that answer the decision, then pay for sharper claims with stronger evidence. A bound, an approximation, and an exact probability are different products. Keeping those labels visible is more important than carrying one extra decimal place.

12.1: Bounds and Patterns That Travel

These ten rules answer common questions before a full distribution has been specified. Some give rigorous worst-case bounds; others identify collision, coverage, or rare-event scales. Their value comes from portability, and their danger comes from forgetting the conditions that make them portable.

12.1.1: Markov’s One-Moment Tail Bound

History

A single ratio, mean divided by threshold, is the entire promise behind this bound: no distribution shape required, nothing beyond the fact that the quantity cannot go negative. Working in St. Petersburg, Pafnuty Chebyshev published Des valeurs moyennes in 1867, proving that averages of many observations concentrate once variance is finite.

That paper supports the moment-bound lineage this rule descends from. Attaching Markov’s name to the compact form below is a later convention, not a claim that Chebyshev wrote this exact inequality.

The equation

If $X\ge0$ and $a>0$, then

$$P(X\ge a)\le\frac{E[X]}{a}.$$

How to read it

On every outcome where $X$ reaches or passes the threshold $a$, it contributes at least $a$ to its own average, written $E[X]$ (its expectation, the mean value). That forces

$$E[X]\ge aP(X\ge a),$$

which rearranges into the bound: a quantity that can never go negative cannot spend much probability far above a high threshold while keeping a small average.

Only one fact is spent, the mean, with no assumption about shape or spread. That is what makes the bound portable and usually loose: once $E[X]/a$ exceeds one, probability’s own ceiling of one is the better number to quote.

What the bound does not tell you is how close the true chance sits to that ceiling. Equality holds only in a contrived case, so read the number as a limit on how bad things could be, not an estimate of how bad they are.

How to use it

An insurance claims analyst reviews a commercial-property line where every claim is nonnegative and last year’s book shows a mean claim of $20{,}000$. The analyst wants a ceiling on the chance a single claim exceeds $100{,}000$, the point where a reinsurance treaty starts paying:

$$P(X\ge100{,}000)\le\frac{20{,}000}{100{,}000}=0.20.$$

The analyst books $0.20$ as a worst-case ceiling, not an estimate of how often the layer is hit. Claims satisfy the nonnegativity the rule needs, but the bound goes silent on a signed quantity like net underwriting profit unless applied to a transform such as $|X|$. If the price hinges on whether the tail sits near $0.20$ or well below it, the analyst reaches for a second moment or a fitted model before signing off.

This is an Independent rule: a trustworthy mean, nonnegativity, and a threshold directly yield a portable tail certificate.

12.1.2: Chebyshev’s Distribution-Free Sigma Rule

History

A confident-looking probability can be worthless when it trusts a bell curve that was never earned. Chebyshev’s 1867 paper, the same St. Petersburg proof behind this chapter’s Markov bound, answered that risk directly: finite variance alone, with no named distribution required, was enough to cap how much probability can wander far from the mean.

The modern two-sided sigma inequality is the standard direct form of that 1867 result: refusing to assume a shape costs sharpness but buys a guarantee holding no matter what the data look like.

The equation

For a random variable with finite mean $\mu$ and positive finite standard deviation $\sigma$, and for $k>0$,

$$P(|X-\mu|\ge k\sigma)\le\frac{1}{k^2}.$$

How to read it

Apply the same one-moment argument to the nonnegative squared deviation $(X-\mu)^2$, whose average is the variance $\sigma^2$, spread measured in squared units:

$$P\!\left((X-\mu)^2\ge k^2\sigma^2\right) \le\frac{E[(X-\mu)^2]}{k^2\sigma^2} =\frac1{k^2}.$$

Thus at least $1-1/k^2$ of the probability lies strictly inside a band running $k$ standard deviations, the usual yardstick for spread, either side of the mean. Distribution-free just means the guarantee holds whatever shape the data take, which is why it is weaker than a percentage built for one named curve.

At $k=2$, the guarantee covers only $75\%$ of the probability; at $k=3$ it rises to $88.9\%$. That slow, algebraic improvement is the price of refusing to assume a tail shape, not a flaw in the proof: some discrete distributions really do sit close to that $75\%$ or $88.9\%$ figure, so the looseness cannot be argued away.

How to use it

A supplier audit is due at a metal-stamping plant where a process engineer tracks a part dimension whose mean and standard deviation are established from gauge data, though its shape has never been confirmed as normal. Setting $k=3$,

$$P(|X-\mu|<3\sigma)\ge1-\frac19=\frac89\approx88.9\%,$$

so at least $88.9\%$ of parts fall within three standard deviations of the population mean, holding regardless of true shape. Quoting the normal-curve figure of $99.7\%$ instead would overstate the guarantee unless bell shape is independently confirmed, which it has not been. Plugging a short, drift-prone gauge run into a theorem about population moments does not deliver that same guarantee, and rare, extreme outliers do not themselves invalidate Chebyshev; an infinite or undefined variance does.

This is an Independent rule: finite population variance alone supplies a rigorous two-sided concentration statement across distributions.

12.1.3: Union Bound for Many Failure Modes

History

Ten separately unlikely failure modes stop being unlikely once the question turns to whether any one of them occurs. George Boole’s 1854 An Investigation of the Laws of Thought developed inequalities for exactly that question, bounding the chance that at least one among several events occurs; the result is still known as Boole’s inequality, or the union bound.

The modern event notation below is a direct expression of Boole’s 1854 result.

The equation

For any events $A_1,\ldots,A_m$,

$$P\!\left(\bigcup_{i=1}^{m}A_i\right) \le\sum_{i=1}^{m}P(A_i).$$

The useful reported bound is the smaller of the sum and $1$.

How to read it

Write the union, the event that at least one thing happens, using flags that equal 1 when an event occurs:

$$\mathbf 1_{\cup_iA_i}\le\sum_i\mathbf 1_{A_i}.$$

Every outcome in at least one event is counted on the right at least once. Outcomes in several events are overcounted, which preserves an upper bound. No independence, no assumption that one event’s chance is unaffected by another, enters anywhere.

Strong overlap makes the result loose, but unknown dependence does not make it invalid.

Together with the obvious lower bound, it gives the quick bracket

$$\max_iP(A_i) \le P\!\left(\bigcup_iA_i\right) \le\min\!\left(1,\sum_iP(A_i)\right).$$

The gap between those endpoints measures how much joint information is missing.

How to use it

Ten independent contamination checks run across a packaged-salad plant’s line, metal detection, wash-water chlorine verification, and the rest, each failing to catch a problem with probability at most $0.001$. A food safety manager needs a bound before clearing the shift:

$$P(\text{at least one check fails})\le10(0.001)=0.010.$$

The manager books $0.010$ as a valid system-risk budget, valid even if some checks share a cause, such as one contaminated supplier lot. A sum exceeding one would be capped at one rather than reported as an impossible probability. Because the bound cannot say how much of that $0.010$ is duplicated across checks sharing a cause, checks tied to the same supplier line get a closer joint review rather than separate sign-off.

This is an Independent rule: individual event ceilings directly produce a guaranteed ceiling for their union without a dependence model.

12.1.4: Two-Term Bonferroni Correction for Overlap

History

Add up several alarm probabilities without checking whether they overlap, and the total silently double-counts every failure that two alarms share. Carlo Emilio Bonferroni, working in Florence, published Teoria statistica delle classi e calcolo delle probabilità in 1936, systematically organizing upper and lower bounds built from alternating sums of event intersections.

Retaining the individual and pairwise terms below is a direct modern use of Bonferroni’s 1936 framework.

The equation

For events $A_1,\ldots,A_m$,

$$\sum_iP(A_i)-\sum_{i<j}P(A_i\cap A_j) \le P\!\left(\bigcup_iA_i\right) \le \sum_iP(A_i).$$

How to read it

The first sum counts an outcome once for every event it enters. Subtracting the pairwise intersections, the chance that two specific events both happen at once, corrects that double counting, but an outcome in three events is then subtracted too many times. The two-term expression is therefore a lower bound, not generally an exact probability.

Inclusion-exclusion alternates: odd partial sums give upper bounds, even partial sums give lower bounds.

For three events, the exact next step adds back $P(A_1\cap A_2\cap A_3)$, which is why pairwise subtraction alone undershoots whenever all three events can occur together.

If all intersections are empty, both Bonferroni endpoints collapse to the ordinary sum.

How to use it

An environmental compliance officer at a chemical plant monitors three pollutant sensors, each firing a false alarm with probability $0.10$ on a given shift, with each pairwise overlap between two sensors firing together at probability $0.02$. Bracketing the chance that any alarm fires at all,

$$0.30-3(0.02)\le P(\text{any alarm})\le0.30,$$

so the union lies in $[0.24,0.30]$. The officer reports that range to plant management rather than either single number, and would intersect it with $[0,1]$ if an algebraic endpoint ever fell outside probability’s range. A large triple overlap, all three sensors firing during one upset, or noisy overlap estimates could leave a wide, unstable bracket, so the lower expression alone is never presented as the answer.

This is a Workflow heuristic: pairwise information tightens an earlier union-bound stage and tells you whether higher-order overlap is worth collecting.

12.1.5: Root-Sum-of-Squares for Independent Noise

History

National laboratories once combined measurement uncertainty by their own local conventions, disagreeing enough that one lab’s result could not be trusted beside another’s. In 1977 the International Committee for Weights and Measures asked BIPM to resolve the dispute; a laboratory questionnaire led to recommendations and, through an ISO working group, the 1993 Guide to the Expression of Uncertainty in Measurement, corrected and reprinted in 1995.

Root-sum-of-squares is the mathematical core of that documented consensus, with covariance restored whenever inputs share error.

The equation

For uncorrelated $X_i$ with variances $\sigma_i^2$,

$$\operatorname{Var}\!\left(\sum_i a_iX_i\right) =\sum_i a_i^2\sigma_i^2,$$

so the combined standard deviation is

$$\sqrt{\sum_i a_i^2\sigma_i^2}.$$

How to read it

Variance is squared spread, so independent or merely uncorrelated noise powers add; standard deviations themselves do not. In the general expansion, every pair also contributes

$$2a_ia_j\operatorname{Cov}(X_i,X_j),$$

where covariance measures whether two quantities tend to move together. Independence is sufficient for those cross terms to vanish; uncorrelatedness, a weaker condition, is sufficient for this identity.

For two unit-coefficient errors, the general standard deviation is

$$\sqrt{\sigma_1^2+\sigma_2^2+2\rho\sigma_1\sigma_2}.$$

With $\sigma_1=3$, $\sigma_2=4$, and correlation $\rho=0.5$, it is $\sqrt{37}\approx6.08$, not $5$. Negative covariance can reduce spread, so coefficient signs and dependence direction both matter.

For $n$ uncorrelated equal-scale errors of standard deviation $\sigma$, the combined value is $\sigma\sqrt n$. Fully coherent errors instead add linearly, producing $n\sigma$.

How to use it

Two uncorrelated error sources sit on a torque wrench, a sensor error with standard deviation $3$ newton-meters and a temperature-drift error with standard deviation $4$ newton-meters; a calibration technician combines them. Root-sum-of-squares gives

$$\sqrt{3^2+4^2}=5,$$

not the naive $7$ from adding the figures, so the wrench is certified to $5$ newton-meters combined uncertainty, with coefficients applied before squaring and units kept consistent. Shared calibration equipment or a common temperature environment can create covariance between the sources; positive covariance would make this answer understate the true uncertainty, so genuinely common systematic effects get separated out rather than labeled independent by default.

This is an Independent rule: once covariance is absent or negligible, component variances directly determine the spread of a weighted sum.

12.1.6: Total Variance: Within Plus Between

History

Every statistics course teaches that total variance splits into a within-group piece and a between-group piece; nobody can point to the paper that first proved it as a probability identity. Attribution chains reach toward analysis of variance, actuarial credibility theory, and a name sometimes attached as “Eve’s law,” but this book’s search found no secure dated event tied to this exact conditional-variance formula, so it has an evidence gap. Closing it would need a located primary paper or applied calculation that actually states the identity, dated with independent context, not a related decomposition borrowed from a nearby field. The fairest description is a standard modern identity: mathematically settled, historically unanchored.

The equation

For square-integrable $X$ and a grouping or conditioning variable $G$,

$$\operatorname{Var}(X) =E[\operatorname{Var}(X\mid G)] +\operatorname{Var}(E[X\mid G]).$$

How to read it

Subtract and add the conditional mean, the average of $X$ within one group $G$. The deviation of $X$ from its overall mean becomes a within-group residual plus a between-group mean shift. After squaring and conditioning on the group, the cross term vanishes because

$$E[X-E(X\mid G)\mid G]=0.$$

Overall variance is therefore internal spread plus variation among group means.

Neither component can be negative, an audit: total variance must be at least each piece alone, and a violated ordering signals sampling error or a bad calculation. The identity cannot tell you whether the grouping itself is meaningful, only how variance splits once a grouping is chosen.

How to use it

A school district data analyst compares end-of-year scores across two equally sized fifth-grade classrooms. Each classroom has within-classroom variance $4$ squared points, with class means $74$ and $80$. The mean of means is $77$, so between-classroom variance is

$$\frac{(74-77)^2+(80-77)^2}{2}=9.$$

Total variance is $4+9=13$ squared points, telling the analyst that classroom placement, not individual scatter, accounts for the larger share of this year’s variation. The identity is exact, but estimated components can be unstable with only two small classrooms behind them, and conditioning on a variable measured after teaching started, such as which students transferred mid-year, could produce a mathematically correct yet misleading decomposition.

This is an Independent rule: a declared conditioning structure directly partitions total variation into within- and between-group components.

12.1.7: Jensen’s Curvature Check

History

Plugging an average input into a curved cost function and calling the result the average cost quietly assumes the curve is a straight line, which most cost functions are not. Johan Ludwig Jensen, working in Copenhagen, published Sur les fonctions convexes et les inégalités entre les valeurs moyennes in 1906, centering convex functions in comparisons between a function of an average and an average of the function.

The displayed inequality is the direct modern form of Jensen’s 1906 result.

The equation

For integrable $X$ and convex $g$ on a convex domain containing both $X$ and $E[X]$, with $E[g(X)]$ well defined,

$$g(E[X])\le E[g(X)].$$

For concave $g$, the inequality reverses.

How to read it

A convex function curves upward, like a bowl, so its graph lies below the straight line, or chord, joining any two of its points. Averaging an input before applying such a curve therefore loses the upward effect of dispersion, the spread among the values being averaged. A concave function, curving downward like a dome, gives the opposite direction.

Jensen predicts a sign, not a magnitude. Equality holds when the transformation is affine over the realized range or when the random input is effectively constant.

For a positive random variable, concavity of $\log$ gives

$$E[\log X]\le\log E[X].$$

For a two-point example with equal probability on $1$ and $3$, $E[X]^2=4$ while $E[X^2]=5$. The one-unit gap is the variance, not a numerical accident.

Both $g(X)$ and the required expectations must exist; curvature alone cannot repair nonintegrable tails.

How to use it

A utility demand planner models the cost of meeting electricity load with a quadratic penalty, $g(x)=x^2$, since grid strain rises faster than load itself, forecasting mean demand $\mu=100$ megawatts with variance $\operatorname{Var}(X)=400$. Costing the mean load alone gives $g(100)=100^2=10{,}000$. But the true expected cost is

$$E[X^2]=E[X]^2+\operatorname{Var}(X)=100^2+400=10{,}400,$$

so budgeting for cost at average demand leaves grid strain running $400$ units higher than planned. The same check applies to logs, reciprocals, and growth factors, but the domain must be verified first: curvature can flip across it, and relevant expectations may not exist.

This is an Independent rule: curvature alone determines the direction of plug-in bias before a full distributional calculation.

12.1.8: At Least One Rare Event

History

A table of sparse counts, mostly zeros, occasionally a one or two, is an unlikely place to find a durable probability law. Working in Leipzig, Ladislaus von Bortkiewicz published Das Gesetz der kleinen Zahlen in 1898, analyzing deaths from horse kicks across Prussian cavalry corps-years; the sparse counts became a celebrated Poisson application.

The data and count model are historical. Interpreting the zero-count probability through today’s “at least one” shortcut is a modern extraction from that event.

The equation

For $n$ independent opportunities with common event probability $p$,

$$P(S\ge1)=1-(1-p)^n.$$

For rare events,

$$1-(1-p)^n\approx1-e^{-np}\approx np,$$

where the final step additionally requires $np\ll1$.

How to read it

The complement, “none occur,” is a product under independence, each opportunity’s outcome unaffected by the others. Since $\log(1-p)\approx-p$, repeated survival becomes exponential on the exposure scale $\lambda=np$. The linear form is only the first small-$\lambda$ expansion; unlike a probability, $np$ does not saturate at one.

For independent but unequal risks, the exact form is

$$1-\prod_{i=1}^{n}(1-p_i).$$

Without independence, the union bound still gives $P(S\ge1)\le\sum_i p_i$, but the product and exponential approximations no longer follow. Distinguishing those statements prevents an assumption from hiding inside arithmetic.

Whenever the exact complement is cheap, use it to audit the exponential shortcut rather than replacing it automatically.

How to use it

A circuit board carries $n=100$ identical solder joints, each with independent failure probability $p=0.01$ over the board’s warranty life; a reliability engineer tracks them. The chance that at least one joint fails is

$$1-0.99^{100}\approx0.634, \qquad 1-e^{-1}\approx0.632,$$

well below the naive linear estimate $np=1$, discarded as poor once the exact complement is this cheap to compute. A shared solder-batch defect or a single reflow-oven fault would break the independence the product assumes, so the engineer checks for a common cause before trusting $0.634$ as the failure rate. Heterogeneous risks would instead require $\prod_i(1-p_i)$, and exposure units must stay consistent before $np$ is interpreted at all.

This is an Independent rule: independent repeated risks directly yield the chance of at least one occurrence and its rare-event scale.

12.1.9: Birthday Collision Threshold

History

A vast space of random identifiers is not safe simply because it is enormous: a system can start colliding identifiers long before it runs out of room. Richard von Mises, working in Istanbul, published an occupancy problem with this birthday-collision structure in 1939. Historical surveys report an earlier oral attribution, but the evidence available to this book does not settle it.

The chapter anchors the story to von Mises’s documented publication and treats the compact square-root threshold as its modern operational form.

The equation

For $n\le M$ independent uniform draws from $M$ labels,

$$P(\text{no collision}) =\prod_{j=0}^{n-1}\left(1-\frac{j}{M}\right) \approx e^{-n(n-1)/(2M)}.$$

The $50\%$ collision threshold is

$$n\approx\sqrt{2M\log2}\approx1.177\sqrt M.$$

How to read it

There are $n(n-1)/2$ pairs that might collide. Each pair matches with probability $1/M$, so the expected collision count reaches order one when $n^2/M$ does. That is why the threshold grows like $\sqrt M$, not $M$.

The exponential approximation uses $n\ll M$, even though $n^2/M$ may be order one.

The halfway constant comes from setting the approximate no-collision probability to $1/2$:

$$\frac{n(n-1)}{2M}\approx\log2.$$

Replacing $n(n-1)$ by $n^2$ then exposes the $1.177\sqrt M$ scale. The exact product remains available when that simplification is too crude.

How to use it

A uniform space of $M=2^{32}$ possible values holds the random 32-bit tracking IDs a backend software engineer assigns to incoming events; the engineer wants to know how many can be generated before duplicate IDs become a coin flip:

$$1.177\sqrt{2^{32}} =1.177(2^{16}) \approx77{,}000$$

events at the halfway point, far fewer than the four billion the space suggests, a signal to widen the ID space or add a collision check well before then. Nonuniform ID generation would shorten that runway further, while dependence between draws could push it either way; near a real design boundary the exact product replaces the $50\%$ threshold, which is no safe operating limit.

This is an Independent rule: the size of a roughly uniform label space directly reveals the scale at which duplicates become likely.

Figure 12.1. For 365 equally likely labels and independent draws, the exact collision probability passes one-half at 23 draws. This constructed uniform model excludes unequal frequencies and dependence.

12.1.10: Coupon Collector Scale

History

A collection feels nearly finished once you’ve drawn about as many times as there are categories, yet the last few categories cost far more than their share. Abraham de Moivre’s De Mensura Sortis, read to the Royal Society in London in 1711, treated waiting-time questions in games of chance. Historical studies trace the equal-probability collection problem to that work, long before commercial coupons supplied its modern name.

The harmonic formula is today’s quantitative summary of de Moivre’s question.

The equation

For $M$ equally likely categories sampled independently,

$$E[T]=MH_M, \qquad H_M=\sum_{k=1}^{M}\frac1k.$$

As $M$ grows,

$$MH_M\approx M(\log M+\gamma)+\frac12.$$

How to read it

When $k$ categories remain unseen, the chance that the next draw is new is $k/M$. The expected geometric wait for that discovery is $M/k$. Summing the stages gives the harmonic factor.

The late stages dominate: finding the final category alone takes $M$ draws on average. Complete coverage therefore costs $M\log M$ draws; $M$ alone is nowhere near enough.

The stage means are

$$1,\ \frac{M}{M-1},\ \ldots,\ \frac{M}{2},\ M.$$

For $M=100$, the harmonic multiplier is about $5.19$: repeated common categories consume most of the budget while the last unseen type remains. The mean is a long-run average, never a completion guarantee for your run.

How to use it

A warehouse operations manager is testing an automated picker against $M=100$ equally likely SKU bins, wanting to know how many random pick cycles are needed before every bin type has been visited at least once:

$$E[T]=100H_{100}\approx518.7,$$

so roughly $519$ cycles is the planning mean, not a guarantee all $100$ bins are covered by then, nor a hard test deadline. The picker samples bins close to uniformly, but if one bin type is rarer in the routing logic, it could dominate the wait and blow past the planning figure. For a real pass or fail deadline, the manager calculates a completion quantile or simulates actual bin-visit frequencies instead of trusting the mean alone.

This is an Independent rule: the number of equally likely categories directly sets the expected complete-coverage scale.

12.2: Updating and Distributional Approximations

The next twelve rules exchange exactness for speed or update uncertainty with evidence. Each shortcut has an admission ticket: base rates for Bayes, a prior for succession, finite moments for the central limit theorem, recognizable distributional shape, or a rare-event regime. Check the ticket before using the approximation.

12.2.1: Bayes Updating in Odds Form

History

When nobody tracks where odds started, a rare condition can look alarmingly common the moment one positive signal comes back. Working at Bletchley Park in 1941, Alan Turing wrote about applying probability to cryptography. In Banburismus, cryptanalysts represented clues as changes in odds and as logarithmic weights of evidence, measured in bans and decibans, before spending scarce machine time.

The modern likelihood-ratio equation directly captures that Bletchley Park practice.

The equation

For complementary hypotheses $H_1$ and $H_0$ with positive prior probabilities, and evidence $E$ with $P(E\mid H_0)>0$,

$$\frac{P(H_1\mid E)}{P(H_0\mid E)} = \frac{P(H_1)}{P(H_0)} \times \frac{P(E\mid H_1)}{P(E\mid H_0)}.$$

Posterior odds equal prior odds times the likelihood ratio.

How to read it

The prior odds carry the base rate. The likelihood ratio measures how many times more compatible the evidence is with one hypothesis than the other. Evidence multiplies existing odds; it does not erase where the update began.

Taking logarithms makes successive conditionally independent likelihood ratios additive, meaning each new piece of evidence adds no leftover information about the others once the hypothesis is fixed. Without that conditional independence, multiplying marginal ratios can count the same information twice.

Probability $p$ converts to odds $p/(1-p)$, and odds $o$ convert back to $o/(1+o)$. A likelihood ratio of one leaves odds unchanged; values above one favor $H_1$, and values below one favor $H_0$. Direction and base rate are therefore visible before conversion back to probability.

How to use it

A condition has $1\%$ prevalence in a patient population, giving prior odds $1:99$; a hospital diagnostician is evaluating a screening test for it. A positive result carries a published likelihood ratio of $20$, updating the odds to $20:99$:

$$P(H_1\mid E)=\frac{20}{20+99} =\frac{20}{119} \approx16.8\%.$$

That is neither the $20\%$ the raw ratio suggests nor near certainty, so the diagnostician orders a confirmatory test rather than acting on $16.8\%$ alone, first checking that the test’s published performance came from a population like this one and that any second signal stays independent of the first once conditioned on each hypothesis.

This is an Independent rule: prior odds and a defensible likelihood ratio directly produce updated odds across binary evidence problems.

12.2.2: Laplace Rule of Succession

History

Few results in probability have been argued over as long as Laplace’s sunrise calculation, and the argument began with Laplace himself. Working in Paris, he gave a predictive probability after repeated successes in his 1814 Essai philosophique sur les probabilités, then immediately warned that the famous stylized sunrise version of it ignored relevant physical knowledge.

Both the rule and the caveat are documented: a mathematical update is only as appropriate as the exchangeability model and prior behind it.

The equation

With $s$ successes in $n$ Bernoulli trials and a $\operatorname{Beta}(1,1)$ prior,

$$p\mid\text{data} \sim\operatorname{Beta}(s+1,n-s+1),$$

and the posterior mean, also the next-trial predictive probability, is

$$E[p\mid\text{data}]=\frac{s+1}{n+2}.$$

How to read it

The uniform beta prior contributes one notional success and one notional failure. This prevents a short perfect record from becoming a probability of exactly one and prevents zero observed successes from becoming permanent impossibility.

The pseudo-counts are a prior choice, not objective data. Other knowledge should produce other prior parameters.

Observed failures equal $n-s$, so the posterior has parameters $s+1$ and $n-s+1$. Before any observations, its mean is $1/2$; as $n$ grows, the two prior counts become less influential. This makes the amount of smoothing explicit.

How to use it

A supplier quality manager reviews a new parts supplier’s qualification run of ten units, all ten passing inspection. Rather than reporting a raw $100\%$ pass rate, the manager applies the smoothed estimate:

$$\frac{10+1}{10+2} =\frac{11}{12} \approx0.917.$$

That $0.917$ leaves room for a future defect while still crediting the clean run, and the manager uses it to set a conservative reorder quantity rather than treating the supplier as flawless. The shortcut only holds if the ten units can reasonably share one constant defect probability and were tested independently; a known process change partway through the run would break that assumption, and no stylized prior should override solid physical knowledge about how the parts were made.

This is an Independent rule: under an explicit uniform beta prior, a success count directly yields a smoothed predictive probability.

12.2.3: Central-Limit Approximation for a Sample Mean

History

Why does averaging many small, unrelated measurements settle into the same bell-shaped pattern almost every time, whatever the individual measurements look like? Working in London, Abraham de Moivre circulated his Approximatio ad Summam Terminorum Binomii in 1733, replacing a large symmetric binomial sum with a bell-shaped integral and calculating its tail areas. His theorem concerned that one symmetric binomial problem, not arbitrary sample means.

The general sample-mean central limit theorem is a later extension of that 1733 normal-approximation breakthrough.

The equation

For iid observations with mean $\mu$ and positive finite variance $\sigma^2$,

$$\frac{\bar X-\mu}{\sigma/\sqrt n} \overset{d}{\longrightarrow}N(0,1)$$

as $n\to\infty$.

How to read it

Centering removes the population mean, and multiplying by $\sqrt n$ offsets the shrinking spread of an average. Under the stated conditions, the standardized distribution approaches a universal normal shape.

The arrow is a limit, not a finite-sample error guarantee. Convergence speed depends on skewness, tails, dependence between observations, and whether a central or an extreme probability is being approximated.

Independence, each observation carrying no information about the others, also gives

$$\operatorname{Var}(\bar X)=\frac{\sigma^2}{n}.$$

That identity explains the standard-error scale even before normality is invoked. Quadrupling $n$ halves the spread, while the CLT separately supplies the approximate shape used for probability calculations.

How to use it

An agronomist samples yield from $n=100$ independently managed test plots of a new wheat variety, mean $50$ bushels per acre, standard deviation $20$ bushels. The sample mean has

$$E[\bar X]=50, \qquad \operatorname{SE}(\bar X)=\frac{20}{10}=2$$

bushels, so central sampling probabilities for the trial average approximate $N(50,2^2)$, but only after checking plot yields are roughly independent and not dominated by a few outlier fields. There is no universal “$n=30$” permission slip: a drought-damaged corner, or irrigation shared across nearby plots, would break independence and demand more plots before the approximation is trusted.

This is an Independent rule: suitable repeated sampling directly supplies the normal scale and $1/\sqrt n$ spread of a sample mean.

12.2.4: The 68-95-99.7 Normal Rule

History

Three memorized percentages, $68$, $95$, and $99.7$, are the entire artifact behind this rule: a portable digest of a tail-area calculation nobody needs to redo by hand. De Moivre’s 1733 binomial approximation, the same bell-shaped curve behind this chapter’s central-limit rule, supplied the early version of that calculation. The three figures are a modern teaching digest, not wording from his pamphlet.

The mnemonic remains explicitly conditional on normal shape.

The equation

For $X\sim N(\mu,\sigma^2)$,

$$P(|X-\mu|\le\sigma)\approx0.683,$$

$$P(|X-\mu|\le2\sigma)\approx0.954, \qquad P(|X-\mu|\le3\sigma)\approx0.997.$$

How to read it

Standardization turns any normal variable into $Z=(X-\mu)/\sigma$. The three numbers are areas under the standard normal curve between symmetric cutoffs.

They describe a distribution, not every dataset that has a mean and standard deviation. Compare with Chebyshev’s much weaker but distribution-free guarantee.

By symmetry, the mass outside $2\sigma$ is about $4.6\%$, split roughly $2.3\%$ into each tail. Outside $3\sigma$ it is about $0.27\%$ total. Those complements are often more useful for alarms than the inside percentages.

These figures use population $\mu$ and $\sigma$; estimating them from the same small sample adds uncertainty.

How to use it

A packaging quality auditor at a snack-food plant reviews $1{,}000$ independent bag weights from a filling line that has run stable and nearly normal for months. The mnemonic predicts $0.954(1{,}000)=954$ bags within $2\sigma$ of target weight and $0.0027(1{,}000)=2.7$ bags beyond $3\sigma$ per batch, expectations the auditor uses to set a routine reject-rate alarm, not fixed quotas to hit exactly. Before trusting the numbers, the auditor checks a histogram of recent weights: a sticking filler valve or a heavy right tail from occasional overfills would create far more extremes than the rule predicts. When a customer complaint hinges on the gap between “about $95\%$” and a calibrated probability, the auditor runs exact normal software instead of the mnemonic.

This is an Independent rule: once a normal model is justified, standard-deviation distance directly translates into memorable probability mass.

12.2.5: Normal Approximation to a Binomial Count

History

Treated as safely bell-shaped without a second thought, a binomial count can betray that assumption exactly where it sits near zero or near its ceiling, right when a decision depends on it. De Moivre’s 1733 work, met once already in this chapter, matched a bell curve to those binomial counts and read their tail areas directly; later work generalized his symmetric case to any success probability.

The normal approximation descends directly from that calculation. The familiar “expected successes and failures near ten” screen is a modern guardrail, not de Moivre’s own constant.

The equation

For $X\sim\operatorname{Bin}(n,p)$, use

$$X\approx N\!\left(np,np(1-p)\right)$$

when both

$$np\gtrsim10, \qquad n(1-p)\gtrsim10.$$

How to read it

A binomial count is a sum of independent Bernoulli variables, each a coin-flip-like trial. The two checks keep both expected successes and failures away from zero, screening out severe skew, a lopsided distribution with one tail longer than the other, and a nearby boundary.

The normal is continuous while the binomial is discrete. A half-unit continuity correction usually improves moderate-sample probability calculations.

The binomial skewness is

$$\frac{1-2p}{\sqrt{np(1-p)}}.$$

The success–failure checks make its magnitude smaller and keep the normal curve from assigning substantial area beyond $0$ or $n$. They are screening rules, not a uniform accuracy guarantee.

How to use it

A clinical trial statistician expects $n=200$ enrolled patients to respond to a treatment at rate $p=0.30$, based on prior trials. The normal approximation gives

$$\mu=np=60, \qquad \sigma=\sqrt{np(1-p)}=\sqrt{42}\approx6.48,$$

so counts from about $47$ to $73$ span roughly two approximating standard deviations, the range the statistician uses to flag whether an interim count looks unusually low or high. Near the trial’s stopping boundary, the statistician switches to exact binomial calculations rather than trusting the approximation, and watches for site-to-site heterogeneity in response rates, which can inflate variance beyond what this formula assumes. For a cumulative event such as $P(X\le k)$, standardizing $k+0.5$ using mean $np$ and standard deviation $\sqrt{np(1-p)}$ keeps the correction explicit and reproducible.

This is a Workflow heuristic: it selects a faster continuous approximation after the binomial structure and success–failure conditions have been checked.

Figure 12.2. For Bin(200,0.30), normal probability over each half-unit bin closely tracks the exact mass near the center. Tail decisions still warrant exact calculation; the success-failure checks are only a screen.

12.2.6: Normal Approximation to a Poisson Count

History

Two famous nineteenth-century results sit on either side of this rule, Poisson’s rare-event counting and de Moivre’s bell-shaped binomial approximation, yet no located paper welds them into this specific Poisson-to-normal shortcut. Combining the two into one invented episode would manufacture an event neither mathematician recorded, so the rule has an evidence gap: the search verified Poisson’s 1837 Recherches and de Moivre’s 1733 work, but found no dated event for this exact approximation or its modern $\lambda$-near-ten threshold. Closing the gap would need a primary source that states or applies the joint rule, not two ingredients standing near each other in a textbook. The chapter presents it as contemporary practice instead.

The equation

If $X\sim\operatorname{Pois}(\lambda)$, approximate

$$X\approx N(\lambda,\lambda),$$

commonly screening for

$$\lambda\gtrsim10.$$

How to read it

A Poisson count has mean and variance both equal to $\lambda$, standard deviation $\sqrt\lambda$, and skewness, a lopsidedness with one tail longer than the other, equal to $1/\sqrt\lambda$. As $\lambda$ grows, relative skew falls and the central part becomes more bell-shaped.

The threshold is a heuristic, not a theorem guaranteeing a requested error. Lower tails remain constrained by zero, and far-tail relative errors can be poor.

Relative standard deviation is

$$\frac{\sqrt\lambda}{\lambda}=\frac1{\sqrt\lambda}.$$

At $\lambda=25$ it is $20\%$, and skewness is also $0.2$. Increasing $\lambda$ reduces both, explaining the direction of improvement without certifying the cutoff.

The screen is intended for central probabilities; a large mean alone does not guarantee accurate extreme tails.

How to use it

An emergency department charge nurse tracks overnight patient arrivals, historically averaging $\lambda=25$ per shift, so the standard deviation is $\sqrt{25}=5$. Two standard deviations bracket the range $25-2(5)=15$ to $25+2(5)=35$ arrivals, with a half-unit correction for a specific integer count, flagging an unusually heavy or light shift for staffing. For a rare quiet night near zero, or a staffing call that hinges on the exact tail, the nurse calculates the Poisson probability directly. If arrival counts run more variable than the mean suggests, perhaps from clustered ambulance runs after a highway incident, that overdispersion signals a different count model is needed.

This is a Workflow heuristic: it routes a verified Poisson model to a normal calculation only after scale and tail location are judged adequate.

12.2.7: Half-Unit Continuity Correction

History

A discrete count meeting a continuous curve without its half-unit adjustment leaves a probability estimate quietly shifted by a small systematic sliver. De Moivre’s 1733 work, the refinement layer on the bell-curve approximation used elsewhere in this chapter, replaced a sum of discrete binomial masses with continuous area; placing an integer mass inside a unit-width cell is the modern geometric refinement of that calculation.

The half-unit convention lines the area up with whole probability bars rather than their centers.

The equation

For an integer-valued $X$ approximated by continuous $Y$,

$$P(X\le k)\approx P(Y<k+0.5),$$

$$P(X\ge k)\approx P(Y>k-0.5).$$

More generally, $a\le X\le b$ maps to $a-0.5<Y<b+0.5$.

How to read it

The mass at integer $k$ corresponds geometrically to the continuous cell $[k-0.5,k+0.5]$. Shifting an endpoint captures the complete boundary cell instead of cutting through its center.

For a continuous $Y$, strict and non-strict inequalities have the same probability, but the integer event must be translated carefully.

The correction is easiest to audit by drawing unit cells centered at the included integers: an event containing one endpoint but not the next ends halfway between their centers, which prevents memorized “plus or minus $0.5$” rules from reversing a tail. The half-unit shift cannot repair skew or a wrong model.

How to use it

A plant quality control lead approximating the count of defective units on a shift, $P(X\ge9)$ defects, with a matching normal distribution standardizes $8.5$, not $9$. For $P(4<X\le9)$, the integer values covered are $5$ through $9$, so the lead uses continuous limits $4.5$ and $9.5$. The correction reduces discretization error only, so the lead writes down the original integer event before moving either boundary; it will not rescue a batch-dependent or genuinely skewed count.

This is a Workflow heuristic: it refines a discrete-to-continuous approximation after the normal route has already been selected.

12.2.8: Berry-Esseen Check on CLT Accuracy

History

The probability estimate that follows a central-limit claim can carry an error nobody has actually bounded. Andrew Berry’s 1941 paper bounded exactly that distance, between a standardized sum and the normal curve; Carl-Gustav Esseen independently sharpened and systematized the analysis in work culminating in 1945.

Their documented achievement gave the central limit theorem a finite-sample rate. The universal numerical constant has been improved since their papers.

The equation

For iid $X_i$ with mean $\mu$, variance $\sigma^2>0$, and $\rho=E|X_1-\mu|^3<\infty$, let $S_n=\sum_{i=1}^n(X_i-\mu)$. Then

$$\sup_x\left| P\!\left(\frac{S_n}{\sigma\sqrt n}\le x\right)-\Phi(x) \right| \le \frac{C\rho}{\sigma^3\sqrt n},$$

with a universal $C<0.475$.

How to read it

The standardized third absolute moment $\rho/\sigma^3$ prices tail weight and asymmetry, while $1/\sqrt n$ gives the worst-case rate. The bound controls the absolute error in the cumulative probability curve (CDF) uniformly in $x$.

Uniform absolute error can still be useless for a tiny tail: an error ceiling of $0.05$ does not validate a probability near $10^{-4}$.

The bound scales linearly with the standardized third absolute moment. Doubling that ratio doubles the certificate; quadrupling $n$ halves it. Those relationships turn Berry–Esseen into a sample-size and tail-shape diagnostic even when its numerical ceiling is conservative.

How to use it

A portfolio risk analyst considers a model of $400$ iid daily returns with finite third absolute moment and an established population bound $\rho/\sigma^3\le2$. To audit a normal approximation for the standardized sum, the analyst calculates

$$\frac{0.475(2)}{\sqrt{400}}=0.0475,$$

so the analyst treats the approximation’s CDF error as at most $4.75$ percentage points, rigorous but possibly loose, before relying on it for a routine risk report. An empirical moment ratio estimated from those same 400 days would be only a diagnostic, not this population guarantee. The analyst must justify the iid model and the moment bound; for the extreme tail behind a stress-test capital requirement, sharper distribution-specific analysis replaces this bound entirely.

This is a Workflow heuristic: it audits whether a proposed CLT approximation has a defensible finite-sample error scale.

12.2.9: Poisson Approximation to a Binomial Count

History

This section already earned a bell curve for binomial counts once both successes and failures pile up by the dozens; the opposite regime, where successes stay rare no matter how many trials run, is what Siméon-Denis Poisson worked out in Paris, publishing Recherches sur la probabilité des jugements in 1837. Beyond its framing around whether juries reach correct verdicts, the treatise developed probability approximations for many Bernoulli opportunities with small individual probabilities.

The rare-event limit is the historical anchor; today’s numerical screening rules were attached later.

The equation

For $X\sim\operatorname{Bin}(n,p)$, set $\lambda=np$ and use

$$P(X=k)\approx e^{-\lambda}\frac{\lambda^k}{k!}$$

when $n$ is large and $p$ is small.

How to read it

As $n$ grows and $p$ shrinks with $np$ held fixed, the binomial mass converges to the Poisson mass. Individual opportunity details collapse into the expected count $\lambda$.

This approximation is built for rare events; large $n$ alone does not justify it. A large $p$ retains binomial asymmetry and finite-capacity effects.

The limiting calculation preserves the expected count because both models have mean $\lambda=np$. Their variances differ:

$$\operatorname{Var}_{\mathrm{Bin}}(X)=\lambda(1-p), \qquad \operatorname{Var}_{\mathrm{Pois}}(X)=\lambda.$$

Small $p$ makes that discrepancy small and shows why the approximation worsens as $p$ grows.

How to use it

An extension agronomist samples $n=100$ plants for a pest that infests each independently with probability $p=0.05$, so $\lambda=5$. The approximation gives

$$P(X\le3) \approx e^{-5}\sum_{k=0}^{3}\frac{5^k}{k!} \approx0.265,$$

close to the exact binomial value near $0.258$, and the agronomist uses that to judge whether an infested-plant count of $3$ or fewer is unremarkable before recommending treatment. Were the infestation rate closer to one, the agronomist would count the rarer healthy plants instead. Clustered infestation can break independence; different plant-level rates can preserve independence but break the common-probability binomial model. The agronomist uses a model matching that structure and an exact calculation whenever the decision is close.

As a quick internal check, the Poisson point mass at three is $e^{-5}5^3/3!\approx0.140$. Summing point masses must stay between zero and one and approach one as the upper limit grows.

This is a Workflow heuristic: it replaces a verified rare binomial count with a simpler Poisson calculation inside a larger probability workflow.

12.2.10: Exponential Approximation to a Geometric Wait

History

Waiting for a rare discrete event step by step turns awkward once the steps are many and each one’s chance is tiny; a continuous stand-in removes the awkwardness. Bortkiewicz’s 1898 horse-kick study, already put to work earlier in this chapter, documented a Poisson application: a Poisson process has exponential waiting times, while Bernoulli trials instead have geometric waits.

Using that connection for the small-$p$ scaling below is a modern process interpretation; the historical book did not derive or name this approximation.

The equation

Let $T$ count trials through the first success, with success probability $0<p<1$. For integers $k\ge0$,

$$P(T>k)=(1-p)^k\approx e^{-pk},$$

so as $p\to0$,

$$pT\approx\operatorname{Exp}(1).$$

How to read it

The approximation follows from $\log(1-p)\approx-p$. When each step carries a tiny hazard, discrete survival over $k$ steps resembles continuous exponential survival over scaled time $pk$.

Stating the geometric convention matters: some definitions count failures before success and begin at zero.

Under the convention used here,

$$E[T]=\frac1p, \qquad E[pT]=1,$$

matching the mean of $\operatorname{Exp}(1)$. If a convention uses failures before success, it replaces $T$ by $T-1$; after multiplication by small $p$, that one-step difference vanishes asymptotically but not in an exact short wait.

The exponential surrogate also puts the median wait near $(\log2)/p$, since $e^{-pk}=1/2$ at that point. This is shorter than the mean $1/p$, reflecting the long right tail.

How to use it

A site reliability engineer models a service crashing on any given deploy with probability $p=0.01$ per deploy, and wants the chance the service survives past $100$ deploys without a crash:

$$P(T>100)=0.99^{100}\approx0.366, \qquad e^{-1}\approx0.368.$$

The approximation is excellent at this scale, so the engineer plans capacity using the exponential surrogate rather than tracking each deploy. If $p$ is not small but remains constant across independent releases, return to the exact geometric calculation. For independent releases with varying crash probabilities $p_i$, survival through $n$ releases is $\prod_{i=1}^n(1-p_i)$ instead; clustered crashes require a model of their dependence.

This is a Workflow heuristic: it selects a continuous waiting-time surrogate after a rare, constant-probability discrete process has been established.

12.2.11: Lognormal Mean Correction

History

Two different numbers can each honestly claim to be the “typical” value of the same skewed measurement, and the gap between them is structural, not an error. Working in London, Donald McAlister described that structure in an 1879 paper on a skewed “law of frequency” arising from multiplicative rather than additive effects; the paper is recognized as an early derivation of the distribution later called lognormal, distinguishing its geometric center from its arithmetic mean.

The compact correction below is the modern parameterization of McAlister’s distinction.

The equation

If

$$Y=\log X\sim N(\mu,\sigma^2),$$

then

$$\operatorname{median}(X)=e^\mu, \qquad E[X]=e^{\mu+\sigma^2/2}.$$

How to read it

Exponentiating the mean log returns the median because $P(Y\le\mu)=1/2$. The arithmetic mean is higher by the factor $e^{\sigma^2/2}$ because the convex exponential transformation gives extra weight to the right tail.

The variance here is log-scale variance, not variance of $X$ in original units.

The mean-to-median ratio is

$$\frac{E[X]}{\operatorname{median}(X)} =e^{\sigma^2/2}.$$

It grows with log-scale dispersion and is another exact illustration of Jensen’s inequality for the convex exponential function. Zero variance makes the two summaries coincide.

The formula uses the natural logarithm. Another log base requires conversion before applying the exponential correction.

How to use it

An environmental scientist models groundwater contaminant concentration, in micrograms per liter, as lognormal, with log mean $\mu=2$ and log variance $\sigma^2=0.25$ estimated from monitoring wells:

$$\operatorname{median}(X)=e^2\approx7.39, \qquad E[X]=e^{2.125}\approx8.37.$$

Reporting only $e^\mu=7.39$ as “the mean” concentration to a regulator would understate the true arithmetic average by nearly a full microgram, since the exact correction factor requires normal log errors and the log-scale variance used here. If different wells show different scatter around their own logs, that heteroskedasticity calls for an observation-specific correction or a smearing estimate rather than one global factor.

This is an Independent rule: a normal log model directly determines the distinction between median and arithmetic mean on the original scale.

12.2.12: Delta Method for a Smooth Transformation

History

A technique used constantly in applied statistics, rescaling a standard error through a smooth function’s derivative, carries a name, “the delta method,” that nobody can trace to a founding paper. Retrospective accounts reach toward early error propagation, Taylor approximations, and general asymptotic theory, and one 1981 paper on standard errors gets mentioned often, but this book’s survey found no securely dated discovery or real application of this exact rule, so it has an evidence gap. Closing it would need a primary source that actually states and applies the transformation rule, not a later paper merely discussing standard errors nearby. The mathematics is well supported; the chapter presents it as established technique rather than inventing an origin.

The equation

If

$$\widehat\theta\approx N(\theta,V)$$

and $g$ is differentiable near $\theta$ with $g'(\theta)\ne0$, then

$$g(\widehat\theta) \approx N\!\left(g(\theta),[g'(\theta)]^2V\right).$$

How to read it

A first-order Taylor expansion, approximating a smooth function by its local slope, gives

$$g(\widehat\theta)-g(\theta) \approx g'(\theta)(\widehat\theta-\theta).$$

The transform therefore multiplies standard error by $|g'(\theta)|$, the size of that slope. The method is local and asymptotic; curvature and boundary effects are omitted at first order.

The variance rule is

$$\operatorname{Var}(g(\widehat\theta)) \approx[g'(\theta)]^2V.$$

Because the derivative is evaluated at an unknown parameter, applications usually plug in $\widehat\theta$. That extra approximation is harmless only when the derivative changes little over the estimator’s uncertainty range.

How to use it

An economist estimates a regional price index at $\widehat\theta=100$ with standard error $5$, and needs the uncertainty of its log for a growth-rate model, $g(\theta)=\log\theta$:

$$\operatorname{SE}(\log\widehat\theta) \approx\frac{1}{100}(5)=0.05.$$

The economist reports that log-scale standard error, trusting the plug-in derivative because the estimate sits well inside a smooth neighborhood, far from zero. A nonsmooth transform, a ratio with a near-zero denominator, or a point where $g'(\theta)=0$ would call for higher-order treatment; transforming a confidence interval endpoint-by-endpoint is the safer fallback when monotonicity is available.

This is a Workflow heuristic: it propagates an existing asymptotic uncertainty calculation through a smooth transformation rather than standing alone.

12.3: Tail Control and Simulation Diagnostics

The final eight rules ask what extra information buys beyond moment-only bounds. Direction, boundedness, variance, sub-Gaussian decay, and a known normal tail each sharpen risk control differently. The section then turns to simulations, where nominal draw count can conceal slow root-$N$ precision or severe weight degeneracy.

12.3.1: Cantelli’s One-Sided Variance Bound

History

Paying for two-sided protection when only one direction of risk actually threatens a system wastes exactly the safety margin that matters. Francesco Paolo Cantelli’s 1928 Italian publication Sui confini della probabilità included the one-sided probability inequality now bearing his name, directing mean-and-variance information toward one tail instead of paying for both.

The directional-risk interpretation below is a direct modern reading of Cantelli’s 1928 result.

The equation

For a random variable with finite mean $\mu$, finite variance $\sigma^2$, and $a>0$,

$$P(X-\mu\ge a) \le \frac{\sigma^2}{\sigma^2+a^2}.$$

When $\sigma>0$, setting $a=k\sigma$ gives $1/(1+k^2)$.

How to read it

Chebyshev bounds the combined event that either tail exceeds a distance. Cantelli uses a shifted squared-moment argument to keep the direction, producing a slightly smaller ceiling for one-sided risk.

The inequality still knows nothing about skewness or tail shape. It is a rigorous worst-case statement, not an estimate of how the present distribution usually behaves.

The corresponding lower-tail statement follows by applying the rule to $-X$:

$$P(X-\mu\le-a) \le\frac{\sigma^2}{\sigma^2+a^2}.$$

Choose one direction before calculating; adding the two one-sided ceilings recreates a valid but usually inferior two-sided statement.

How to use it

A grid operations analyst worried only about overload, not shortage, bounds the chance that load reaches three standard deviations above the historical mean:

$$P(X-\mu\ge3\sigma)\le\frac{1}{1+3^2}=0.10.$$

Two-sided Chebyshev would give the looser $1/9\approx0.111$ after discarding lower-tail coverage the analyst never needed. Cantelli fits because only one direction, overload, threatens the grid, and the population moments are credible; estimated moments carry uncertainty the formula doesn’t show, so the true overload chance could sit well below this bound.

This is a Workflow heuristic: it selects the one-sided member of the moment-bound toolkit after the risk direction has been identified.

12.3.2: Hoeffding Bound for a Bounded Sample Mean

History

Wait for a sample to grow large enough for the central limit theorem to apply, and a decision that cannot wait is left with no finite-sample guarantee at all. Working in Chapel Hill, Wassily Hoeffding published a systematic family of probability inequalities for sums of bounded independent variables in 1963, supplying exponential guarantees without specifying a full distribution.

The sample-mean formula is a direct specialization of Hoeffding’s 1963 paper.

The equation

If independent $X_i\in[a,b]$ with $a<b$, then for $\varepsilon>0$,

$$P\!\left( |\bar X-E\bar X|\ge\varepsilon \right) \le 2\exp\!\left[ -\frac{2n\varepsilon^2}{(b-a)^2} \right].$$

Cap the right side at $1$ when necessary.

How to read it

Independence makes exponential moments multiply, while the hard range limits how large each factor can become. The result shrinks exponentially in $n\varepsilon^2/(b-a)^2$ and does not need a variance estimate or identical distributions.

That robustness costs tightness: observations concentrated in a small part of the allowed range may concentrate much faster.

The inequality can also be inverted. To make the failure probability at most $\alpha$, it is sufficient that

$$n\ge \frac{(b-a)^2}{2\varepsilon^2} \log\!\left(\frac{2}{\alpha}\right).$$

This turns a tail theorem into a conservative sample-size plan.

How to use it

An education researcher scores $1{,}000$ independent student assessments on a $[0,1]$ scale and wants a guarantee on the sample mean within $\varepsilon=0.05$ of the true average score:

$$P(|\bar X-E\bar X|\ge0.05) \le2e^{-2(1000)(0.05)^2} =2e^{-5} \approx0.0135.$$

The researcher trusts this only because the $[0,1]$ range is genuinely honest and every assessment is independent; students sharing one teacher effect, or scores collected only from those who finished early, would invalidate the basic form. If the true score variance is known to be much smaller than the range suggests, Bernstein’s bound gives a tighter guarantee instead.

This is an Independent rule: independence, a hard interval, and sample size directly supply a nonasymptotic mean-error guarantee.

12.3.3: Multiplicative Chernoff Bound for Counts

History

When only an expected count gets reported, without asking how far short a real count could plausibly fall, a system failure can hide as ordinary noise until it’s too late. Herman Chernoff, working at Stanford in 1951 and 1952, developed exponential-moment bounds while studying the efficiency of hypothesis tests based on sums of observations.

The optimization principle is historical. The compact multiplicative formulas for independent Bernoulli counts are standard modern descendants of that documented method.

The equation

For a sum $S$ of independent Bernoulli variables with $E[S]=\mu$ and $\delta>0$,

$$P(S\ge(1+\delta)\mu) \le \left[ \frac{e^\delta}{(1+\delta)^{1+\delta}} \right]^\mu.$$

For $0<\delta<1$,

$$P(S\le(1-\delta)\mu) \le e^{-\mu\delta^2/2}.$$

How to read it

Exponentiating a sum turns it into a product. Markov’s inequality bounds the exponential tail event, and optimizing the exponent produces a decay rate tuned to the requested relative deviation.

Unlike an additive bound, the scale is the expected count $\mu$. A $50$-count shortfall means something different when $\mu=100$ than when $\mu=10{,}000$.

The exponent grows linearly with $\mu$ for a fixed relative deviation, so doubling the expected count squares a bound of the form $e^{-c\mu}$: multiplicative concentration becomes powerful for large, stable counts. The bound is a ceiling, never an estimate; near the mean it is wildly conservative.

How to use it

A data center capacity planner expects $\mu=100$ successful health-check pings per hour across a server fleet; observing $50$ or fewer corresponds to $\delta=0.5$:

$$P(S\le50) \le e^{-100(0.5)^2/2} =e^{-12.5} \approx3.7\times10^{-6}.$$

The planner treats a drop to $50$ as strong evidence against the healthy independent-ping model, given this small upper bound, and investigates an outage or another model failure. The bound alone does not give the probability that an outage occurred; that would also require an outage prior and an alternative model. Dependence between pings is one possible reason the healthy model could fail.

This is an Independent rule: the expected count and a relative deviation directly yield an exponential tail certificate under Bernoulli independence.

12.3.4: Bernstein’s Variance-Aware Tail Bound

History

Choosing between a variance-based bound and a range-based bound turns out to be a false choice: the sharpest guarantee uses both, trading one for the other as the deviation grows. Sergei Bernstein’s 1920s Soviet work struck exactly that compromise, strengthening moment-based bounds with exponential tails for sums of independent variables.

The constants vary among formulations; the equation below is the standard modern bounded-sum form supported by that history.

The equation

If independent variables satisfy, for some $M>0$,

$$|X_i-E[X_i]|\le M$$

almost surely, let

$$V=\sum_i\operatorname{Var}(X_i).$$

Then, for $t>0$,

$$P\!\left(\sum_i(X_i-E[X_i])\ge t\right) \le \exp\!\left[ -\frac{t^2}{2(V+Mt/3)} \right].$$

How to read it

When $t$ is moderate, $V$ dominates the denominator and the exponent looks Gaussian, proportional to $-t^2/V$. For very large $t$, the $Mt$ term prevents a falsely optimistic Gaussian tail.

Bernstein can improve on a range-only Hoeffding bound when actual total variance is small, but it needs that variance information.

The crossover is visible in the denominator. When $t\ll V/M$, the variance term dominates; when $t\gg V/M$, the range term dominates. A two-sided version can be obtained by applying the bound to both the sum and its negative and adding the two tail ceilings.

How to use it

Many independent route legs each contribute delivery-time deviations bounded by $M=1$ hour from their own mean, with total variance $V=25$ across the legs; a logistics network planner sums the centered deviations and wants the chance the network-wide deviation reaches $t=20$ hours:

$$\exp\!\left[ -\frac{400}{2(25+20/3)} \right] \approx0.0018,$$

a tight guarantee the planner trusts only after checking the legs are genuinely independent and that $M=1$ is a real almost-sure bound rather than the worst leg observed so far. Plugging in a sample variance instead of the population value would need an empirical-Bernstein correction not applied here.

This is a Workflow heuristic: it selects a variance-sensitive concentration bound after boundedness and variance information have been verified.

12.3.5: Maximum of Many Sub-Gaussian Errors

History

Why does the largest of many independent measurements grow so slowly as more are added, rather than growing in proportion to how many there are? R. A. Fisher and L. H. C. Tippett’s 1928 Cambridge paper founded limiting theory for sample maxima, explicitly studying normal samples and showing that extremes live on a different scale from typical observations.

The $\sqrt{\log m}$ sub-Gaussian threshold is a modern concentration summary of that 1928 extreme-value phenomenon.

The equation

For $m\ge1$ and $\sigma>0$, if every $X_i$ satisfies, for $t\ge0$, the one-sided tail bound

$$P(X_i\ge t)\le e^{-t^2/(2\sigma^2)},$$

then

$$P\!\left(\max_{1\le i\le m}X_i\ge t\right) \le m e^{-t^2/(2\sigma^2)}.$$

Setting the right side near one suggests

$$t\approx\sigma\sqrt{2\log m}.$$

How to read it

The union bound alone proves the upper probability inequality, so it does not require independence. The square-root-log scale follows by balancing one light tail against $m$ opportunities.

Independence matters when treating this upper scale as typical. Strong positive dependence can make the effective number of distinct opportunities much smaller, while heavy tails change the scale entirely.

For a desired familywise ceiling $\alpha$, solve the union bound for

$$t=\sigma\sqrt{2\log(m/\alpha)}.$$

Then $P(\max_iX_i\ge t)\le\alpha$. This adds the confidence requirement that the order-one scale deliberately omits.

How to use it

A research lab running $m=1{,}000$ parallel assay channels, each with unit-scale normal noise, expects its largest fluctuation near

$$\sqrt{2\log1000}\approx3.72$$

standard deviations by chance alone, not evidence of a real effect. This first-order scale isn’t a calibrated extreme quantile, so the lab still checks channel dependence, tail shape, and whether the flagged channel was chosen only after looking at the data.

This is an Independent rule: a sub-Gaussian marginal tail and the number of comparisons directly set a portable maximum-error scale.

12.3.6: Mills-Ratio Approximation for a Normal Tail

History

Printed normal tables stopped short exactly where a calculation needed the deep tail. John P. Mills tabulated a ratio of tail area to curve height at the cutoff in a 1926 paper, giving small normal-tail calculations a fast, hand-computable shortcut.

The approximation and bracket below are direct modern forms of Mills’s 1926 quantity, valuable when tables or digital computation are unavailable.

The equation

For $z>0$, with standard normal density $\phi$ and CDF $\Phi$,

$$1-\Phi(z)\approx\frac{\phi(z)}{z}.$$

The rigorous bracket is

$$\frac{z}{1+z^2}\phi(z) < 1-\Phi(z) < \frac{\phi(z)}{z}.$$

How to read it

The normal density falls so quickly that the area beyond $z$ is governed largely by its boundary height. Integration by parts shows that the local logarithmic slope $z$ supplies the dividing scale.

The upper approximation becomes relatively accurate far into the tail. Near zero, division by $z$ makes it useless.

At $z=3$, the bracket itself gives

$$\frac{3}{10}\phi(3) < 1-\Phi(3) < \frac13\phi(3),$$

or approximately $0.00133<1-\Phi(3)<0.00148$. The exact tail $0.00135$ sits close to the lower endpoint, illustrating both the guarantee and the approximation’s direction.

For mental use, wait until $z$ is at least roughly two; near the center, use normal tables or software.

How to use it

A floodplain engineer checking the chance rainfall exceeds three standard deviations above normal, $z=3$, gets

$$\frac{\phi(3)}{3} \approx0.00148,$$

close to the exact one-sided tail of about $0.00135$, good enough for a quick design check before running the full survival-function routine used for the final report. But rainfall tails are rarely this tidy: a skewed, non-normal tail can make $\phi(z)/z$ overstate or understate the true exceedance, so the engineer cross-checks the empirical tail before filing.

This is an Independent rule: a positive normal score directly yields a fast tail approximation and rigorous elementary bracket.

12.3.7: Monte Carlo Error Falls as One Over Root N

History

Doubling a simulation’s sample size feels like real progress until the error barely moves, and a budget planned around that mistaken intuition runs out fast. Nicholas Metropolis and Stanislaw Ulam, working at Los Alamos, published The Monte Carlo Method in 1949, describing repeated random sampling made practical by electronic computation.

The root-$N$ precision law is the modern sampling-theory account of the computational method established by that paper.

The equation

For iid draws and an integrand with finite variance $\sigma_f^2$,

$$\widehat\mu =\frac1N\sum_{i=1}^{N}f(X_i), \qquad \operatorname{SE}(\widehat\mu) =\frac{\sigma_f}{\sqrt N}.$$

How to read it

Independent variances add across the sum, while dividing by $N$ squares in the variance. The resulting $1/N$ variance becomes $1/\sqrt N$ standard error.

Halving random error needs four times as many draws. Gaining one decimal digit, tenfold smaller error, needs roughly one hundred times as many.

In practice $\sigma_f$ is unknown and is replaced by the sample standard deviation of $f(X_i)$. That estimates random sampling error only. It does not reveal deterministic coding error, discretization bias, burn-in bias, or a target distribution implemented incorrectly.

Solving the rule for a target standard error $s$ gives the planning equation $N\approx(\sigma_f/s)^2$. A pilot run can estimate $\sigma_f$, after which the budget should be updated rather than guessed.

How to use it

A grid planning team simulating outage risk finds that $10{,}000$ draws give standard error $0.02$; reaching $0.002$ for a regulatory filing requires approximately

$$10{,}000\left(\frac{0.02}{0.002}\right)^2 =1{,}000{,}000$$

draws, a hundredfold increase the team budgets compute time for rather than assuming a modest rerun will do. The team estimates $\sigma_f$ from the pilot run and reports Monte Carlo uncertainty alongside the answer, aware that serial correlation between draws would replace nominal $N$ with a smaller effective sample size.

This is an Independent rule: finite-variance independent sampling directly prices the simulation budget needed for a target random error.

Figure 12.3. For iid Uniform(0,1) draws, the mean has exact standard error sqrt(1/(12N)). The simulated RMSE across 400 independent replications follows that scale with finite-replication noise.

12.3.8: Importance-Weight Effective Sample Size

History

Ten thousand simulation draws can flatter a result that a handful of dominant weights actually produced. Augustine Kong, Jun Liu, and Wing Hung Wong published a sequential-imputation method for Bayesian missing-data problems in 1994, interpreting importance-weight variability through an effective sample size.

The widely used inverse-squared-weight formula comes straight from that 1994 diagnostic; treating it as a universal variance equivalence goes beyond it.

The equation

For proposal draws $X_i\sim q$, let

$$w_i=\frac{\pi(X_i)}{q(X_i)}, \qquad \widetilde w_i=\frac{w_i}{\sum_jw_j}.$$

The diagnostic is

$$\widehat{\operatorname{ESS}}_{\mathrm{IS}} =\frac{1}{\sum_i\widetilde w_i^2}.$$

How to read it

Uniform normalized weights give $\operatorname{ESS}=N$; one dominant weight drives it toward one. The formula measures realized weight concentration and quickly exposes a proposal that sends most computation to nearly irrelevant draws.

It ignores how the integrand aligns with those weights and cannot reveal target regions that the proposal never visited.

Because normalized weights sum to one,

$$1\le\widehat{\operatorname{ESS}}_{\mathrm{IS}}\le N.$$

Multiplying all raw weights by the same positive constant leaves the diagnostic unchanged. These properties make it an easy implementation check, but not a proof of accurate integration.

How to use it

A data science team running an importance-sampling simulation with $100$ draws finds one normalized weight at $0.50$ and the other $99$ each at $0.50/99$:

$$\widehat{\operatorname{ESS}} = \frac{1}{0.50^2+99(0.50/99)^2} \approx3.96,$$

meaning nearly all $100$ nominal draws contribute almost nothing, so the team inspects the maximum weight, replicate estimates, and proposal support before trusting the result. Heavy-tailed weight ratios can make the diagnostic itself unstable.

This is a Workflow heuristic: it audits an importance-sampling stage and decides whether proposal repair or resampling must follow.

Chapter Synthesis: Climb the Uncertainty Ladder Deliberately

Probability shortcuts become reliable when their information requirements are visible.

Start with the weakest defensible statement. Markov needs nonnegativity and a mean. Chebyshev and Cantelli spend a variance. The union bound needs only marginal event probabilities, while Bonferroni spends intersection information. These are guarantees, often conservative, and should be labeled as bounds rather than predictions.

Next recognize scale. Independent noise combines through squared standard deviations. Rare repeated risks accumulate through $np$, collisions emerge near $\sqrt M$, and complete coverage costs $M\log M$. These patterns travel widely because they describe structure rather than one named dataset.

Earn approximations through assumptions. Bayes requires a base rate and likelihood model. Normal, Poisson, exponential, lognormal, and delta approximations require their own shape, rarity, smoothness, or moment conditions. Continuity correction improves a viable approximation; it cannot rescue a bad one. Berry–Esseen asks how much finite-sample confidence the central limit theorem has actually earned.

Finally audit tail and simulation claims. Boundedness, variance, and sub-Gaussian decay buy progressively sharper concentration. Monte Carlo still pays the $1/\sqrt N$ tax, while importance weights can reduce thousands of nominal samples to only a handful of effective contributors.

Across all thirty rules, ask:

1. Is this number exact, an approximation, or a bound?
2. Which assumption, independence, finite moments, boundedness, normality, rarity, or smoothness, licenses it?
3. Is the rule answering the question or checking one stage of a larger workflow?
4. Would dependence, heterogeneity, tail shape, or model drift reverse the decision?

One-Page Probability Toolkit

Decision Path

• Need a guaranteed bound with little distributional information? Start with Markov, Chebyshev, Cantelli, or the union bound according to the available moments and tail direction.
• Know event overlaps? Add the Bonferroni pairwise correction and decide whether the resulting bracket is narrow enough.
• Combining uncertainty? Check covariance before using root-sum-of-squares, and condition explicitly before splitting total variance.
• Facing repeated rare opportunities? Use the exact complement first, then recognize Poisson, birthday, coupon, or exponential scales only in their regimes.
• Updating evidence? Convert the base rate to prior odds and multiply only conditionally independent likelihood ratios.
• Replacing a distribution? Check support, expected successes and failures, skew, moments, and tail location before selecting a normal, Poisson, or exponential approximation.
• Transforming an estimate? Use Jensen for bias direction, the lognormal formula for an exact normal-log model, or the delta method for local uncertainty propagation.
• Need sharper tail control? Ask what extra fact is defensible: direction, a hard bound, total variance, or sub-Gaussian decay.
• Running a simulation? Report root-$N$ Monte Carlo error, then audit correlation, bias, weight concentration, and proposal support.

Transfer Problems

1. Build a reliability bracket

A service has twenty failure modes, each with probability $0.002$. The pairs $(A_1,A_2)$, $(A_1,A_3)$, $(A_2,A_3)$, $(A_4,A_5)$, and $(A_6,A_7)$ have intersection probability $0.0004$ each, and every other pairwise intersection is zero. Give the union-bound ceiling and the two-term Bonferroni lower bound. Explain what higher-order dependence information is still missing and why neither endpoint should automatically be reported as the actual failure probability.

2. Choose and audit an approximation

Let $X\sim\operatorname{Bin}(400,0.02)$. Compare the success–failure screen for a normal approximation with the rare-event case for a Poisson approximation. Select a route for estimating $P(X\ge12)$, state any continuity correction, and identify an exact calculation you would use to audit the result.

3. Price a simulation before trusting it

An importance sampler reports $N=10{,}000$ draws and an iid-style standard error of $0.01$. Its normalized weights have $\sum_i\widetilde w_i^2=0.02$. Compute the weight-based effective sample size, explain why it does not automatically rescale the reported standard error, and list the proposal-support, correlation, and bias checks needed before using the estimate.

Where These Ideas Reappear

• Analysis: Jensen, moment bounds, normal tails, and convergence rates turn qualitative limiting behavior into inequalities.
• Combinatorics: union bounds, collision counts, coupon collection, and Chernoff methods control large discrete families.
• Statistics: Bayes updates, CLT approximations, transformed standard errors, and normal rules become inference tools only after sampling assumptions are checked.
• Numerical methods: Monte Carlo error, importance weights, and rare-event simulation determine computational accuracy.
• Optimization and machine learning: concentration bounds control generalization, stochastic gradients, multiple searches, and randomized algorithms.
• Reliability and engineering: event unions, covariance, one-sided limits, and bounded-sum guarantees turn component information into system risk.
• Information theory: likelihood ratios, sub-Gaussian maxima, and exponential bounds quantify evidence and coding failures.
• Operations research: waiting-time and coverage scales guide inventory, testing, queues, and randomized planning.

Historical Notes and Sources

The profiles separate documented events from modern thresholds and operational language. Total variance, the Poisson-to-normal screen, and the delta method retain explicit historical gaps; the mathematics remains supported by their research notes.

• Chebyshev and the moment-bound lineage: 1867 Des valeurs moyennes and MacTutor’s Chebyshev biography.
• Boole and probability unions: digitized 1854 Laws of Thought and a historical discussion of union bounds.
• Bonferroni’s alternating bounds: bibliographic record of the 1936 monograph.
• International measurement uncertainty: official GUM text and historical foreword and NIST Technical Note 1297.
• Jensen’s convexity inequality: digitized 1906 record and the original journal DOI.
• Bortkiewicz and sparse counts: digitized 1898 book and the Harvard Data Science Review account.
• Von Mises and birthday collisions: ETH Zürich historical note.
• De Moivre and complete collection: Royal Society publication record and a historical coupon-collector review.
• Turing’s odds updates at Bletchley Park: typeset probability paper, scholarly GCHQ commentary, and U.S. National Archives history.
• Laplace’s rule and its caveat: digitized 1814 Essai, its English translation, and a scholarly attribution history.
• De Moivre’s normal approximation: University of York edition and Hald’s historical account.
• Poisson-to-normal historical gap leads: Poisson’s 1837 Recherches and Hald on de Moivre.
• Berry and Esseen’s finite-sample rate: Berry’s 1941 paper and the archive record for Esseen’s work.
• Poisson’s rare-event approximation: digitized 1837 edition and a scholarly translation and introduction.
• McAlister’s multiplicative distribution: 1879 Nature paper and the NIST lognormal reference.
• Delta-method gap lead: Efron’s 1981 standard-error paper, which supports surrounding practice but is not counted as the method’s origin.
• Cantelli’s one-sided inequality: Generali archive record and the Encyclopedia of Mathematics context.
• Hoeffding’s bounded-sum inequalities: 1963 JASA paper and the 1962 technical report.
• Chernoff’s exponential-moment method: 1951 Stanford report and Stanford’s memorial account.
• Bernstein’s variance–range compromise: Encyclopedia of Mathematics history and Hoeffding’s later comparison.
• Fisher and Tippett on sample maxima: 1928 Cambridge paper and its CiNii record.
• Mills and the normal tail ratio: 1926 Biometrika paper and the NIST normal reference.
• Metropolis and Ulam’s Monte Carlo method: 1949 JASA paper and the Los Alamos historical report.
• Kong, Liu, and Wong on importance-weight ESS: 1994 JASA article and the author-hosted derivation.
• Modern mathematical references used by the research notes: MIT OpenCourseWare probability, Penn State STAT 414, Stanford STATS 311 notes, and the NIST probability-distribution handbook.