1Demonstration 1 of 4
Rank uncertainties by their swing
Which uncertain quantity moves the decision metric most?
A wide range moves the metric more than a narrow one, so a quantity can fall down the ranking once its range is pinned down, whatever its importance in the model.
The decision metric is adopters at week 20 from a market of 1000. The swing of one quantity is the absolute change in the metric from the low to the high end of its range, with the others at midpoint.
Predict first. With the market range narrowed to 950 to 1050, does market size still rank first?
Choose an example
Constructed example: the chapter's three uncertainties and ranges, with narrower ranges defined for this reader, ranked by the pack's one_at_a_time function.
Calculated values
- Ranked first
- Market size
- Swing, market size
- 572.1
- Swing, imitation
- 526.3
- Swing, innovation
- 127.2
- Ranges
- market 700 to 1300, imitation 0.1 to 0.5, innovation 0.005 to 0.02
Each swing runs the model at the low and high end of one range with the others at their midpoints: Market size |1240.9 - 668.8| = 572.1; Imitation |999.6 - 473.3| = 526.3; Innovation |978.8 - 851.6| = 127.2. The largest is market size.
Use the idea
Rank by swing before commissioning measurement, so effort goes where the decision moves.
Where the conclusion applies
One-at-a-time swings with the others at midpoint. The method misses interactions, and the ranges are defined for this reader, not measured.
Check your understanding: If the market range is 950 to 1050 and imitation is 0.1 to 0.5, which is ranked first?
Chapter 29 source: section "Ranking by effect". Demonstration C29-D01.
2Demonstration 2 of 4
Rank by what it costs to find out
Does the biggest uncertainty deserve the first measurement?
Dividing by cost can reverse the order. An expensive study of a large swing can lose to a cheap estimate of a small one. If the large one becomes cheap enough, it moves back to first.
Value per cost is the swing divided by the cost to reduce the uncertainty. Costs are in arbitrary units: imitation costs 5, and the market and innovation costs are the controls.
Predict first. At the chapter's costs of 20, 5 and 1, which uncertainty comes first once cost is included?
Choose an example
Constructed example: the chapter's ranges and costs 20, 5 and 1, with other costs defined for this reader, computed by the pack's value_per_cost function.
Calculated values
- First by effect
- Market size
- First by effect per cost
- Innovation
- Order reversed
- yes
- Per cost, innovation
- 127.2
- Per cost, imitation
- 105.3
- Per cost, market size
- 28.6
Effect divided by cost: Innovation 127.2 / 1 = 127.2; Imitation 526.3 / 5 = 105.3; Market size 572.1 / 20 = 28.6. Per cost, innovation comes first instead of market size, so the order reverses.
Use the idea
Estimate the cost of reducing each of the top uncertainties before choosing which to study.
Where the conclusion applies
Costs are single numbers in one unit, and measurement removes the uncertainty entirely. Real studies shrink a range only partly.
Check your understanding: If measuring market size cost 5 and innovation cost 4, what is market size's value per cost?
Chapter 29 source: section "Ranking by what it costs to find out". Demonstration C29-D02.
3Demonstration 3 of 4
One at a time against a joint sample
Does a sample across all ranges at once reveal more than the individual swings?
For an additive f(x, y) = x + y on [0, 1] each swing is one and the joint range is two. A wider joint spread flags outcomes the swings did not cover, but a small sample can miss the extremes.
The two uncertainties are imitation (0.1 to 0.5) and innovation (0.005 to 0.02). Each draw picks both uniformly at random from their ranges. The seed fixes the draws.
Predict first. Is the joint spread of a five-draw sample wider than the largest single swing?
Choose an example
Constructed example: the chapter's imitation and innovation ranges on its model, sampled with the pack's sample function; the additive example is the chapter's own.
Calculated values
- Draws and seed
- 20 draws, seed 1
- Imitation swing
- 526.3
- Innovation swing
- 127.2
- Lowest sampled
- 358.3
- Highest sampled
- 998.9
- Joint spread
- 640.6
Joint spread = 998.9 - 358.3 = 640.6, against the largest single swing 526.3, a difference of 640.6 - 526.3 = 114.3. The joint spread is wider than any single swing. That flags outcomes the swings did not cover; it does not prove an interaction.
Use the idea
Follow a ranking with a few dozen joint draws as a screen, and record the region sampled.
Where the conclusion applies
Uniform draws, two uncertainties, and a fixed seed. A similar spread does not prove independence.
Check your understanding: For f(x, y) = x + y on [0, 1] for each, what are each swing and the joint range?
Chapter 29 source: section "Interactions, and where one-at-a-time misleads". Demonstration C29-D03.
4Demonstration 4 of 4
Endpoints can miss a reversal
If one policy wins at both ends of a range, does it win throughout?
B scores 0 at both endpoints and 1 at the middle, so a check of the endpoints finds A ahead while an interior value reverses the order. Without a property such as monotonicity, test interior points.
p is an uncertain quantity between 0 and 1. Policy A scores a constant. Policy B scores 1 - (2p - 1)^2. Larger scores are preferred.
Predict first. Where A scores 0.2, which policy wins at p = 0.5, though A wins at both endpoints?
Choose an example
Constructed example: the chapter's own scores, A at 0.2 and B at 1 - (2p - 1)^2, with A at 0.75 added by this reader to show a tie.
Calculated values
- A score
- 0.20
- B score at the point checked
- 1.00
- Winner at the point checked
- B wins
- B score at p = 0 and p = 1
- 0.00
- Winner at both endpoints
- A
At p = 0.50: B = 1 - 0.00 x 0.00 = 1.00, against A = 0.20. B beats A at p = 0.50. At p = 0 and p = 1, B = 1 - (-1) x (-1) = 0, so A wins at both endpoints; checking only the endpoints would stop the search for a reversal too early.
Use the idea
Compare policies at interior values and near thresholds before stopping the search for a reversal.
Where the conclusion applies
Two policies and one uncertain quantity. This is the chapter's own constructed example, plain arithmetic rather than a pack function, and finite sampling is never a proof.
Check your understanding: If A scores 0.75, which policy wins at p = 0.25?
Chapter 29 source: section "When to stop experimenting". Demonstration C29-D04.