Demonstration 1 of 4
Start with what happened to comparable projects
Where does a twelve month plan sit among the outcomes of projects that were also planned at twelve months?
Each bar counts past projects by how long they took. The dashed line is the plan made from the inside; the solid line is the median or the 80th percentile of the class. The outside view starts from the bars, not the plan.
Scroll sideways for the whole equation
P is the planned duration (12 months), r the ratio of actual to planned duration for one past project, and D the completion time that ratio implies.
Predict first. Among all 1000 constructed projects planned at twelve months, will most finish within the plan?
Choose an example
Scroll sideways for the whole figure
Constructed data: notebook 25 cell 1, 1000 seeded lognormal overrun ratios (mean log 0.25, spread 0.45) times a 12 month plan.
Calculated values
- Projects in the class
- 1000
- Median
- 15.36 months
- Ratio to the 12 month plan
- 1.280
- Projects longer than the plan
- 697 of 1000
Among the first 1000 comparable projects, 697 took longer than 12 months. The median is 15.36 months, so 15.36 / 12 = 1.280 times the plan. The long right tail is the part a plan made from the inside never imagines.
Worked steps
- Count projects over 12 months: 697 of 1000.
- Median: 15.36 months.
- Ratio to plan: 15.36 / 12 = 1.280.
Use the idea
Before committing to a date, list what comparable past efforts actually took, including your own, and read your plan against that list.
Where the conclusion applies
The ratios are seeded lognormal draws (seed 20260943) built to be right skewed; they are not Flyvbjerg's project database, and the share over plan here says nothing about any real class of projects.
Check your understanding: Your last five projects of this scope took 4, 6, 7, 9 and 18 months. What is the median, and how does a plan of 3 months compare?
Chapter 25 source: section "Section Four: Reference Class Forecasting".
Demonstration 2 of 4
The class you choose changes the forecast
What happens to the outside view if the long projects are quietly left out of the class?
Grey bars are the whole class; teal bars are the projects kept. Any rule that removes projects because they ran long pulls every quantile toward the plan, the median included.
Scroll sideways for the whole equation
P is the 12 month plan and r a past project's ratio of actual to planned time; D is its completion time. The selected class keeps only projects below a cutoff chosen after seeing their durations.
Predict first. Dropping the projects of 24 months or more removes 156 of 1000. How much does the median fall?
Choose an example
Scroll sideways for the whole figure
Constructed data: notebook 25 cells 1 and 3, the 1000 seeded durations and the notebook's 'exclude long projects' estimate, with an 18 month cutoff added.
Calculated values
- Projects kept
- 844 of 1000
- Full class median
- 15.36 months
- Selected class median
- 14.00 months
- Optimism from the selection
- 1.36 months
With the class without projects of 24 months or more, the median is 14.00 months against 15.36 for all 1000: 15.36 - 14.00 = 1.36. Dropping the long projects because of how they turned out moves the median 1.36 months toward the plan: a class chosen after seeing outcomes favours the answer you wanted.
Worked steps
- Projects kept: 844 of 1000.
- Full class median: 15.36.
- Selected class median: 14.00.
- Difference: 15.36 - 14.00 = 1.36 months.
Use the idea
Write down the class definition (size, kind, period, setting) before you look at the outcomes, and keep the overruns and the abandoned efforts in it.
Where the conclusion applies
The cutoffs are chosen to show outcome based selection, the way motivated reasoning would; they are not a recommended class. The durations are the notebook's seeded constructed values.
Check your understanding: A full class has median 20 months; after dropping the worst overruns the median is 17. How optimistic is the selected forecast?
Chapter 25 source: section "Section Four: Reference Class Forecasting".
Demonstration 3 of 4
Unfinished projects belong in the class
What should the outside view do with projects that have not finished yet?
The curve falls each time a project finishes. Dropping unfinished projects pretends they never existed; counting them as finished at their elapsed time pretends they stopped; Kaplan-Meier counts them as running until they leave observation.
Scroll sideways for the whole equation
r is the ratio of time taken to time planned. At each finishing ratio, d is the number of projects that finish there and n the number still running just before it; S-hat is the estimated share still running.
Predict first. Will dropping the 14 unfinished projects make the median forecast shorter or longer than the Kaplan-Meier one?
Choose an example
Scroll sideways for the whole figure
Constructed data: the companion's seeded 60 case example for this chapter (46 finished, 14 still running at 2.0 x plan), analysed with the companion's Kaplan-Meier tool as in the notebook's workshop cell.
Calculated values
- Projects used
- 60
- Median ratio
- 1.298
- Median forecast for a 12 month plan
- 15.58 months
- 80th percentile ratio
- undefined (the curve never falls below 0.2)
By treating unfinished projects as still running (Kaplan-Meier), the median ratio is 1.298, so a 12 month plan becomes 12 x 1.298 = 15.58 months. Kaplan-Meier uses an unfinished project only while it is known to be running. Because all 14 sit at 2.0 x plan, the curve stops above 0.2 and the 80th percentile cannot be estimated, which is reported, not invented.
Worked steps
- Median ratio: 1.298.
- Forecast: 12 x 1.298 = 15.58 months.
- 80th percentile ratio: undefined (the curve never falls below 0.2).
Use the idea
Keep every started project in the class with its elapsed time and a finished flag, and report any percentile the data cannot reach as not estimable.
Where the conclusion applies
Kaplan-Meier assumes unfinished projects are not systematically the worst ones (independent censoring); abandoned projects need a separate outcome. The 60 cases are the companion's seeded example, not real projects.
Check your understanding: An unfinished project has been running for 20 months on a 12 month plan. What do you know about its ratio?
Chapter 25 source: section "Section Five: The Hardest Reference Class".
Demonstration 4 of 4
A commitment needs a chosen risk level
How much longer than the plan should you promise, if you want a given chance of keeping the promise?
The curve is the notebook's: for each chance of finishing on time, the contingency above plan that the class supports. It steepens toward the top because the tail is long.
Scroll sideways for the whole equation
P is the plan in months, q the chance of finishing on time you want, Q of q the q quantile of the class's overrun ratios, and D-hat the commitment.
Predict first. Going from a 50 percent to a 90 percent chance of finishing a 12 month plan on time, will the commitment grow by more than 10 months?
Choose an example
Scroll sideways for the whole figure
Constructed data: notebook 25 cells 1 and 4, the seeded overrun ratios and the empirical quantile curve of contingency against completion probability.
Calculated values
- Percentile
- 80
- Ratio at that percentile
- 1.852
- Commitment
- 22.22 months
- Contingency above plan
- 85.2 percent
For a 12 month plan and a 80 percent chance of finishing on time, commit to 12 x 1.852 = 22.22 months, a contingency of (1.852 - 1) x 100 = 85.2 percent. Each step up in certainty costs more contingency; which level to choose depends on what an overrun costs.
Worked steps
- Ratio at the 80th percentile: 1.852.
- Commitment: 12 x 1.852 = 22.22 months.
- Contingency: (1.852 - 1) x 100 = 85.2 percent.
Use the idea
Decide what an overrun costs before choosing the percentile, then promise the plan times that percentile's ratio, not the median.
Where the conclusion applies
The ratios are seeded constructed values. The chapter says the median suits absolute error, while budget or deadline commitments may need a higher quantile set by the cost of overrunning; no level is right in general.
Check your understanding: The median overrun ratio of a class is 1.4. What is the outside view median for a new 10 month plan?
Chapter 25 source: section "Section Seven: The Full Methodology".