1Demonstration 1 of 4
Scoring a cycle by point-by-point distance
Can a model with the right period and amplitude score worse than a flat line?
A half-cycle shift puts the model at the opposite phase, so each point is far from the record. The flat line is wrong by the amplitude but never by twice it. shape ignores scale and rewards matching times of rise and fall, but it cannot score a flat line at all.
The observed cycle has mean 50 and amplitude 10 over a 12 month period. The model has the same period and amplitude, shifted by a fraction of a cycle. rmse is the root mean squared distance; shape is one minus the correlation of the two paths.
Predict first. With a half-cycle shift, does rmse score the model better or worse than a flat line at the mean?
Choose an example
Constructed example: a sine cycle defined for this reader to illustrate the chapter's rule on scoring oscillators, scored with the pack's rmse and shape functions.
Calculated values
- Phase shift (cycles)
- 0.500
- Error function
- rmse
- Shifted model
- 14.14
- Flat line at the mean
- 7.07
rmse of the shifted model = 2 x 10 x |sin(90.0 deg)| / sqrt(2) = 20.00 / 1.4142 = 14.14. The flat line is 10 / 1.4142 = 7.07. The flat line scores better than a model with the right period and amplitude.
Use the idea
Score an oscillator on period and amplitude, and report the phase beside them, rather than on distance alone.
Where the conclusion applies
Twelve evenly spaced points over one full period, so the rmse formula is exact. With a few points or a partial period the hand formula would not hold.
Check your understanding: With a quarter-cycle shift and amplitude 10, what is the rmse?
Chapter 35 source: section "The rules, and the reason for each". Demonstration C35-D01.
2Demonstration 2 of 4
How many knobs the record allows, and what they cost
When does the pack refuse a fit, and how fast does a grid get expensive?
Each free knob is a degree of freedom a wrong structure can spend on looking right, so the pack compares three times the knob count with the points available before it runs anything. A full grid multiplies the runs by the step count for every knob added.
A knob is a parameter the fit may move. The rule allows at most one knob per three observations. A grid of 9 steps per knob is searched in 3 passes, so the runs are 9 to the power of the knob count, times 3.
Predict first. With a five-point record, does the pack allow a fit of two knobs?
Choose an example
Constructed example: record lengths defined for this reader from the chapter's tank, with the guard and counts from the pack's grid_fit function.
Calculated values
- Knobs
- 2
- Observations
- 5
- Points required
- 6
- Outcome
- refused
- Runs at 9 steps and 3 passes
- 243
- Time at one second a run (minutes)
- 4.0
Required points = 3 x 2 = 6, against 5. The fit is refused: 2 knobs against 5 observations, because the rule allows at most one knob per three points. Runs = 9^2 x 3 = 81 x 3 = 243, which at one second a run is 243 / 60 = 4.0 minutes.
Use the idea
Count the evaluations and the seconds per run before choosing the number of knobs for a fit.
Where the conclusion applies
One second per model run is a stand-in for a slow model; the tank itself runs in milliseconds. The third knob here moves the starting level, only to count runs, and a real fit would not.
Check your understanding: A record has 8 points. What is the largest number of knobs the rule allows?
Chapter 35 source: section "A grid, on purpose". Demonstration C35-D02.
3Demonstration 3 of 4
The tank, fitted by a grid
What residual does a one-knob grid leave, and how does it depend on steps and passes?
The fit lands on the nearest grid point, so it misses 0.07 by up to half the finest spacing. More steps or more passes shrink the spacing; a leftover residual is expected and not a flaw.
The rate is the fraction of the level leaving each month, searched from 0.01 to 0.2. The record was made with a rate of 0.07. mape is the mean absolute percentage error over the 25 fit months.
Predict first. With 9 steps and 2 refinements, is the fitted rate exactly 0.07?
Choose an example
Constructed example: the chapter's tank and synthetic record; steps and passes varied for this reader, run with the pack's grid_fit function.
Calculated values
- Steps per pass
- 9
- Refinement passes
- 2
- Model runs
- 27
- Fitted rate
- 0.069375
- Fit-window error (mape)
- 0.0050
- Holdout error (mape)
- 0.0085
- Within the 1 percent tolerance
- yes
Runs = 9 x (2 + 1) = 27. Grid spacing: 0.19 / 8 = 0.02375, then 2 x previous / 8 = 0.00594, then 2 x previous / 8 = 0.00148. The fitted rate gives month 1 = 100 + 5 - 0.069375 x 100 = 98.062, against the record's 98.000. Distance from the rate that made the record: 0.069375 - 0.07 = (-0.000625).
Use the idea
Report the residual and the number of runs with every fitted value, so a reader can repeat the search.
Where the conclusion applies
The record is synthetic and noiseless, so a perfect rate exists. A real record has noise, and the best grid point then fits the noise as well.
Check your understanding: With 5 steps over 0.01 to 0.2, what is the first-pass grid spacing?
Chapter 35 source: section "The tank, fitted". Demonstration C35-D03.
4Demonstration 4 of 4
The wrong knob passes the window and fails the holdout
If the search may move a measured quantity, does the fit still look good?
With the rate fixed at the wrong placeholder, a larger inflow can bend the curve through the fit window. It cannot also match the later months, where the mismatch compounds, so the holdout error exceeds its tolerance.
The rate knob is the unmeasured quantity. The arrivals knob is the quantity the ledger already reports as 5. The fit window is months 0 to 24 and the holdout is months 25 to 36. Tolerances: 0.01 on the fit, 0.02 on the holdout. Each search uses 9 steps per pass.
Predict first. If the search moves arrivals instead of the rate, does the fit pass its own 1 percent tolerance?
Choose an example
Constructed example: the chapter's tank and synthetic record, with fits run by the pack's grid_fit and holdout functions.
Calculated values
- Knob moved
- arrivals
- Fitted arrivals
- 7.5781
- Fit-window error (mape)
- 0.0089
- Holdout error (mape)
- 0.0291
- Passes the fit tolerance (0.01)
- yes
- Passes the holdout tolerance (0.02)
- no
Month 36 in closed form: level = a/r + (100 - a/r) x (1 - r)^36 = 7.578/0.1000 + (100 - 75.78) x 0.0225 = 76.33, against the record's 73.52: |76.33 - 73.52| / 73.52 = 0.0381. The fit passes its own window and fails the holdout: the wrong knob moved and the holdout caught it.
Use the idea
Refuse a knob that could be measured, and always score a withheld window in a separate call.
Where the conclusion applies
The holdout catches a fit that worked only where it looked. A wrong structure that happens to track the later months would pass it too.
Check your understanding: If the model ends at 76.3 and the record at 73.5, what is the relative miss?
Chapter 35 source: section "What the fit cannot tell you". Demonstration C35-D04.