Choosing a Method by Testing It
A short procedure for choosing between forecasting methods on your own data, the mistakes that invalidate the test, and what to do with the result.
A forecasting method should be chosen by testing candidates on your own data, at your own horizon, against the error measure your decision cares about. This takes about a day, it is decisive, and it is skipped almost universally in favour of reading about methods.
The method in “Choosing a Method by Testing It” is only as useful as the definitions and source records behind it. A team exploring review this official Monitask page for employee monitoring software with screenshots can add timely work and project context, with transparent settings, limited access and a documented correction route around every consequential use.
The reason it is skipped is that the test can be set up wrongly in four ways, and getting it right requires thinking about what the forecast is for — which is uncomfortable if nobody has.
Teams documenting “Choosing a Method by Testing It” may use pandas data-analysis documentation as an outside reference. Test the method against the real forecast horizon, known events and operational constraints.
How the comparison is run
- Choose the horizon, from the decision.
- Choose the error measure, from the cost of being wrong.
- Hold back several periods at several points in the series.
- Forecast each held-back period using only prior information.
- Compare candidates, including a weighted average.
Five steps. The fourth is where tests go wrong.
Only prior information
The forecast for a period four weeks out must use only what would have been known four weeks before. Not the actual driver values. Not a model fitted on the whole series. Not a seasonal factor estimated using the period being forecast.
Violating this produces results that cannot be reproduced in production, and it is the commonest flaw in in-house comparisons.
Several hold-back points, not one
A single hold-back window measures how a method did in one particular quarter. Five windows at different points measures the method.
Methods often look excellent in one window and poor in another, and that variation is itself a finding about stability.
Choosing the error measure
From the decision. Where over and under cost differently, a symmetric measure hides the thing that matters. Where the decision is about a total, measure the total; where it is about peaks, measure the peaks.
A method chosen on mean absolute percentage error and used for a peak-staffing decision has been chosen on the wrong criterion.
Always include the simple baseline
A weighted average of comparable periods. It is the reference point that makes every other number interpretable.
"Eight per cent error" means nothing. "Eight per cent against a baseline of eleven" means something.
Include doing nothing
Where relevant: last year's actual, or the current period's level carried forward. If a method cannot beat those, it is not earning its maintenance.
This comparison is unflattering often enough that it is worth running first.
Testing at the right granularity
If the decision uses weekly figures, test weekly. A method that is excellent on monthly totals and poor on weekly distribution will look good in a monthly test and fail in use.
The test should produce exactly the output the decision consumes.
What to do with a close result
Choose the simpler one. Where two methods are within a point of each other out of sample, the difference is not reliably real, and the tiebreaker is maintenance and explicability.
This resolves most comparisons, because most comparisons are close.
Recording the test
The candidates, the windows, the measure, the results, and the date. One page.
It is what prevents the same comparison being re-argued in a year, and it is the evidence when somebody proposes replacing the method with something fashionable.
Re-testing
Annually, or after any structural change. A method that was best three years ago on a different operation is not necessarily best now.
The re-test is cheap once the harness exists, which is a good reason to build the harness rather than doing the comparison by hand.
What the test will not tell you
Whether the event calendar is complete, whether the series is clean, or whether the horizon matches the decision. Those dominate accuracy and no method comparison touches them.
A method test run before that work is done compares several methods operating on the same bad inputs, and the winner will not stay the winner once the inputs improve.
Testing the whole pipeline, not the method
The comparison should include the cleaning, the event handling and the adjustment rules, because those are what will run in production.
A method tested on a hand-prepared series and deployed against an automated one is not the thing that was tested.
The result that depends on the window
Where one method wins in three windows and loses in two, the honest conclusion is that they are not reliably different.
Picking the winner anyway, which is the usual outcome, means adopting a method on the strength of noise — the exact failure the test was meant to avoid.
The order that works
Fix the series. Build the calendar. Set the horizon from the decision. Then test methods. Run the other way round, which is the usual order, and the test has to be repeated anyway.