Measuring the Error at All
Most forecasting functions do not keep an error record. What one contains, why it has to be built at the time, and what becomes possible once it exists.
One week forecast five times, at five distances, against what happened
Under by a wide margin; no calendar yet.
Under; the release was not known.
Under; release now in the calendar but undersized.
Under; close.
Under; effectively right.
All five vintages of the same week were under, and the forecast converged only in the last month. A record kept by horizon shows this; one kept as a single accuracy figure cannot. One employer's own record, not a benchmark.
Most forecasting functions cannot say how wrong they were last quarter, because nobody wrote down what the forecast said before the outturn arrived. The record cannot be reconstructed afterwards: by then the file has been overwritten, the version is ambiguous, and the horizon of each figure is lost.
The error measure in “Measuring the Error at All” needs both the original forecast and a trustworthy record of what actually happened. Teams researching project time recording can use see the available reporting options to strengthen the actual-work side of that comparison without treating activity data as proof of demand, performance or intent.
Keeping it costs a row per period. Not keeping it makes every argument about forecasting unresolvable.
Teams documenting “Measuring the Error at All” may use Qlik analytics resources as an outside reference. Test the method against the real forecast horizon, known events and operational constraints.
What a row contains
The date the forecast was made. The period it was about. The horizon between them. The point. The band. The outturn. Whether the outturn was a measured or a filled value. And one line on anything known to have been unusual.
| Field | Example |
|---|---|
| Made on | 14 March |
| Period | week of 6 June |
| Horizon | 12 weeks |
| Point / band | 126 / ±23 |
| Outturn | 141, measured |
| Note | release on the 14th, not in the calendar |
Eight fields. A spreadsheet.
Why the horizon field matters
Because a twelve-week-ahead forecast and a two-week-ahead forecast of the same week are different claims, and averaging their errors together produces a number describing neither.
Without this field the record cannot answer the one question it exists for: does accuracy degrade with distance, and how fast.
Why it must be written at the time
Because the forecast changes. A rolling process produces a new figure for the same week every cycle, and after the event there is no way to say which one was the twelve-week view.
A dated file per cycle, kept unchanged, is the whole apparatus.
Vintages
The same period forecast at several distances is a vintage series, and reading it down the column is extremely informative: it shows when the forecast started to converge on the truth.
A period where the forecast only converged in the final fortnight is a period the long horizon could not see, and that is worth knowing.
What to measure the forecast against
The actual outturn, not a cleaned or adjusted version, except where the actual is a known data failure. Measuring against a cleaned series flatters the forecast by the amount of the cleaning.
Where a day was filled, exclude it rather than scoring against an estimate.
Excluding driver failures
A forecast that was right about everything except that a campaign was cancelled has a different kind of error from one whose method is wrong.
A column for "driver known wrong" lets both be counted and separated. Without it, cancelled campaigns make the method look bad and nobody fixes the confirmation step that would have caught them.
How many periods before it says anything
Twelve to twenty for a stable read on the size of the error. Fewer for direction, which shows up in four or five.
This is why the record is started before anybody asks for it: by the time somebody asks, a year of it is needed.
What becomes possible once it exists
Saying whether the forecast is good enough for the decision. Comparing methods honestly. Detecting bias. Detecting degradation. Pricing the value of improvement. Defending the forecast when it is questioned.
None of these is possible without it and all of them are routine with it.
Where it should live
Next to the forecast, visible to the people who read the forecast. A record kept privately by the planner is a working file; one published with the forecast changes the conversation.
Publishing it also removes the suspicion that it is being curated.
Publishing it is uncomfortable at first
Because the first honest number is usually worse than people assumed, and because nobody has had to say one before.
The thing that makes it survivable is publishing the baseline alongside: eight per cent against a naive eleven is a defensible position, and eight per cent alone is not.
What not to do with it
Use it to appraise the planner. The moment accuracy becomes a personal measure, the forecast starts being managed, and the record stops being information.
The record is about the method and the inputs. The appraisal belongs elsewhere.
The record that covers more than the forecast
Worth extending by two columns: what the naive baseline would have said, and what the system said before adjustment.
Both are free to calculate and both turn the record from a score into a diagnostic. Without them the record says how wrong, and with them it starts to say why.
Starting one from nothing
From the next cycle. Save the file as issued, with a date. Add a row when the outturn arrives.
In three months there is something to read; in a year there is something to act on. There is no shortcut and no way to build it retrospectively, which is why it is worth starting on an ordinary Tuesday rather than waiting for a project.