Percentage Error and What It Hides
The standard accuracy measures each hide something different. Which to use for which decision, and why a single headline accuracy figure is usually misleading.
Every accuracy measure throws something away, and which thing it throws away decides whether it is the right measure for your decision. A single headline figure — "the forecast is 93% accurate" — has thrown away the direction, the distribution and the horizon, which are the three things a decision needs.
The error measure in “Percentage Error and What It Hides” needs both the original forecast and a trustworthy record of what actually happened. Teams researching billable and non-billable hours can use how should teams assess billable and non-billable hours? to strengthen the actual-work side of that comparison without treating activity data as proof of demand, performance or intent.
Choosing the measure is part of defining the forecast, not a technical afterthought.
As a separate benchmark for “Percentage Error and What It Hides”, see SAP Analytics Cloud resources. Keep source data, model, override and resulting decision visible as distinct steps.
Mean absolute percentage error
The default, and reasonable for comparing methods on the same series. It throws away direction entirely: a run of four under-forecasts and a mixture of over and under produce the same figure.
- Four periods under by 6, 5, 7 and 6 per cent: absolute error 6%.
- Four periods out by +9, −4, −7 and +6: absolute error 6.5%.
- The first is a correctable bias. The second is noise. The headline measure calls them the same thing.
It also behaves badly on small values, where a miss of two on a base of six is 33%.
Mean error, signed
The one almost nobody reports and the one that matters most. It is the bias: the average amount by which the forecast is above or below.
A mean error near zero with a large absolute error is noise. A mean error of −6% is a systematic problem that can be corrected.
Reporting both
The practical minimum: absolute error for size, signed mean error for direction. Two numbers, and together they say almost everything.
One without the other is misleading in opposite ways.
Weighted versus unweighted
Averaging percentage errors across periods treats a quiet week and a peak week equally. Where the decision is about total resource, weight by volume; where it is about service on each day, do not.
The two can differ substantially in an operation with pronounced peaks.
Absolute error in units
Underused. "Out by 14 contacts a day on average" is often more useful to an operations manager than "out by 11%", because it converts directly into people.
For decisions about headcount it is the better measure.
Accuracy against a baseline
The only form in which an accuracy figure is interpretable. Eleven per cent error means nothing on its own; eleven per cent where a naive forecast gives sixteen means the method is adding something.
Where the baseline is not reported, the figure cannot be judged, and most reporting omits it.
By horizon
Always separately. A combined figure across horizons is dominated by whichever horizon has the most observations, which is usually the nearest and easiest.
Three figures for three horizons; one figure for all of them is actively misleading.
Peak error versus average error
Different decisions. A method that is excellent on average and misses peaks is a problem for staffing and fine for budgeting.
Where peaks drive the cost, measure the peaks: the error on the highest decile of days is a separate and more relevant number.
Error on the total versus error on the shape
A week forecast correctly in total and wrongly distributed across days scores well on a weekly measure and causes real operational trouble.
Where distribution matters, measure it: the error of the daily split given the weekly total, reported separately.
The measure that gets gamed
Any measure used to appraise the person producing the forecast. Once that happens the measure improves and the forecast does not.
The defence is to measure the method and the inputs, publish the baseline, and keep the appraisal conversation somewhere else entirely.
The four lines to report
Four lines. Mean absolute error with the baseline beside it. Signed mean error. Both by horizon. And the number of periods behind the figures.
That last one prevents a three-period record being read as a verdict, which happens constantly.
Error in hours rather than contacts
The conversion that matters for staffing. A ten per cent volume error and a ten per cent handling time error compound, and the resulting hours error can be twenty.
Where the forecast is a volume and the decision is about people, the error should be reported after conversion, not before.
The measure for a skewed series
Where most periods are small and a few are large, averaging percentage errors gives the small periods most of the weight.
Reporting both the unweighted percentage and the total absolute quantity resolves it, and the gap between them is itself informative about where the error sits.
Reporting a distribution rather than a mean
Worth doing once a year: the full spread of errors, not the average. It shows the tail, which is where the operational pain lives, and it usually looks worse than the headline figure implies.
That is the point. The headline was hiding it.
The figure that should never stand alone
"Accuracy: 93%." It conceals direction, horizon, baseline, sample size and the definition of accuracy being used. It is the figure most likely to appear in a presentation and the one that carries the least information of any number in this collection.