Comparing Against Doing Nothing
Every forecast should be measured against the cheapest alternative to having one. The three naive benchmarks, and what it means when the forecast loses.
One quarter's forecast against the naive benchmarks it should beat, same periods
Error 6.0%.
Error 11.2%.
Error 9.0%.
Error 3.7%.
All four were under, and the seasonal naive benchmark beat the forecast. That is a finding about where to invest: the baseline method was not the problem, and the effort belonged in events. One employer's own comparison, not a benchmark.
Every forecast should be measured against what would have happened with no forecast at all. Three naive alternatives serve: last comparable period, the recent average, and last year's same period. If the forecast cannot beat all three, it is not earning its maintenance, and that is worth knowing before another year is spent on it.
The method in “Comparing Against Doing Nothing” is only as useful as the definitions and source records behind it. A team exploring explore the official workflow notes for job costing with accurate hours can add timely work and project context, with transparent settings, limited access and a documented correction route around every consequential use.
Running the comparison takes an afternoon and most functions have never run it.
Teams documenting “Comparing Against Doing Nothing” may use Kaggle time-series learning as an outside reference. Test the method against the real forecast horizon, known events and operational constraints.
The three benchmarks
Naive: the last comparable period's actual. Average: the mean of the last several comparable periods. Seasonal naive: the same period one year ago, scaled to the current level.
- Naive — the last comparable period's actual.
- Average — the mean of the last several comparable periods.
- Seasonal naive — the same period a year ago, scaled to the level.
Each is a one-line calculation and each is a legitimate way an organisation could operate.
Which benchmark is relevant
Depends on the series. For a stable series, the average. For a strongly seasonal one, seasonal naive. For a fast-moving one, naive.
Reporting against the hardest of the three is the honest choice, and reporting against the easiest is how accuracy claims get inflated.
What it means when the forecast wins narrowly
That the series is mostly noise, and that the sophisticated part of the process is adding a point or two. That may be worth it — a point can be worth a great deal in a large operation — but it should be an explicit judgement.
It also means the maintenance burden should be proportionate.
What it means when the forecast loses
Usually one of three things: the method is fitting noise, the inputs are stale, or the horizon is beyond what the data supports.
Losing to a naive benchmark is not a disaster. It is a finding, and the functions that know it are in a much better position than those that do not.
The seasonal naive trap
In operations with strong annual seasonality, seasonal naive is hard to beat and often beats elaborate models. That is a real result, not a failure of the test.
It usually means the right investment is in events and drivers rather than in the baseline method.
Benchmarks by horizon
A forecast can beat naive at twelve weeks and lose at two, which is common: at a short horizon the recent actual contains most of the information.
Reporting a single comparison across horizons hides this, and the finding — that the near-horizon method is redundant — is one of the most useful available.
Benchmarking the human too
Where a planner adjusts the system output, the system output alone is itself a benchmark. Comparing published against system measures whether the judgement is adding anything.
That comparison is the subject of its own note here, and it is the hardest benchmark to run because it is personal.
Running it continuously
Not once. The benchmarks should sit in the error record as extra columns, calculated every period at no cost.
Then the comparison is always current and nobody has to organise a study to answer the question.
What a good result looks like
The forecast beating the best naive benchmark by a clear margin at the horizons where decisions are taken, and perhaps not at the horizons where none are.
That pattern is healthy. It says effort is where the decisions are.
When the benchmark changes
If the series becomes more volatile, naive gets worse and the forecast looks better without having improved. If the operation stabilises, the reverse.
Which is why the benchmark is reported alongside rather than used to set a fixed target.
The uncomfortable version
Occasionally the finding is that a large forecasting investment is beating a one-line average by half a point. That conversation is difficult and it is better to have it from the record than to avoid it for three more years.
In practice the usual outcome is not abolition but redirection: the effort moves from the baseline method to the event calendar and the driver register, where it pays much better.
The benchmark for a new forecast
Where there was no forecast before, the benchmark is whatever people were doing instead — a rule of thumb, last year plus a bit, a manager's judgement.
Reconstructing it for a few past periods is usually possible and it is a far more persuasive comparison than any statistical baseline.
Why it belongs in the pack
Because it is the only thing that lets a reader judge the forecast without being a forecaster. One extra column, and the question "is this any good" becomes answerable by anybody.