Cleaning a Series Without Inventing One
The order in which to clean a demand series, what to leave alone, and the rule that separates legitimate cleaning from quietly fabricating history.
Cleaning a series means removing things that were never demand and labelling things that were unusual. It does not mean making the series smooth. The line between the two is the difference between a usable history and a fabricated one, and it is crossed by degrees.
The series discussed in “Cleaning a Series Without Inventing One” is easier to interpret when changes in work capture are visible rather than silently folded into the history. Used for how to monitor employees without being intrusive, view the reporting setup here can provide consistent time and project evidence, provided corrections, category changes and missing periods remain traceable.
The rule that holds: every change to a value is recorded, and anybody can reconstruct the raw series from the record.
Teams documenting “Cleaning a Series Without Inventing One” may use official Scrum Guide as an outside reference. Test the method against the real forecast horizon, known events and operational constraints.
The order
- Fix the calendar: mark closed days, part days and known outages.
- Resolve the gaps, with flags.
- Identify definition changes and decide the usable start.
- List the extremes with causes and decide each one.
- Only then look at the shape.
Working out of order produces decisions that have to be unmade.
Why the calendar comes first
Because most apparent anomalies turn out to be calendar facts, and resolving them removes perhaps half of what looked like a data problem.
It is also the cheapest step: the operating calendar exists and nobody has joined it to the demand series.
What cleaning is not
Smoothing. A moving average applied to the stored series destroys information permanently and makes the variance look smaller than it is.
Smoothing belongs inside a method, applied at the point of forecasting, not in the history.
The raw series is kept
Always, unchanged, alongside the cleaned one. Storage is trivial and the ability to go back is not.
Where only the cleaned series survives, every cleaning decision ever made becomes permanent and unexaminable.
The change log
One row per change: date or range, what was changed, from what to what, why, who. The same file that holds definition changes.
A cleaned series with no log is an assertion. With one, it is an analysis anybody can check.
Cleaning that should not be done
Removing days because they are inconvenient. Adjusting a period to match a report. Replacing a real low with a plausible value because somebody doubted it.
Each of these is defensible in the moment and each makes the series describe what people expected rather than what happened.
The series that gets cleaned twice
A real hazard where cleaning happens in the extract and again in the model. The same outage gets filled, then the filled value gets flagged as an outlier and replaced again.
A single cleaning stage with a single owner prevents it.
Judging whether cleaning helped
By the error record, not by how the chart looks. A cleaned series that produces no improvement in out-of-sample error was cleaned for aesthetic reasons.
That test takes an hour and it is rarely run, because by the time the cleaning is done nobody wants to find out it was unnecessary.
How much cleaning is normal
Less than people expect. In a reasonably maintained operation, a two- year daily series typically has a handful of outage days, one or two definition changes and ten to fifteen labelled extremes.
Where cleaning is touching a large share of the values, the problem is in the collection and no amount of post-processing will fix it.
Cleaning and the error record
The record should measure the forecast against the actual outturn, not against the cleaned value, except where the actual is known to be a data failure.
Measuring against a cleaned series flatters the forecast by exactly the amount of the cleaning.
When to stop
When the remaining anomalies have causes that are real and recurrent. Those are events to be modelled, not noise to be removed, and the cleaning stage is finished.
Continuing past that point is where invention starts.
Who should do it
Whoever will use the series, with whoever knows the operation. Not a data team alone: the questions are almost all operational, and the answers live with people who were there on the day.
An afternoon with the operations manager and a plotted series resolves more than a week of statistical testing.
Cleaning that should be automated
The mechanical parts: joining the calendar, flagging zeros on open days, flagging incomplete recent periods. These run every cycle and should not be anybody's manual task.
The judgement parts — what caused an extreme, whether a step is a definition change — should never be automated, because the inputs to them are conversations.
Who checks the cleaner
Worth saying: the cleaning code is itself a source of error, and a change to it alters the history retrospectively.
Versioning it alongside the series, and re-running the error record when it changes, is the only way to keep the record comparable across a cleaning change.
The output
A usable series with a stated start date, a flag column, and a change log. Three artefacts, all small, and together they are what makes every forecast built afterwards assessable rather than merely produced.