Testing a Forecast as Time Moves Forward

Botie's forecast looks convincing on a plot of the whole record. But the model was fitted after all those observations existed. We need to reconstruct the decisions he could have made at earlier dates.

Move a cutoff through the record. At each cutoff, fit or update using only the available past, save the requested forecast, and score it against the later target. This makes the evaluation follow the intended use.

Both methods made their predictions before the same target days. Compare their errors date by date before reducing each method to one average.

An expanding window retains all earlier training observations. A sliding window keeps a fixed recent span. The first can estimate from more data; the second can respond to a changed process by discarding older conditions.

Specify the forecast horizon, minimum history, update frequency, and training window before comparing scores. Different choices represent different deployment tasks.

Origin 3: history positions [0,1,2], target position 3
Origin 4: history positions [0,1,2,3], target position 4

Array indices here identify chronological positions, not shuffled rows.

Any learned scaling, imputation, feature selection, or model coefficients must be learned using that origin's training data. Fitting preprocessing once on the complete sequence still leaks future information even if the model's rows are split correctly.

When labels or measurements arrive late, move the usable cutoff back accordingly. The replay should match actual availability rather than an idealised instant report.

A backtest evaluates the complete prediction procedure, including how often it is allowed to update.

Calculate an error for each saved forecast and its target. Report overall summaries and inspect errors across time. Overlapping forecasts and shared training histories can make those errors dependent.

If a method predicts several horizons, report performance by horizon as well as any chosen aggregate. A good next-day forecast need not be a good next-month forecast.

Targets = [9,8]
Persistence errors = [3,1]; MAE = 2
Trailing-two errors = [2.5,0.5]; MAE = 1.5

This tiny comparison illustrates bookkeeping. It is far too small to establish reliable superiority.

import numpy as np
history = np.array([4., 6., 8.])
predictions = np.array([history[-1], history[-2:].mean()])
target = 10.0
print(predictions.tolist())
print(np.abs(target - predictions).tolist())

[8.0, 7.0]
[2.0, 3.0]

Use rolling validation to select settings, then evaluate the chosen procedure on a final untouched period. Repeatedly adjusting a model after inspecting that period turns it into another validation set.

Botie's workshop compares aligned absolute errors. The coding task creates those predictions from the historical record itself, then a second task checks performance separately by horizon.

A deployment project later adds monitoring because the process can change after even a careful backtest.

If daily demand is finalised the following afternoon, tomorrow morning's forecast cannot use yesterday's final total yet. Replaying the task requires both event dates and reporting dates.

At each origin, build the available history from records that had arrived by that time. A later corrected reading can be useful for scoring, but should not replace the version actually available to the historical predictor.

An expanding window retains all past observations. A fixed recent window forgets older regimes. Neither is universally preferable: the first reduces estimation noise under stability, while the second can respond faster to change.

Choose the window rule using validation origins. Compare predictions on the same targets, including dates after a plausible operating change.

import numpy as np
y = np.array([5.,5.,5.,10.,10.,10.])
for origin in [3,4,5]:
    expanding = y[:origin].mean()
    recent = y[origin-2:origin].mean()
    print(origin, round(float(expanding), 2), round(float(recent), 2), float(y[origin]))

3 5.0 5.0 10.0
4 6.25 7.5 10.0
5 7.0 10.0 10.0

Keep each method's error beside the same origin and horizon. The difference between their errors tells us on which decisions one helped. An overall mean can hide a method that improves ordinary days but fails on the peak days Botie most cares about.

Origins and horizons can overlap, so errors are not generally independent. A simple standard error treating every cell as an unrelated observation can be misleading. Report the evaluation design and use an uncertainty method compatible with its dependence.