What Must a Model Improve On?

Botie has a new model for greenhouse temperature. Its reported error sounds small. But could we get a similar result by predicting the training average every time? Let's measure that simple rule before deciding whether the new model is helping.

We will start with a deliberately limited predictor: use the average training temperature for every new observation. That gives us a baseline, a result the more elaborate method should improve under the same evaluation conditions.

Training temperatures: [10, 14, 18]
Reference = (10 + 14 + 18) / 3 = 14

Fit one temperature from the training readings and use it for every held-out case. Changing a held-out answer should change the error, but it must not change that fitted prediction.

20

A regression baseline might predict a mean or median. A classification baseline might use the majority class. A forecast might repeat the latest observation or the matching season. The baseline should be simple but relevant.

For our temperature example we will use the mean, because we will score squared errors. Among constant predictions, the training mean minimises the sum of squared training errors.

Training mean = (10 + 14 + 18) / 3 = 14

This fitted constant ignores inputs such as sunlight and heating settings. It establishes what we can achieve before using those features.

Apply the fitted constant to held-out observations. Do not replace it with the held-out mean. That would use information the predictor did not have when making those predictions.

Held-out predictions: [14, 14]
Held-out answers:     [12, 20]
Errors:               [2, -6]

A baseline must obey the same information boundary as the model it is compared with.

Mean squared error averages the squared differences between predictions and answers. We need the same units, same held-out examples, and same metric for a fair comparison.

Baseline MSE = (2² + (-6)²) / 2
             = (4 + 36) / 2
             = 20

A model with lower error on different examples has not yet won this comparison. Shared evaluation examples let us compare their mistakes directly.

A small reduction in average error may not justify greater latency, maintenance, or instability. Inspect which observations improve and which get worse. Repeat the evaluation when the dataset is small or the fitting procedure is variable.

The baseline is a reference, not a target that guarantees usefulness once beaten. A temperature predictor may still be unsuitable for a particular control decision even after improving its average score.

The workshop builds the whole baseline evaluation from training outcomes and held-out answers. It must not learn its constant from the test branch.

If Botie must predict one temperature for every greenhouse, squared error favours the training mean. Absolute error favours a training median. A baseline is therefore part of the evaluation design, not an arbitrary weak opponent.

Keep its fitted value fixed on held-out rows. Taking the test mean gives the baseline access to the answers. For classification, a majority-label baseline and a constant class-probability baseline serve different metrics and should be named accordingly.

A distant reading pulls the mean because squaring makes a large residual expensive. The median depends on ordering and is less affected by the size of that extreme. Neither is universally better: they minimise different losses.

Compare candidate models with a baseline that is appropriate to the same target, loss and information available at prediction time.

import numpy as np
y = np.array([10., 11., 12., 40.])
for value in [y.mean(), np.median(y)]:
    print(float(value), round(float(np.abs(y-value).mean()), 3), round(float(((y-value)**2).mean()), 3))

18.25 10.875 158.188
11.5 7.75 203.75