Deciding When a Training Run Has Done Enough

The model keeps fitting the training records more closely, but its error on the validation records has begun to increase. Continuing just because the training loss falls can make the final model less useful.

We will distinguish optimisation progress from generalisation, keep the best validation checkpoint, and make the rules for reducing the learning rate or stopping explicit.

The latest checkpoint is not automatically the best one. Change the validation sequence and check which saved state the declared criterion selects.

1

Training loss reports the objective on records used to update parameters. Validation measures a held-out choice criterion. Use consistent evaluation mode, preprocessing, and aggregation so a change in the curve has an interpretable meaning.

Training and validation loss need not be directly comparable when regularisation or augmentation differs. Log those conditions rather than assuming every gap is overfitting.

A curve is evidence about a defined calculation, not a diagnosis by itself.

We need to decide what counts as an improvement before watching the run. Does a better metric go up or down, and how large must the change be? Whenever the criterion improves, save a copy of the model. A patience count lets us allow some unsuccessful checks before stopping. Reset that count on improvement, and stop when it reaches the limit we chose.

Validation losses = [0.8, 0.5, 0.55, 0.6]
Best zero-based epoch = 1
With patience 2, stop after epoch 3
Restore the checkpoint from epoch 1

Off-by-one errors are common, so write whether patience counts completed unsuccessful checks.

A fixed schedule uses the update or epoch index. A plateau schedule responds to a monitored metric. Both require clear timing: a rate used for the current update is different from a rate changed afterward.

Changing batch size changes updates per epoch. If a schedule is specified in updates, do not silently reinterpret it in epochs when comparing runs.

Learning-rate scheduling and early stopping can coexist, but their counters and purposes differ.

import numpy as np
validation_loss = np.array([0.2, 0.4, 0.3])
best_position = int(np.argmin(validation_loss))
print(best_position)
print(f'{validation_loss[best_position]:.2f}')

0
0.20

Repeatedly selecting settings against validation can overfit that validation set. Keep an untouched final test and report the selection process, seed variation, and important subgroup errors.

A plateau may come from a bad learning rate, poor features, wrong labels, or a bug. Check a small batch fit, gradient correctness, data alignment, and a simple baseline before spending more time on a long run.

Botie’s workshop finds the checkpoint index. The coding task also implements the stopping counter.

Early stopping needs a record of the best validated state, not only a counter of disappointing epochs. If training continues for several checks after the best score, the final parameters can be worse than the saved ones.

Copy the parameter arrays when recording the checkpoint. Storing another reference to mutable weights lets later updates overwrite the supposed best state. Keep training and validation losses separate, and never use the final test score to decide when to stop.

Patience counts consecutive validation checks without a qualifying improvement. Declare whether equality counts, what minimum improvement is required and how often validation runs.

The example stores a step identifier instead of parameter arrays so we can see the decision clearly. A real run would save the full fitted state at that step.

import numpy as np
losses = [1., .8, .7, .71, .72, .69]
best, best_step, stale = float('inf'), None, 0
for step, loss in enumerate(losses):
    if loss < best:
        best, best_step, stale = loss, step, 0
    else:
        stale += 1
    if stale >= 2:
        break
print(step, best_step, best)

4 2 0.7

If a flexible network cannot fit a tiny clean batch, inspect labels, shapes, loss conventions and gradients before collecting more data. Check that parameters change and that a small step can reduce the chosen loss.

Once that mechanism works, restore the intended regularisation and evaluate on independent data. Successfully memorising a tiny batch is a debugging check. It is deliberately not evidence of useful generalisation.