Section 8 of 24 · 5 lessons

Optimisation and Training

A model learns through changes to its parameters. We will follow those changes directly, from a small batch's gradient to momentum and adaptive updates. We will also check derivatives, inspect training curves, and decide when a saved model is a better choice than the final one.

0 of 5 finished

Making Parameter Updates

A derivative tells us a local direction, but training still needs a rule for using it. Follow small batches, stored velocity, and adaptive scaling through actual parameter changes.

  1. 01Learning From a Few Records at a TimeBotie has more training records than he can process in one step. A batch gives him a manageable estimate of the direction in which to change the model.
    Build with Botie →Read the lesson
  2. 02Remembering the Direction of Earlier UpdatesSuccessive batches can pull a weight in different directions. A stored velocity lets Botie carry some of the previous update direction into the next step.
    Build with Botie →Read the lesson
  3. 03Giving Different Weights Different Step ScalesSome parameters repeatedly receive larger gradients than others. Adam keeps separate running estimates of gradient direction and squared magnitude for each parameter.
    Build with Botie →Read the lesson

Understanding a Training Run

A stalled loss can have several causes. Check the starting values and derivatives, then use separate training and validation evidence to decide whether to continue.

  1. 04Checking a Model Before a Long Training RunA network can fail because its starting values suppress the signal or because its derivative is wrong. Test these separately before tuning the optimiser.
    Build with Botie →Read the lesson
  2. 05Deciding When a Training Run Has Done EnoughTraining loss can improve after predictions on unseen records stop improving. Keep separate curves, choose a stopping rule in advance, and restore the best saved model.
    Build with Botie →Read the lesson