Remembering the Direction of Earlier Updates

One batch tells Botie to increase a weight, while the next tells him to decrease it. Some of that variation comes from sampling, and some comes from the shape of the loss.

Could the recent gradients help us decide which way to move? Momentum keeps a velocity between updates and combines it with the current gradient. We need to track that velocity separately from the parameter: a parameter can keep moving in its previous direction even after the latest gradient changes sign.

Previous velocity: 2
Current gradient: −1

Retained fraction beta: 0.5
New velocity = (0.5 × 2) + (−1) = 0

Old weight: 4
New weight = 4 − (learning rate × 0) = 4

Try a gradient that agrees with the stored velocity, then one that opposes it. Compare the resulting parameter movement before discarding that history.

4

A plain descent step uses only the current gradient. Momentum also stores velocity, usually initialised to zero. The retention factor beta determines how much old velocity survives.

Velocity has the same shape as the parameter array. It must persist between batches, rather than be recreated inside each update. Keeping a state for the wrong parameter shape can silently mix unrelated quantities.

Momentum needs more state than the weights alone.

Multiply old velocity by beta and add the new gradient. Then subtract the learning-rate-scaled velocity from the weight. In this convention, we do not multiply the current gradient by one minus beta.

Old velocity = 2; beta = 0.5; gradient = -1
New velocity = 0.5 × 2 - 1 = 0
New weight = 4 - 0.1 × 0 = 4

Other conventions are valid, but their learning rates and stored values are not interchangeable.

With repeated gradients in one direction, velocity grows until decay balances the incoming contribution. When a gradient reverses, old velocity can delay the reversal of the parameter’s motion.

That memory can help in narrow loss valleys, but can also carry an update past a minimum. Momentum does not know where the minimum is; it only combines gradients.

Plot the parameter and velocity separately when debugging a training run.

import numpy as np
weight = 4.0
old_velocity = 2.0
gradient = 1.0
retention, rate = 0.5, 0.1
velocity = retention * old_velocity + gradient
print(velocity)
print(weight - rate * velocity)

2.0
3.8

Restarting with the same weights but zero velocity changes the next update. For a faithful continuation, save the optimiser state, update count, learning-rate schedule, and relevant random state.

Compare momentum settings on a validation procedure. A high retention factor is not always better, and changing it changes the scale of accumulated velocity under this convention.

The workshop makes one velocity update, then uses it to change the weight.

Momentum carries part of the previous update direction into the next step. If gradients repeatedly point the same way, those contributions accumulate. If signs alternate, they can partly cancel.

That memory means the current update need not point along the current gradient alone. Near a minimum, old velocity can carry the parameter past it. A learning rate and momentum coefficient must therefore be considered together, rather than treating momentum as an automatic acceleration button.

We will use a velocity that accumulates gradients, followed by subtracting learning rate times velocity. Another common convention includes learning rate inside the velocity. The two forms need consistent state definitions, especially if the learning rate changes.

Track gradient, velocity and parameter separately. Their signs answer different questions: current slope, retained direction and current location.

import numpy as np
w, velocity = 0., 0.
for gradient in [2.,2.,-1.]:
    velocity = .8*velocity + gradient
    w -= .1*velocity
    print(round(velocity, 3), round(w, 3))

2.0 -0.2
3.6 -0.56
1.88 -0.748

A training checkpoint needs the velocity as well as the model parameters. Restoring only the weights and zeroing velocity starts a different trajectory, even on the same next batch.

Test checkpoint continuation against an uninterrupted run. Save the learning-rate schedule position and random state when they influence the next updates. If a run oscillates, inspect these states and gradient scales before adding more training passes.