Checking a Model Before a Long Training Run

Botie starts a training run and the loss hardly changes. Increasing the learning rate might help, but it might also hide a broken derivative or make an unstable calculation worse.

Before changing the learning rate, let's inspect the starting weights. Do they allow useful signals to reach the later layers? Then we can check a derivative independently by nudging one parameter and measuring the change in loss. Agreement gives us a reason to trust the update calculation.

Loss(w) = (w − 3)²
Test weight: 1
Loss(1) = (1 − 3)² = 4

Step h = 0.00001
Loss(1 + h) = 3.9999600001
Loss(1 − h) = 4.0000400001
Central difference = (3.9999600001 − 4.0000400001) / 0.00002 ≈ −4

Analytical derivative = 2 × (1 − 3) = −4
Numerical derivative ≈ −4

The loss has been measured on either side of a parameter. Compare the numerical slope with the proposed gradient and consider how the perturbation size affects that comparison.

-4

If every hidden unit starts with identical incoming weights and receives identical updates, they can remain copies. Random initialisation breaks that symmetry. But arbitrarily large weights can saturate activations, while tiny scales can shrink signals through many layers.

A variance rule uses the number of inputs and outputs to set a starting scale. Which rule fits depends on the activation and architecture. Biases can often start at zero without making all hidden weights identical.

Use a seeded generator for a reproducible check, not the same identical value for every weight.

For a layer with fan_in inputs and fan_out outputs, a common Glorot normal scale uses a variance based on both. This is a design starting point, not a claim that one tiny sampled matrix has exactly that variance.

Variance = 2 / (fan_in + fan_out)
fan_in = 4; fan_out = 4
Standard deviation = sqrt(2 / 8) = 0.5

ReLU networks often use an incoming-connection-based rule instead; activation assumptions matter.

Hold data and all other parameters fixed. Evaluate loss slightly above and below one parameter value, subtract, and divide by twice the perturbation. Compare with the implemented derivative at the unperturbed value.

Choose a smooth point and use floating-point precision appropriate for the test. Too large a perturbation introduces approximation error; too small one can suffer cancellation.

Numerical derivative ≈ [L(w+h) - L(w-h)] / (2h)
L(w) = (w-3)²; w = 1
Analytical derivative = 2 × (1-3) = -4

Repeat for several parameters and several perturbation sizes before trusting a complex backward pass.

import numpy as np
def loss(weight):
    return (weight - 2.0) ** 2

weight, h = 4.0, 0.01
above = loss(weight + h)
below = loss(weight - h)
slope = (above - below) / (2 * h)
print(f'{slope:.4f}')

4.0000

Disable random dropout and freeze sampled data during a gradient check. Do not compare two losses evaluated with different random masks. At a ReLU corner, the derivative is not unique, so use points away from that corner.

Compare with a tolerance that combines absolute and relative error. A derivative check establishes local consistency with the coded loss; it does not show that the loss or labels represent the intended task.

The workshop calculates a central difference from two supplied loss evaluations.

If hidden units start with identical incoming and outgoing weights, they receive the same activations and updates under a symmetric architecture. Several units can then behave like copies of one unit instead of learning different features.

Small random differences break that symmetry. Their scale also matters: overly large values can saturate some activations, while overly small values can weaken signals through many layers. Fan-in based initialisation rules address signal scale under assumptions about the layer and activation; they are not universal constants.

A coordinate check nudges one parameter at a time. A directional check nudges several together and compares the finite difference with the gradient's dot product with that direction. It is an inexpensive additional check for a larger parameter vector.

Use deterministic evaluation during the comparison. Dropout draws, changing batches or stateful updates can change the loss for reasons unrelated to the parameter perturbation.

import numpy as np
w = np.array([1., -2.])
direction = np.array([.6, .8])
def loss(v):
    return float((v*v).sum())
eps = 1e-5
numerical = (loss(w+eps*direction)-loss(w-eps*direction))/(2*eps)
analytic = (2*w)@direction
print(round(float(numerical), 4))
print(round(float(analytic), 4))

-2.0
-2.0

A large perturbation measures curvature over too wide an interval. A very small one subtracts nearly equal floating-point losses and can amplify rounding error. Try a sensible range and look for agreement over a stable region.

At a nondifferentiable point, such as the corner of ReLU, a symmetric finite difference can disagree with the implementation's selected derivative without revealing a bug. Move away from the corner for the smooth check, and test the chosen boundary convention separately.