Transfer Learning

Botie wants to distinguish healthy leaves from damaged ones, but he has only a small labelled collection. A large image network contains far more adjustable settings than this small collection can reliably teach from scratch.

Suppose another network has already learned to measure edges, textures, and more complicated visual patterns. Could those measurements help with our leaves? We will first keep that measuring process fixed and learn a new rule from its outputs.

The feature rows have already been computed. Change the head's coefficients and compare its class decisions without changing what the feature extractor supplied.

A feature extractor is the part of a model that turns an input into a vector or feature map. A prediction head is the part that turns that representation into the output we need. We usually replace the old head because its classes differ from ours.

In this lesson we supply tiny feature vectors directly. They are invented measurements that let us inspect the head's arithmetic, not outputs from a downloaded pretrained network.

Image → fixed feature extractor → feature vector h
Score = w · h + b
Probability of class 1 = sigmoid(score)

The class meaning comes from the new labelled task. A pretrained network does not know our class definitions merely because it has useful features.

Let's first keep the feature extractor's parameters fixed and train only the new head. Disabling optimizer updates is one part of that job. Dropout and batch normalization also have training and evaluation behaviours, which are controlled separately from whether parameters receive updates.

The images must be prepared as the pretrained weights expect. Check their channel order and resizing policy, as well as normalization. If we cache features to reuse later, run the extractor in the intended deterministic evaluation mode so the saved values represent the same procedure.

The PyTorch reference shows these controls in a real model. Here we work directly with fixed feature rows so the boundary is explicit.

For a binary task, a linear head supplies one weight per feature and a bias. The sigmoid turns its score into a probability. We can then use binary cross-entropy to fit the head.

Each training row contributes the difference between its predicted probability and observed label. Combining those differences with the feature columns gives the weight gradient. Averaging the differences gives the bias gradient.

scores = H @ w + b
p = sigmoid(scores)
weight_gradient = H.T @ (p - y) / number_of_rows
bias_gradient = mean(p - y)

The gradient changes the head only. The supplied feature rows stay fixed.

import numpy as np
H = np.array([[1.,0.], [0.,1.], [1.,1.]])
w = np.array([2.,-1.])
b = -0.5
scores = H @ w + b
print(scores.tolist())
print((scores >= 0).astype(int).tolist())

[1.5, -1.5, 0.5]
[1, 0, 1]

If the frozen features are insufficient, we can unfreeze some or all of the extractor and continue training. This is fine-tuning. It adds flexibility but also more ways to overfit a small dataset or damage useful existing features.

A common experiment starts with the head, then tries limited fine-tuning with a smaller learning rate for the extractor. The appropriate settings depend on the source and target data. Keep a held-out comparison and inspect class-specific errors.

Pretraining on ordinary photographs may transfer poorly to a very different measurement process. Reuse is a hypothesis to evaluate, not a guarantee.

Separate related photographs before fitting or augmenting. Compare the transferred model against a simple baseline using the same held-out groups. Fit any new preprocessing statistics on training data only.

Botie's workshop follows a frozen feature row through weights, bias, and a decision threshold. The programming challenges then make one head update so you can see exactly which numbers training changes.

This exercise covers the head and the transfer protocol. It does not measure the performance of an actual pretrained vision backbone.

Preventing parameter updates and selecting evaluation behaviour are separate decisions. Dropout can still randomise features in training mode. Batch normalisation can update running statistics even when its trainable coefficients are frozen.

If we cache features, use a documented deterministic extractor configuration and preserve its preprocessing. Otherwise the head may be trained on representations that differ from those used at prediction time. Changing the extractor invalidates a cache made from its earlier state.

For a binary head, subtract labels from sigmoid probabilities to obtain score derivatives under mean binary cross-entropy. Multiply those derivatives by the feature table to collect weight gradients. The bias receives their mean.

Repeat the update on the same supplied features to see the head learn. A real transfer experiment would replace these invented features with a pretrained extractor and evaluate on held-out leaves.

import numpy as np
features = np.array([[-2.],[-1.],[1.],[2.]])
y = np.array([0.,0.,1.,1.])
w, b = np.zeros(1), 0.
for _ in range(200):
    probability = 1/(1+np.exp(-(features@w+b)))
    error = probability-y
    w -= .1*(features.T@error)/len(y)
    b -= .1*error.mean()
print(((features@w+b) >= 0).astype(int).tolist())
print(features[:,0].tolist())

[0, 0, 1, 1]
[-2.0, -1.0, 1.0, 2.0]

If the source training discarded exactly the variation our target needs, a frozen head may have little information to use. Compare frozen features, limited fine-tuning and a suitable simpler baseline under the same split.

Inspect which classes improve and which regress. A smaller learning rate for the extractor is a reasonable experiment, not a guarantee against damaging useful features. Save the best validated state and its preprocessing together, rather than selecting the last training step automatically.