Support Vector Machines and Margins

Imagine two groups of inspected parts plotted by their measurements. We can draw a line between the groups, but we can also slide or rotate that line and still classify every training part correctly. Which line should we choose?

Measurements are not perfectly stable. A boundary pressed against one part leaves little room for a slightly different reading. Let's look for space between the boundary and the nearby examples, then decide what to do when a few observations intrude into that space.

A point can be on the correct side and still lie inside the requested margin. Change a score or label and inspect the shortfall before averaging over the parts.

0.25

A linear score multiplies each measurement by a weight and adds a bias. The zero-score locations form the boundary. For this binary model, use class labels minus one and plus one. The score's sign chooses the side.

Multiplying the score by the known label makes a positive value mean correct-side placement for either class. This product is the functional margin.

score = w · x + b
signed score = y × score
For y = -1 and score = -0.5:
  signed score = (-1) × (-0.5) = 0.5

A positive signed score is on the correct side. Its numerical size still depends on how the weights and bias are scaled.

Doubling every weight and the bias preserves the zero boundary while doubling all scores. It has not moved any observation. Dividing by the weight vector's length removes that arbitrary scale and gives signed perpendicular distance.

The conventional margin boundaries have scores minus one and plus one. Once we use that convention, a smaller weight norm means a wider strip.

signed distance = y(w · x + b) / ||w||
width between score -1 and score +1 = 2 / ||w||

The distances depend on the feature units. Fit a scaler using training data before comparing features with incompatible scales.

An observation with signed score at least one lies on or beyond its margin boundary and receives zero hinge loss. A correctly classified observation inside the strip still receives a positive loss. A wrong-side observation receives a still larger loss.

Hinge loss measures this shortfall. It does not represent the probability of a class.

hinge = max(0, 1 - signed score)
signed score 1.5 → loss 0
signed score 0.5 → loss 0.5
signed score -0.5 → loss 1.5

The workshop calculates these violations for supplied weights. It lets you inspect an objective component without pretending the weights were learned there.

The soft-margin objective combines a weight penalty with hinge losses. In the sum convention below, a larger C makes violations more expensive relative to widening the margin. Some formulations average losses and rescale the regularisation parameter; compare conventions before copying settings.

An optimiser chooses weights and bias to reduce the combined objective. Examples on or inside the margin can matter to the solution. The examples with nonzero dual coefficients are the support vectors.

Objective = (1/2)||w||² + C × Σ hinge_i
If ||w||² = 0.25, C = 0.5, and Σ hinge_i = 1:
  objective = 0.125 + 0.5 = 0.625

This objective tolerates overlap. It does not promise that a linear boundary can express every useful pattern.

If a straight boundary is unsuitable in the measured coordinates, transformed features may help. Kernel support-vector machines express comparisons through kernel evaluations. The kernel lesson explains how such comparisons can correspond to a more useful representation.

Choose the penalty, kernel, and scaling through validation. A classifier's raw decision score is not automatically a calibrated probability, and increasing flexibility can fit accidental patterns.

We now have a reason to choose a boundary, an explicit cost for exceptions, and a connection to the kernel machinery already in the course.

A point outside its margin already satisfies the hinge requirement, so that loss contributes no gradient there. A point inside the margin, including a correctly classified one, pushes the boundary toward a larger signed score. The weight penalty simultaneously pulls the weights toward zero.

The bias is not penalised in the convention used here. We average hinge losses and multiply them by C. This differs by a dataset-size factor from a sum-loss convention, so a copied numerical C need not represent the same tradeoff.

Begin with zero weights, calculate every signed score, collect only margin violations and make a small subgradient step. Repeat that procedure, then examine classifications and margin values separately.

Hinge loss is not differentiable exactly at its corner. The code uses zero contribution when the margin is exactly one, an allowed subgradient choice. A fixed step size can oscillate near the optimum; this small demonstration is not a replacement for a production solver.

import numpy as np
x = np.array([-2., -1., 1., 2.])
y = np.array([-1., -1., 1., 1.])
w, b, C = 0., 0., 2.
for step in range(1000):
    active = y*(w*x+b) < 1
    dw = w - C*np.mean(np.where(active, y*x, 0.))
    db = -C*np.mean(np.where(active, y, 0.))
    rate = 0.1 / np.sqrt(step+1)
    w -= rate*dw
    b -= rate*db
print(np.sign(w*x+b).astype(int).tolist())
print(bool(w > 0), bool(abs(b) < 1e-10))

[-1, -1, 1, 1]
True True

Changing a feature from metres to millimetres changes its coefficient scale and the geometry of the weight penalty. Standardise using training statistics when that matches the feature meaning, and use the same transform for validation and prediction.

Inspect sensitivity to C on held-out records rather than choosing the setting with the widest-looking training strip. Overlapping classes, measurement errors and nonlinear relationships can all make a perfect separating line an unsuitable target.