Bounding Boxes and Overlap

An image classifier can tell Botie that a photograph contains a seed. If he needs to inspect that seed, an answer about the whole photograph is not enough. He needs a location, and possibly several locations for several objects.

One way to describe a location is a rectangle around the object. We can supply labelled rectangles during training and ask a detector to predict them. Before discussing a whole detector, let's work out how to represent a rectangle and judge whether two rectangles refer to roughly the same region.

First box area: 16
Second box area: 16
Shared area: 8
Union = 16 + 16 − 8 = 24
Intersection over union = 8 / 24 ≈ 0.3333

Move an edge of the predicted box. Compare the shared area with the total area covered by either rectangle, including the point where the boxes stop overlapping.

0.3333

We store each rectangle as x-min, y-min, x-max, y-max. The first two entries identify its upper-left boundary in image coordinates; the last two identify its lower-right boundary. The right and bottom boundaries are exclusive.

A valid rectangle has positive width and height. These boundary coordinates may be floating-point values. Do not add one when calculating their lengths. An inclusive integer-pixel annotation format uses different bookkeeping and should be converted deliberately.

box = [1, 2, 5, 6]
width = 5 - 1 = 4
height = 6 - 2 = 4
area = 4 × 4 = 16

Some models instead output centre, width, and height. Translate representations before comparing them.

To find the shared part of two rectangles, choose the farther-right left boundary and the lower top boundary. For the other side, choose the nearer-left right boundary and the higher bottom boundary. Those choices give the intersection limits.

A negative width or height means there is no overlap in that direction. Set that length to zero before calculating area. Otherwise two negative lengths could multiply to a positive number and report an overlap that does not exist.

intersection_width = max(0, min(right_A, right_B) - max(left_A, left_B))
intersection_height = max(0, min(bottom_A, bottom_B) - max(top_A, top_B))
intersection_area = intersection_width × intersection_height

Rectangles that only touch along an edge have zero intersection area under this convention.

Adding the two areas counts their shared region twice. Subtract the intersection once to obtain the union. Intersection over union, or IoU, is the shared area divided by that combined covered area.

For valid positive-area rectangles, IoU lies between zero and one. Identical boxes score one. Disjoint boxes score zero. A small box fully inside a large one does not score one because the extra area still belongs to the union.

A = [0,0,4,4], B = [2,0,6,4]
intersection = 2 × 4 = 8
union = 16 + 16 - 8 = 24
IoU = 8 / 24 = 1 / 3

This ratio is unitless, provided both boxes use the same coordinate system.

import numpy as np
a = np.array([0.,0.,4.,4.])
b = np.array([2.,0.,6.,4.])
def iou(a, b):
    overlap = np.maximum(0, np.minimum(a[2:], b[2:]) - np.maximum(a[:2], b[:2]))
    intersection = np.prod(overlap)
    union = np.prod(a[2:] - a[:2]) + np.prod(b[2:] - b[:2]) - intersection
    return intersection / union
print(f'{iou(a, b):.4f}')

0.3333

A detector often predicts a class, a score, and a rectangle for each candidate object. A high class score does not guarantee that the rectangle is well placed. Conversely, a well-placed rectangle with the wrong class is still a wrong detection.

Training and evaluation need labels appropriate to the task. A collection containing only whole-image class labels does not automatically supply the object coordinates required by a supervised box detector.

A bounding box also differs from a segmentation mask: it can include background around an irregularly shaped object.

An evaluation protocol specifies an overlap threshold, class matching, and how predictions are matched to annotated objects. Duplicate predictions cannot all count as separate correct detections of one object.

An identical coordinate error can hurt a small object more than a large one. Inspect examples across object sizes and read the protocol rather than treating one IoU threshold as universally correct.

Botie's workshop computes IoU for one pair. The next lesson uses overlap to handle duplicate candidates; a full detection metric also needs matching across an entire labelled dataset.

To score a detector, sort predictions by confidence under the evaluation protocol and match eligible predictions to labelled objects. One labelled seed should not become several true positives because several boxes cover it.

Class agreement and sufficient overlap are both necessary. Keep unmatched predictions as false positives and unmatched labels as misses. Average precision summarises a ranking under a particular matching and overlap protocol; one good IoU calculation is only a component of that procedure.

If width and height change by different factors, x coordinates and y coordinates need different multipliers. The box order must be known before applying them. A detector output in resized coordinates must be mapped back before displaying it on the original image.

Letterboxing introduces padding offsets as well as scaling. Keep both transformations, and undo them in reverse order. Clipping to image bounds is a separate policy and can turn a box into an empty one.

import numpy as np
box = np.array([2., 1., 6., 5.])
scale = np.array([.5, 1., .5, 1.])
resized = box * scale
print(resized.tolist())
print((resized / scale).tolist())

[1.0, 1.0, 3.0, 5.0]
[2.0, 1.0, 6.0, 5.0]