Did the Revision Help on the Same Cases?

A lower score on an easier dataset tells us little about a revision. Match the cases, inspect their gains and regressions, then resample the paired observations to explore uncertainty.

14 sections · 3 questions · 1 coding challenge

1 / 14

The arrow keys move between sections too.

Preserving the Comparison When We Resample

BOTIE

Comparing learning procedures requires separating a method change from variation in the evaluation data. Cawley and Talbot examined how selection itself can overfit an estimated performance criterion. A favourable comparison becomes harder to interpret when it was selected from many attempts.

Here we begin with a simpler fixed comparison. Both frozen models see the same cases. We keep each pair together when resampling so the uncertainty calculation still describes their difference on shared observations.

Sources for this history

Did We Improve the Model or Give It Easier Days?

Botie tests the old temperature model during a cold week and the candidate during a mild one. The candidate has a lower error. We still do not know how much of that difference belongs to the revision.

Run both models on the same held-out observations. Then each day supplies a comparison: how much error did the change remove, or add? That pairing also matters when we ask how uncertain the average improvement is.

Invented per-case losses for frozen models. The workshop counts improvements; the Python exercise explores resampling uncertainty.

Step 1 of 2

The candidate helps some days and hurts another

Each row is one shared observation.

Gain is baseline loss minus candidate loss. The mean alone would hide day B.
Day	Baseline loss	Candidate loss	Gain
A	4	2	+2
B	1	2	-1
C	3	2	+1

A
Baseline loss
4
Candidate loss
2
Gain
+2
B
Baseline loss
1
Candidate loss
2
Gain
-1
C
Baseline loss
3
Candidate loss
2
Gain
+1

Gain is baseline loss minus candidate loss. The mean alone would hide day B.

Step 2 of 2

Resample whole comparisons

Each draw selects case positions with replacement. Use the same positions for both models.

These are two illustrative draws, not enough to estimate an interval.
Draw	Positions	Gains	Mean gain
1	[A, A, C]	[2, 2, 1]	1.6667
2	[B, B, C]	[-1, -1, 1]	-0.3333

1
Positions
[A, A, C]
Gains
[2, 2, 1]
Mean gain
1.6667
2
Positions
[B, B, C]
Gains
[-1, -1, 1]
Mean gain
-0.3333

These are two illustrative draws, not enough to estimate an interval.

The unit of comparison must survive resampling

Pairing removes a difference in case difficulty from the direct comparison. It does not remove sampling or selection uncertainty.

Keep each gain attached to its case

Subtract candidate loss from baseline loss before summarising. Positive gains mean lower candidate loss.

BOTIE

Botie needs to compare frozen revisions on shared cases.

Choose an example

Baseline per-case losses

Candidate per-case losses

Paired comparison

{
  "meanGain": 0.6667,
  "improved": 2,
  "regressed": 1,
  "tied": 0
}

Follow the operations

1. Baseline per-case losses
1. 04
2. 11
3. 23
2. Candidate per-case losses
1. 02
2. 12
3. 22
3. Baseline loss minus candidate loss
1. 02
2. 1-1
3. 21

4. Count repairs and regressions

{
  "meanGain": 0.6667,
  "improved": 2,
  "regressed": 1,
  "tied": 0
}

5. Result

{
  "meanGain": 0.6667,
  "improved": 2,
  "regressed": 1,
  "tied": 0
}

This workshop summarises supplied losses. Use the Python exercise and an appropriate sampling design to investigate uncertainty.

1. Pair by identity before subtracting

An evaluation service may return predictions in a different order. Join them by case ID, check for duplicate or missing IDs, then calculate both losses against the same observed outcome. Equal array lengths cannot establish that correspondence.

The workshop begins after that join. Every position represents the same case in both input arrays. Its output keeps improved, regressed and unchanged counts beside mean gain.

Gains = baseline losses - candidate losses
[4, 1, 3] - [2, 2, 2] = [2, -1, 1]
Mean gain = (2 - 1 + 1) / 3 ≈ 0.6667

A positive mean is evidence about this evaluation sample. Inspect the negative entries before deciding the revision is acceptable.

2. Choose a difference that matters to the task

A measured improvement can be too small to matter operationally. Declare a useful effect size in the metric’s units before inspecting the candidate. If a slower model saves a negligible temperature error, the trade may be unattractive.

Also retain group performance. A meaningful average gain does not excuse every local regression, and a nonzero estimate does not by itself establish that the gain will persist.

Example development requirement:
MAE improvement at least 0.5 °C
No declared shift increases MAE by more than 0.2 °C

These are invented requirements for an example. A real application must justify its own limits.

3. Resample pairs rather than separate score lists

The bootstrap treats the observed independent cases as an empirical population. Draw the same number of positions with replacement and average the selected paired gains. Repeat many times to inspect how the estimate changes under that resampling model.

Drawing baseline and candidate positions independently breaks the shared-case relationship. Easy and difficult cases can then be compared against unrelated observations, changing the uncertainty calculation.

gain = baseline_loss - candidate_loss
indices = rng.integers(0, n, size=(repeats, n))
bootstrap_means = gain[indices].mean(axis=1)

The practice function receives explicit indices so its tests are reproducible. A complete analysis would generate many draws with a recorded seed.

4. Read an interval without turning it into certainty

A percentile bootstrap interval takes lower and upper quantiles of the resampled means. For example, the 2.5th and 97.5th percentiles form a conventional approximate 95% interval under suitable sampling conditions. It is not a statement that a fixed unknown gain has a 95% probability of lying in that realised interval.

Few cases, heavy tails or a badly represented population can make the interval unreliable. A narrow interval around a biased sample still describes the wrong population precisely.

lower, upper = np.quantile(bootstrap_means, [0.025, 0.975])

Compare the interval with the practical improvement requirement, not only with zero. Do not estimate a serious interval from the two teaching draws above.

5. The independent unit may be a greenhouse

If one greenhouse contributes hundreds of related readings, resampling individual rows can understate uncertainty. Resample whole greenhouses when the claim concerns new greenhouses, preserving both models’ predictions within each selected group. For dependent time series, use an appropriate temporal design rather than assuming row independence.

This exercise conditions on already fitted models. To assess a learning procedure that will be retrained, the design must also account for fitting and selection variation. Repeating seeds on the same held-out cases does not create new independent cases.

Choose the sampling unit from the claim you want to make, then choose the resampling method.

6. Inspect the indexing on a small draw

NumPy can use a two-dimensional integer array to select several draws from a one-dimensional gain array. Each resulting row is one bootstrap sample. Reducing along axis 1 computes one mean per draw.

Keep the operation inspectable before scaling up the number of draws.

Repeated positions really are repeated observations

This shows indexing only; the challenge adds loss pairing and validation.

Python

import numpy as np
gain = np.array([2., -1., 1.])
positions = np.array([[0, 0, 2], [1, 1, 2]])
print(gain[positions].tolist())

Output

[[2.0, 2.0, 1.0], [-1.0, -1.0, 1.0]]

The repeated positions retain whole comparisons. They do not refit either model.

7. Record the attempts that led here

Save the baseline version, candidate version, case IDs, metric definition and selection history. If this was the best of many candidates on the same data, a fixed-pair interval does not erase that selection.

Use development comparisons to decide what to try. Reserve independent evidence for the chosen procedure or candidate, with a plan for when that evidence has been used up by further decisions.

The next lesson separates choosing a model from estimating how well the whole choosing procedure works.