When a Measurement Is Missing

An empty thermometer reading does not mean the greenhouse was at zero degrees. We need to distinguish what was measured from what we supplied to keep a calculation working.

12 sections · 3 questions · 3 coding challenges

1 / 12

The arrow keys move between sections too.

The Missing Observation Has a Story Too

BOTIE

Incomplete records posed a problem for statistical inference long before modern neural networks. Ignoring an absent observation can change the group we describe, especially when absence is related to the value we wanted to measure.

Rubin's 1976 paper established conditions under which the missingness process can be ignored for particular forms of inference. It did not say that replacing every blank with an average is universally valid. Our invented greenhouse example uses mean replacement as a transparent baseline so we can inspect exactly what it assumes.

Sources for this history

What Should We Do With an Empty Cell?

Botie opens the greenhouse report and finds a blank temperature. Perhaps the sensor lost power, or perhaps it stopped reporting when the greenhouse became too hot. Those possibilities have different implications.

The calculation we want to run may still need a number in that cell. We can estimate one from the available readings. But we should keep track of where we made that substitution: we now have a usable input, while the actual temperature remains unknown.

Missing entries and measured temperatures are kept distinct in this invented report.

Step 1 of 3

Leave the blank visible

A blank is an unknown temperature. Writing zero there would assert something we did not observe.

An invented example for this lesson.
Training record	Training readings
Reading 1	10
Reading 2	missing
Reading 3	14

Reading 1
Training readings
10
Reading 2
Training readings
missing
Reading 3
Training readings
14

Step 2 of 3

Use only the recorded training values

The two observed training readings give a replacement reference. A later observation must not help choose it.

Observed training readings: 10 and 14
Replacement = (10 + 14) / 2 = 12

Estimate the replacement from the recorded training values. The missing entry contributes no measured value to this average.

Step 3 of 3

Keep the new report's measured entries

Only the missing entry changes. The recorded temperature stays intact.

An invented example for this lesson.
New record	New report	Filled report
Reading 1	missing	12
Reading 2	18	18

Reading 1
New report
missing
Filled report
12
Reading 2
New report
18
Filled report
18

Learn the replacement before filling the later report

The training branch supplies one reference. The report branch retains its measurements and uses that reference only at absent entries.

What should fill a gap in the report?

A blank temperature is different from a measured zero. Change the observed training values and watch the learned replacement change, then check that real report readings remain untouched.

BOTIE

Botie's report has absent temperatures. Learn a replacement from observed training readings, then fill only the missing report entries.

Choose an example

Training temperatures

New greenhouse report

Report with missing entries filled

Follow the operations

1. New greenhouse report
1. 0null
2. 118
2. Training temperatures
1. 010
2. 1null
3. 214
3. Keep observed readings
1. 010
2. 114
4. Mean
```
12
```
5. Fill missing readings
1. 012
2. 118
6. Result
1. 012
2. 118

Mean replacement is a baseline. It cannot recover the true missing values or represent uncertainty about them.

1. Separate Missing From Zero

Zero is a possible temperature. Missing means we do not know the temperature. These must remain different values. The workshop uses a blank cell represented internally by null; Python uses a floating-point missing marker called NaN.

First identify which entries were observed. Do not let the missing marker participate in an ordinary average.

Training readings: [10, missing, 14]
Observed readings: [10, 14]

The set of observed readings may itself be biased. If the hottest sensors stop reporting, its average will not describe the missing temperatures well.

2. Learn the Replacement From Training Data

Mean imputation uses the average of observed training values as a replacement. Imputation means supplying missing entries. We estimate that reference once during fitting, then reuse it on later data.

Training reference = (10 + 14) / 2 = 12
New report:    [missing, 18]
Filled report: [12, 18]

Including the new report's measured temperature when choosing the reference would change the fitted preprocessing after seeing new data. For evaluation, it would let held-out observations influence the preparation.

3. Keep Track of What Was Replaced

A missingness indicator records which values were supplied. A one in that indicator means the original entry was absent; a zero means it was measured. That information can matter if missingness is related to how the records were collected.

New report:          [missing, 18]
Missing indicator:   [1, 0]
Filled temperatures: [12, 18]

The indicator preserves a distinction, not a guarantee. It cannot recover the absent reading. Mean replacement also understates variability if we later treat supplied values as independent observations.

4. Choose the Assumption Before the Convenience

A median can resist extreme observed values. A fixed replacement may have domain meaning. More elaborate methods use relationships among features, and some models handle missing entries directly. Each choice still needs evaluation under the intended conditions.

If every training value is missing, there is no observed mean to calculate. Stop and choose a documented fallback or remove that feature for a reason. Silently inventing a zero would hide the failure.

Botie's construction must learn from the training readings and fill the new report. It must preserve every value that was actually observed.

5. Fit the Replacement Once, Then Carry It Forward

The training rows provide the reference for missing measurements. Incoming rows receive that reference even when their observed values have a very different average. Otherwise a prediction depends on which other records happened to arrive in the same batch.

Keep a missingness indicator if absence may itself be informative. Decide separately what to do with a training column that has no observed values at all; there is no sample mean to learn from it.

An unusually hot batch must not redefine the training mean

Training observations are 10 and 14 degrees. A later batch contains one observation and one blank.

Python

import numpy as np
training = np.array([10., np.nan, 14.])
reference = training[np.isfinite(training)].mean()
incoming = np.array([30., np.nan])
missing = ~np.isfinite(incoming)
print(np.where(missing, reference, incoming).tolist())
print(missing.tolist())

Output

[30.0, 12.0]
[False, True]

The filled value describes our training-based rule. It is not evidence that the second greenhouse was actually at 12 degrees.

6. What If the Hottest Sensors Stop Reporting?

Mean replacement is particularly misleading when missingness depends on the unobserved value. If thermometers fail above a temperature limit, the available readings systematically omit the hottest conditions. Their mean can be too low for the missing group.

Compare reporting rates by device, time and known operating conditions. An indicator may help prediction, but it does not identify the missing temperatures. Investigating the measurement process or obtaining another sensor can be more valuable than choosing a more elaborate filling rule.