How Much Does Our Estimate Depend on This Sample?

One average hides how it might change if different observations had been collected. Resampling the observed data gives us a way to study that dependence under explicit assumptions.

12 sections · 3 questions · 2 coding challenges

1 / 12

The arrow keys move between sections too.

Using Computation to Recreate the Act of Sampling

BOTIE

Statisticians often need the uncertainty of an estimate even when a convenient formula is unavailable. Repeating the entire real-world data collection may be impractical.

Efron's 1979 bootstrap paper developed a computational approach based on resampling the observed data. Our invented greenhouse readings show how replacement and repeated estimates work. Three displayed resamples make the mechanism inspectable; they are far too few to justify a useful confidence interval.

Sources for this history

Efron: Bootstrap Methods: Another Look at the Jackknife (1979)

Would Another Set of Readings Give the Same Average?

Botie estimates the average greenhouse temperature from a small collection of readings. The arithmetic gives one answer, but another collection could give a different one. We want some sense of how dependent the estimate is on the sample.

The bootstrap repeatedly samples from the observations we already have, with replacement, and recalculates the estimate. It treats the observed collection as a stand-in for the population we could sample again. That assumption is useful, but it is also a limitation.

The draws reuse an invented observed sample; they are not new measurements.

Step 1 of 3

Start with the observed readings

We have three observations. Each bootstrap resample will also contain three entries.

An invented example for this lesson.
Original record	Observed readings
Reading 1	10
Reading 2	20
Reading 3	30

Reading 1
Observed readings
10
Reading 2
Observed readings
20
Reading 3
Observed readings
30

Step 2 of 3

Sample with replacement

Repeated indices repeat an observation. Other observations can be absent from a particular resample.

An invented example for this lesson.
Resample	Draw 1	Draw 2	Draw 3
First index draw	0	0	2
Second index draw	1	1	1
Third index draw	0	2	2

First index draw
Draw 1
0
Draw 2
0
Draw 3
2
Second index draw
Draw 1
1
Draw 2
1
Draw 3
1
Third index draw
Draw 1
0
Draw 2
2
Draw 3
2

Step 3 of 3

Recalculate the estimate each time

The collection of estimates describes variation under the resampling procedure. It does not add independent greenhouse measurements.

An invented example for this lesson.
Resample	Resample means
Resample 1	16.6667
Resample 2	20
Resample 3	23.3333

Resample 1
Resample means
16.6667
Resample 2
Resample means
20
Resample 3
Resample means
23.3333

Resampling changes which observations support the estimate

The resample changes which observed values contribute and how often. Recalculating the mean turns that changed composition into one estimate.

How much can the sample mean move?

Use the saved draw positions to select another sample from the observed readings. A repeated position contributes more than once; an omitted position contributes nothing to that draw.

BOTIE

Botie has a recorded bootstrap draw. Retrieve its observations with replacement and calculate the estimate for that draw.

Choose an example

Original observations

Recorded bootstrap indices

Mean for the recorded bootstrap draw

16.6667

Follow the operations

1. Original observations
1. 010
2. 120
3. 230
2. Recorded bootstrap indices
1. 00
2. 10
3. 22
3. Gather selected entries
1. 010
2. 110
3. 230
4. Mean
```
16.6667
```
5. Result
```
16.6667
```

The construction receives a fixed draw and performs no random sampling. A few illustrative means are not a confidence interval.

1. Treat the Observed Sample as a Temporary Population

For an ordinary nonparametric bootstrap, draw as many observations as the original sample contains, with replacement. Each original observation has equal chance on each draw.

Replacement is essential. Sampling every observation once without replacement would only reorder the same sample and leave its mean unchanged.

Original sample: [10, 20, 30]
A valid resample: [10, 10, 30]
Another resample: [20, 20, 20]

Repeated values are repeated selections of the same observed value. They are not new sensor measurements.

2. Recompute the Complete Statistic

For each resample, calculate the quantity we are studying again. Here we want a mean, so that is all we need to recompute. If we were studying a fitted model, we might need to repeat its preprocessing and fitting too. Resampling a final score alone would miss the variation introduced by those earlier choices.

First mean = (10 + 10 + 30) / 3 ≈ 16.6667
Second mean = (20 + 20 + 20) / 3 = 20
Third mean = (10 + 30 + 30) / 3 ≈ 23.3333

These three fixed examples explain the mechanism. A real bootstrap uses many random resamples and checks whether the summary is stable enough for its purpose.

3. Summarise Variation Without Overclaiming

The spread of bootstrap estimates can estimate a standard error under appropriate conditions. Their quantiles can support a percentile interval, although different interval methods have different assumptions and accuracy.

Do not call the range of three hand-picked resample means a 95 percent confidence interval. Neither their number nor their selection supports that interpretation.

A confidence interval is a property of a repeated-sampling procedure. It is not a guarantee that every future individual greenhouse reading lies inside it.

4. Resample the Independent Units

Ordinary row resampling assumes the observed rows are suitable independent sampling units. Consecutive readings from one greenhouse may be dependent. Group or block resampling can preserve some of that structure when justified.

Resampling also cannot repair a biased collection process or create examples of conditions absent from the original data. If every original reading came from one season, bootstrap copies do not supply evidence about another season.

The workshop gathers one supplied resample and calculates its mean. Python then processes several recorded draws, making the distinction between a draw and its estimate explicit.

5. Keep Readings from the Same Greenhouse Together

If Botie has ten readings from each greenhouse, those readings may share the same device bias or environment. Resampling individual rows treats them as independent pieces of evidence. Resampling greenhouses carries their related rows together.

Choose the unit to match the data collection process and the target of inference. A time series may require blocks that retain local dependence. Merely drawing more bootstrap replicates cannot repair an inappropriate resampling unit.

6. A Constant Sample Cannot Reveal Variation It Never Contains

The bootstrap reuses the empirical distribution of the observed data. If every observed value is identical, every resampled mean is identical too. That narrow result describes the sample we supplied, not proof that the wider population has no variation.

Percentile endpoints summarise the simulated estimates under this resampling scheme. Their quality depends on sample size, the statistic, dependence and bias. State those assumptions beside the interval rather than treating any two quantiles as a universal uncertainty guarantee.

Inspect a degenerate resampling distribution

All three observed readings happen to be identical.

Python

import numpy as np
x = np.array([12., 12., 12.])
rng = np.random.default_rng(4)
means = rng.choice(x, size=(100, len(x)), replace=True).mean(axis=1)
print(np.quantile(means, [.025, .975]).tolist())

Output

[12.0, 12.0]

More resamples would reproduce the same limitation. Collecting a more informative sample is a different operation from resampling this one.