Does Eighty Percent Mean Eighty Percent?

A model can rank failures well and still overstate its probabilities. We need to compare its confidence with what actually happens across similar predictions.

12 sections · 3 questions · 2 coding challenges

1 / 12

The arrow keys move between sections too.

A Correct Choice Can Still Come With Misleading Confidence

BOTIE

As classifiers improved, a separate question remained: did the number attached to a prediction accurately describe its uncertainty? A model could choose the right class often while being too confident about individual choices.

Guo and colleagues' 2017 study examined this issue in modern neural networks and evaluated calibration methods, including temperature scaling. Our invented sensor group teaches how to inspect reliability before attempting a correction. It does not reproduce their experiments or establish that one adjustment works for every model.

Sources for this history

Can We Trust the Number Beside the Prediction?

Suppose Botie's model gives a group of sensors an 80 percent chance of failure. If we saw many comparable predictions, we would expect failures in roughly eight out of ten cases. That interpretation is what gives the probability practical meaning.

Now look at our small report: four sensors received that prediction, and two failed. Is the model overstating the risk, or could this be ordinary variation in a small group? Four cases cannot settle that question. We will compare predicted probabilities with observed frequencies and keep the number of observations beside each comparison.

The small evaluation group illustrates the calculation; it cannot establish calibration on its own.

Step 1 of 3

Group comparable probability estimates

All four estimates are 0.8, so their average also represents the group's stated confidence.

An invented example for this lesson.
Same sensor	Failure probabilities
Sensor 1	0.8
Sensor 2	0.8
Sensor 3	0.8
Sensor 4	0.8

Sensor 1
Failure probabilities
0.8
Sensor 2
Failure probabilities
0.8
Sensor 3
Failure probabilities
0.8
Sensor 4
Failure probabilities
0.8

Step 2 of 3

Record what happened

One means a failure occurred. Zero means it did not.

An invented example for this lesson.
Same sensor	Outcomes
Sensor 1	1
Sensor 2	0
Sensor 3	1
Sensor 4	0

Sensor 1
Outcomes
1
Sensor 2
Outcomes
0
Sensor 3
Outcomes
1
Sensor 4
Outcomes
0

Step 3 of 3

Compare probability with frequency

The observed fraction is below the predicted probability in this small group. We need more evidence to estimate reliability precisely.

Mean estimated probability = (0.8 + 0.8 + 0.8 + 0.8) / 4 = 0.8
Observed failure frequency = (1 + 0 + 1 + 0) / 4 = 0.5

The model estimated an 80% failure probability for each of these four sensors. Two actually failed. This tiny group illustrates the comparison; it is not enough to establish calibration.

Confidence can be checked against observed frequency

The first pair compares probability with frequency. The separate score reports average squared error without pretending it measures calibration alone.

Do confident predictions match what happened?

Botie has a group of predicted probabilities and their later outcomes. Compare average confidence with the observed failure share, then inspect the separate squared-error score.

BOTIE

Botie needs to compare the group's confidence with its observed failure rate, while also reporting squared probability error.

Choose an example

Failure probabilities in one group

Observed failures

Mean confidence, observed frequency and squared error

[
  [
    0.8,
    0.5
  ],
  0.34
]

Follow the operations

1. Failure probabilities in one group
1. 00.8
2. 10.8
3. 20.8
4. 30.8
2. Mean
```
0.8
```
3. Observed failures
1. 01
2. 10
3. 21
4. 30
4. Mean
```
0.5
```
5. Collect two results
1. 00.8
2. 10.5
6. Subtract
1. 0-0.2
2. 10.8
3. 2-0.2
4. 30.8
7. Square
1. 00.04
2. 10.64
3. 20.04
4. 30.64
8. Mean
```
0.34
```
9. Collect two results
```
[
  [
    0.8,
    0.5
  ],
  0.34
]
```
10. Result
```
[
  [
    0.8,
    0.5
  ],
  0.34
]
```

These tiny groups illustrate the arithmetic. They do not estimate calibration precisely or fit a correction.

1. Calibration Is About Repeated Predictions

We cannot judge an 80 percent prediction as wrong merely because one sensor stayed healthy. A probabilistic statement allows both outcomes. Calibration concerns the outcome frequencies among many comparable probability estimates.

A useful reliability diagram groups predictions into probability ranges and compares each group's average probability with its observed positive fraction.

Grouping trades detail for enough observations to estimate a frequency. Different bin choices can change the appearance, particularly with little data.

2. Calculate One Reliability Group

Our four sensors all received the same estimated probability. Their binary outcomes sum to the number of failures. Averaging those outcomes gives the group's observed frequency.

Mean predicted probability = (0.8 + 0.8 + 0.8 + 0.8) / 4 = 0.8
Observed failure frequency = (1 + 0 + 1 + 0) / 4 = 0.5
Predicted minus observed = 0.8 - 0.5 = 0.3

This group looks overconfident about failure. Four examples are too few to establish a precise long-run discrepancy, so the report must preserve its count.

3. Score the Probability Errors Too

The binary Brier score averages squared differences between probabilities and zero-or-one outcomes. Lower is better. It penalises confident wrong predictions strongly.

Squared errors: [0.04, 0.64, 0.04, 0.64]
Brier score = (0.04 + 0.64 + 0.04 + 0.64) / 4 = 0.34

Brier score is not a pure calibration measure. It also reflects how usefully the model distinguishes different risks. Inspect reliability and predictive discrimination separately rather than collapsing both questions into one label.

4. Fit Corrections on Separate Data

A calibration method learns a mapping from scores to probability estimates. It needs labelled data excluded from the original model fit or an appropriate cross-validation procedure. Evaluate the corrected probabilities on data not used to fit that correction.

A model can be calibrated overall while failing for a subgroup or after a population change. It can also predict the same base rate for everyone and be calibrated while offering little useful ranking.

The workshop audits one group. It does not fit a calibrator or certify that four observations establish reliable probabilities.

5. A Reliability Bin Is a Small Estimation Problem

A group of predictions near the same probability supplies an observed failure fraction. That fraction itself is noisy when the group is small. Keep the number of observations beside every bin, and avoid reading a jagged small-sample diagram as precise evidence of miscalibration.

Binning also hides differences within a bin. Check several sensible summaries and a proper probability score. A low average score does not mean every subgroup is calibrated, especially after the deployment population changes.

6. Equal Decisions Can Hide Different Probability Quality

Two models can classify every observation identically at the halfway threshold while assigning different confidence. The Brier score compares each probability with the observed binary label, squares that discrepancy and averages it.

Use held-out predictions to fit or assess a calibration mapping. Fitting that mapping and reporting its improvement on the same observations gives an optimistic assessment.

Score confidence on the same outcomes

Both models predict the labels correctly, but the second assigns more probability to the observed outcomes.

Python

import numpy as np
y = np.array([0., 0., 1., 1.])
for probability in [np.array([.4, .4, .6, .6]), np.array([.1, .1, .9, .9])]:
    print(round(float(((probability-y)**2).mean()), 3))

Output

0.16
0.01

This batch favours the second set of probabilities. It is a score calculation, not enough evidence to establish calibration across all future predictions.