Which Alerts Deserve an Inspection?

A fault detector can look accurate while missing almost every fault. Follow the alerts and the faults separately, then choose a threshold the inspection team can actually use.

13 sections · 4 questions · 1 coding challenge

1 / 13

The arrow keys move between sections too.

Finding Relevant Items in a Much Larger Collection

BOTIE

Precision and recall are established measures in information retrieval: returning many documents is only useful if the relevant ones are among them, and a short clean list can still omit most relevant documents. The same counting problem appears in a rare-fault detector.

Modern classification libraries expose these measures over many score thresholds. Our pump example keeps the underlying counts visible so a curve cannot hide the inspection workload.

Sources for this history

Scikit-learn: precision-recall and average precision

What If Almost Every Pump Is Healthy?

Most pumps on Botie’s route work normally. A detector that always says healthy could get a very high accuracy simply because faults are rare. It would also give the inspection team no help finding them.

We need to ask what fraction of actual faults we find and what fraction of our alerts lead to a real fault. Those questions use different denominators. Let’s keep the individual pumps visible while we calculate both.

Invented fault scores and binary labels. Scores are supplied; their calibration has not been established.

Step 1 of 2

A score orders the inspection queue

A larger score puts a pump nearer the front. The actual label remains separate from the score.

Predict an alert when the score is at least the chosen threshold.
Pump	Score	Fault label
A	0.95	1
B	0.80	0
C	0.70	1
D	0.10	0

A
Score
0.95
Fault label
1
B
Score
0.80
Fault label
0
C
Score
0.70
Fault label
1
D
Score
0.10
Fault label
0

Predict an alert when the score is at least the chosen threshold.

Step 2 of 2

A lower threshold finds another fault and a false alert

The positive class is fault. Both thresholds use the same four pumps.

The team gains coverage of actual faults at the cost of another unnecessary visit.
Threshold	Alerts	True alerts	Precision	Recall
0.90	1	1	1	0.5
0.70	3	2	0.6667	1

0.90
Alerts
1
True alerts
1
Precision
1
Recall
0.5
0.70
Alerts
3
True alerts
2
Precision
0.6667
Recall
1

The team gains coverage of actual faults at the cost of another unnecessary visit.

Two denominators, two questions

Each wire carries a set of decisions or their counts. Neither ratio can be calculated from the score alone.

Lower the threshold and follow the alerts

Connect score selection to the actual fault labels before calculating either ratio.

BOTIE

The detector needs to find faults without making every healthy pump an inspection.

Choose an example

Fault scores

Confirmed fault labels

Alert threshold

Alert report

{
  "alerts": 1,
  "precision": 1,
  "recall": 0.5
}

Follow the operations

1. Fault scores
1. 00.95
2. 10.8
3. 20.7
4. 30.1
2. Alert threshold
```
0.9
```
3. Accept scores at the threshold
1. 01
2. 10
3. 20
4. 30
4. Confirmed fault labels
1. 01
2. 10
3. 21
4. 30
5. Count true and false alerts
```
{
  "tp": 1,
  "fp": 0,
  "fn": 1
}
```

6. Measure precision and recall

{
  "alerts": 1,
  "precision": 1,
  "recall": 0.5
}

7. Result

{
  "alerts": 1,
  "precision": 1,
  "recall": 0.5
}

The fixtures illustrate threshold arithmetic. Deployment rates need representative observations and reliable labels.

1. Name the positive event before counting

Here a positive means a fault requiring inspection. A true positive is an alerted pump with that fault. A false positive is an alerted healthy pump. A false negative is a fault the detector did not flag.

The definition matters. If a record’s label means “inspection booked” instead of “fault confirmed”, we are evaluating a different prediction. Confirm how labels were collected and which failures might never receive a label.

At threshold 0.70:
TP = 2; FP = 1; FN = 0

A confusion matrix begins with a clear target and a credible observation process.

2. Ask about the alerts and the missed faults

Precision asks how many issued alerts were genuine. Recall asks how many actual faults were found. Counting all healthy pumps in the denominator would answer neither question.

We use null for an undefined ratio in the workshop and None in Python. With no alerts, there is no observed precision. With no actual positive labels, there is no observed recall. Libraries sometimes substitute zero or another convention; declare that choice when comparing results.

Precision = TP / (TP + FP) = 2 / 3
Recall = TP / (TP + FN) = 2 / 2 = 1

A score without its denominator can be misleading, especially in a small evaluation set.

3. Move through the thresholds together

Sort distinct score values from high to low. At each value, include every observation whose score reaches it. Tied scores enter together; their row order must not create an imaginary operating point.

Recall cannot decrease as we include more observations. Precision can rise or fall depending on which labels enter next. A precision-recall curve displays these operating points so we can inspect the tradeoff.

Scores: [0.95, 0.80, 0.70, 0.10]
Labels: [1,    0,    1,    0]
Thresholds: 0.95 → 0.80 → 0.70 → 0.10

Choose a threshold on development data. Report the frozen rule on an independent assessment instead of picking the best point after seeing final labels.

4. A curve summary still leaves a decision to make

Average precision, AP, weights precision by each increase in recall as the threshold falls. It summarises ranking across operating points. Trapezoidal area joins neighbouring points with straight lines and is a different summary. Name the calculation you report.

ROC curves compare true-positive rate with false-positive rate among actual negatives. They can be useful, but a small false-positive rate may still create many alerts when negatives greatly outnumber positives. Precision makes that workload visible.

For this ranked example:
Recall increments: [0.5, 0, 0.5, 0]
AP = 0.5 × 1 + 0 × 0.5 + 0.5 × (2/3) + 0 × 0.5
   ≈ 0.8333

AP does not choose a usable threshold, price a missed fault or establish calibration.

5. Check whether the queue fits the team

A threshold that finds every labelled fault may send more pumps than the team can inspect. Record alerts per day beside precision and recall. A capacity rule might take the highest scores within a shift; tied scores then need a declared operational policy.

Precision changes with prevalence, the proportion of actual faults. If healthy pumps become much more common while sensitivity and false-positive rate stay the same, the alert queue becomes less precise. A benchmark with balanced classes can misrepresent a rare-event deployment.

If sensitivity = 0.8, false-positive rate = 0.1, prevalence = 0.01:
Expected precision = (0.8 × 0.01) / ((0.8 × 0.01) + (0.1 × 0.99))
                   ≈ 0.0748

This calculation assumes those rates transfer to the new population. Measure whether they do.

6. Build the mask before the report

A NumPy comparison produces a Boolean array with one position per pump. Combining that mask with labels identifies true positives. Sum only after the masks express the intended set.

The challenge adds shape checks and explicit undefined-ratio handling.

Count selected faults

Inspect one threshold using a different set of pumps.

Python

import numpy as np
scores = np.array([.6, .4, .9])
y = np.array([0, 1, 1])
alert = scores >= .5
print(alert.tolist())
print(int(np.sum(alert & (y == 1))))

Output

[True, False, True]
1

The middle pump is a missed fault. The first is a false alert; selecting it does not make it positive.