← All workshopsRead Which Alerts Deserve an Inspection?

Evaluating and Improving Models

Make the inspection queue measurable

BOTIE

Return the inclusive threshold mask before looking at the actual labels.
Ready to test
0 passed · 0 failed · 2 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

Return the inclusive threshold mask before looking at the actual labels.

The detector needs to find faults without making every healthy pump an inspection.

Inspect the component calculation
Accept scores at the threshold(
  Scores in [0, 1]: Fault scores
  Threshold: Alert threshold
)
Examples and results
Fault scores
  1. 00.95
  2. 10.8
  3. 20.7
  4. 30.1
Alert threshold
0.9
Required result
  1. 01
  2. 10
  3. 20
  4. 30
Fault scores
  1. 00.2
  2. 10.1
Alert threshold
0.8
Required result
  1. 00
  2. 10
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

The fixtures illustrate threshold arithmetic. Deployment rates need representative observations and reliable labels. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

Your last tested pieces can start the full construction. An existing draft there keeps its own pieces.

Alerts arrive on a shift with no confirmed faults

This check uses the pieces from “Follow a score into the inspection queue”.

BOTIE

Scores [0.8, 0.8, 0.2] belong to three healthy pumps. At threshold 0.8, predict alert count and which ratio is undefined.
Fault scores
  1. 00.8
  2. 10.8
  3. 20.2
Confirmed fault labels
  1. 00
  2. 10
  3. 20
Alert threshold
0.8

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

What additional measurement would tell you whether the team can use this threshold?

Keep alerts per shift and capacity beside precision and recall.

The fixtures illustrate threshold arithmetic. Deployment rates need representative observations and reliable labels.

Work through the Python challenges →
Continue the courseBetter for Which Cases? →
← What Are the Errors Trying to Tell Us?