Evaluating and Improving Models
Separate choosing from checking
The inner scores tie
Botie is tempted to choose a setting from the outer scores. Keep the selection inside its allowed data boundary.
BOTIE
The inputs we are comparing
Inner losses by fold and candidate
The outer scores tempt a different choice
| 1 | 2 |
| 3 | 1 |
Tied inner results
| 2 | 2 |
| 1 | 3 |
Outer losses by fold and candidate
The outer scores tempt a different choice
| 4 | 2 |
| 2 | 5 |
Tied inner results
| 7 | 1 |
| 3 | 2 |
Result for the outer scores tempt a different choice
4.5
The candidate order changes
This check uses the pieces from “Give the search an outside assessment”.
BOTIE
- Inner losses by fold and candidate
5 4 1 1 3 2 - Outer losses by fold and candidate
1 2 8 6 1 0
Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.
Build a construction in the workspace above first. This case will test those pieces.
If outer errors cause you to add another feature, what role have those observations now played?
Assessment data that guides a revision has become development evidence.
Recorded losses audit the protocol. A real nested run must also enforce split boundaries while fitting each complete pipeline.
Work through the Python challenges →