Prompting, Evaluation and Iteration
Decide whether the next shift can use this version
The score improves while approval is bypassed
The new maintenance assistant improves average answers, but Botie must still respect approval requirements and the technician’s waiting budget. A release decision needs all of those requirements.
BOTIE
The inputs we are comparing
Baseline case scores
Same in both cases
[
{
"caseId": "lookup",
"score": 0.5
},
{
"caseId": "approval",
"score": 0.5
}
]Candidate scores, milliseconds and critical flags
A better answer within the budget
[
{
"caseId": "approval",
"score": 1,
"latency": 800,
"critical": 0
},
{
"caseId": "lookup",
"score": 1,
"latency": 700,
"critical": 0
}
]A critical action hides behind better scores
[
{
"caseId": "approval",
"score": 1,
"latency": 800,
"critical": 1
},
{
"caseId": "lookup",
"score": 1,
"latency": 700,
"critical": 0
}
]Minimum mean score gain
Same in both cases
0
Maximum permitted case latency (ms)
Same in both cases
1000
Result for a better answer within the budget
1
The repaired candidate reaches the budget exactly
This check uses the pieces from “Keep release requirements independent until the final gate”.
BOTIE
- Baseline case scores
[ { "caseId": "manual", "score": 0.5 }, { "caseId": "approval", "score": 0.8 }, { "caseId": "missing", "score": 0.4 } ]- Candidate scores, milliseconds and critical flags
[ { "caseId": "missing", "score": 0.65, "latency": 900, "critical": 0 }, { "caseId": "manual", "score": 0.75, "latency": 650, "critical": 0 }, { "caseId": "approval", "score": 0.9, "latency": 800, "critical": 0 } ]- Minimum mean score gain
0.15
- Maximum permitted case latency (ms)
900
Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.
Build a construction in the workspace above first. This case will test those pieces.
Which input could change the release decision without changing any answer score, and why does a pass on these cases still require observation after rollout?
Version the system and evaluator, compare matched cases, and keep release policy explicit. Fresh failures should become replayable cases before claiming a repair.
This finite fixture gate is a policy exercise. It omits uncertainty intervals, population latency estimates, cost accounting and live deployment side effects.
Work through the Python challenges →