← All workshopsRead Knowing Which Model Produced an Answer

Deploying and Maintaining Models

Compare a candidate with the current version

Observe changes in opposite directions

Botie has predictions from a current model and a candidate on the same fixtures. He first needs to measure behaviour change.

BOTIE

The candidate differs by +2 and -2. What can their mean squared change establish?
The inputs we are comparing

Current-version fixture predictions

One changed result

  1. 04
  2. 12

Both changed

  1. 00
  2. 10

Candidate fixture predictions

One changed result

  1. 05
  2. 12

Both changed

  1. 02
  2. 1-2
Result for one changed result
0.5
Choose an explanation to test
Changes cancel in the mean but remain substantial

This check uses the pieces from “Measure candidate changes on aligned fixtures”.

BOTIE

Botie runs the current model and a candidate model on the same three requests. Measure the mean squared difference between their predictions. No ground-truth outcomes are supplied in this comparison.
Current-version fixture predictions
  1. 010
  2. 120
  3. 230
Candidate fixture predictions
  1. 013
  2. 117
  3. 230

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Which identifiers and fixtures would you save to reproduce this comparison later?

Behavioural regression checks and task-quality evaluations answer different version-review questions.

This is prediction change, not prediction error. It cannot choose the better model without a task criterion.

Work through the Python challenges →
Continue the courseChecking the Model After It Meets New Data →
← Keeping a Prediction Consistent Across Requests