Section 5 of 24 · 9 lessons

Evaluating and Improving Models

The model has a score. What should we change next? Follow its individual mistakes, check which cases a revision helps, and keep the next assessment separate from the decisions that produced the model. Botie will help you test repairs against group errors and the conditions the service actually needs to handle.

0 of 9 finished

Find the Mistakes Behind the Score

An average can hide the mistakes we most need to understand. Keep predictions beside their observations, follow the inspection queue, and ask which conditions the model struggles with.

  1. 01What Are the Errors Trying to Tell Us?The temperature errors average to zero, but the greenhouse is still too hot some mornings. Keep each mistake beside the reading that produced it and work out what the average concealed.
    Build with Botie →Read the lesson
  2. 02Which Alerts Deserve an Inspection?A fault detector can look accurate while missing almost every fault. Follow the alerts and the faults separately, then choose a threshold the inspection team can actually use.
    Build with Botie →Read the lesson
  3. 03Better for Which Cases?Most readings come from the day shift. A model can improve its overall score while getting worse at night. Keep group sizes beside group errors before calling that a repair.
    Build with Botie →Read the lesson

Compare Revisions Fairly

A lower score may reflect easier cases or repeated searching. Compare revisions on shared observations and keep the model-choosing process inside its own assessment boundary.

  1. 04Did the Revision Help on the Same Cases?A lower score on an easier dataset tells us little about a revision. Match the cases, inspect their gains and regressions, then resample the paired observations to explore uncertainty.
    Build with Botie →Read the lesson
  2. 05Who Checks the Model-Choosing Process?Trying enough settings can produce an attractive validation score by chance. Put the search inside a separate assessment so the result includes the cost of choosing.
    Build with Botie →Read the lesson

Repair the Evidence and the Behaviour

Some failures begin in the records or the input adapter. Others need a person to review them. Work out which part needs repair, then test the change without hiding the remaining work.

  1. 06Is the Model Wrong, or Is the Record Wrong?A suspicious label deserves a closer look. Use out-of-fold disagreements to organise review while preserving original records and the boundary around the final assessment.
    Build with Botie →Read the lesson
  2. 07When Should the Model Ask for Help?Sending uncertain cases to a person changes both the error rate and the workload. Measure what the model handles and what it leaves for the review team.
    Build with Botie →Read the lesson
  3. 08Should This Change the Prediction?The same measurement can arrive in different units or a different batch. State the behaviour that should survive those changes, then test the actual prediction pipeline.
    Build with Botie →Read the lesson

Build and Check a Model Revision

Botie's temperature model misses the night shift. Fit a specific repair, compare it with the original, and check the next shift against requirements written before seeing its results.

  1. 09Repair the Night-Shift Model Without Breaking the Day ShiftDiagnose a missing condition, fit a revised predictor and compare it with the baseline. Botie then checks whether the overall gain survives group and runtime requirements on a fresh shift.
    Build with Botie →Read the lesson