Checking the Model After It Meets New Data

A model passed its release checks, but the new season brings different temperatures and a changed sensor firmware. The service still returns numbers quickly. That does not tell us whether they remain useful.

We need to look beyond whether the service responds. Are the inputs still valid and familiar? Once outcomes arrive, are the predictions still useful? Service failures and delays need their own checks. For each signal, decide what evidence we would inspect and what response it could justify.

Input changes alone do not reveal whether predictions became worse. Match the later outcomes to the saved predictions and measure their errors on the same records.

2

Validate input contracts first. Then inspect feature and prediction distributions, track outcome-based quality as labels become available, and measure service latency and failures. These monitors answer different questions.

A unit conversion can change input values while leaving requests syntactically valid. A latency spike can occur even when predictions remain accurate. Preserve enough context to distinguish them.

Give each alert a reason, owner or review route, and useful diagnostic evidence.

Choose reference and recent windows with comparable operating modes. Seasonal shifts may be expected, while a sudden missingness change after a firmware update may indicate a pipeline failure.

A difference in means is a simple signal, not a complete distribution test. It can miss changes in spread or subgroup composition. Set thresholds using expected variation and investigation costs.

Reference mean = 20
Recent mean = 25
Mean shift = 5

Distribution drift is not by itself evidence that prediction error increased.

Store stable identifiers, timestamps, and model versions so later outcomes can be matched to the predictions that preceded them. Avoid using an outcome that was already known at prediction time as an independent evaluation.

Calculate errors by time window and relevant subgroup. Label delays and selective feedback can bias a dashboard toward easy-to-observe cases.

The exercise uses already aligned predictions and outcomes; the join remains a real system responsibility.

import numpy as np
predicted = np.array([7., 8., 9.])
observed = np.array([6., 8., 11.])
errors = np.abs(predicted - observed)
print(errors.tolist())
print(float(errors.mean()))

[1.0, 0.0, 2.0]
1.0

If requests violate the input contract, follow the service's rejection or quarantine policy. If a new release makes predictions worse, restore a compatible earlier procedure and check it on the saved fixtures.

Retraining will not fix every change. A unit-conversion bug, for example, may affect the new training data too. Record what went wrong, keep the relevant artifacts, and verify the repair before promoting another version.

Botie’s workshop calculates observed error against later outcomes rather than treating drift as an accuracy label.

Latency, error rates and resource use tell us whether the service operates. Input distributions and missingness tell us whether the records changed. Outcome-based metrics tell us whether predictions remain useful once labels arrive.

Give each signal a reference period and a response owner. A spike in request failures calls for a different investigation from a gradual rise in forecast error. One dashboard number cannot distinguish these causes.

Retain the prediction, issue time and model version under a stable request or event identifier. When an outcome arrives, join it to that stored prediction. Recomputing with today's model answers a different question.

Do not treat unresolved outcomes as negative labels. Track label coverage and delay so the reported score's population is visible.

import numpy as np
prediction = {'a': 10., 'b': 20., 'c': 30.}
observed = {'a': 12., 'c': 27.}
matched = sorted(set(prediction) & set(observed))
error = [abs(prediction[k]-observed[k]) for k in matched]
print(matched)
print(float(np.mean(error)))
print(len(matched), len(prediction))

['a', 'c']
2.5
2 3

A shadow run produces candidate predictions beside the active model without letting the candidate control the decision. It can reveal input-contract, latency and prediction disagreements before a rollout.

A limited rollout then exposes the candidate to a declared share of eligible traffic. Keep assignment and outcomes so comparisons remain meaningful. Shadow predictions alone cannot measure how users respond to decisions the candidate never actually made.

Define the signals that trigger review or rollback, the person or process authorised to act, and the earlier compatible release. Test switching the full artifact and preprocessing together, then verify preserved fixtures and service health.

Record why the change occurred and which requests used each version. Rollback limits exposure to a suspected regression; it does not guarantee the earlier model is suitable after a genuine population change. Continue the investigation after service is restored.