Repair the Night-Shift Model Without Breaking the Day Shift

Diagnose a missing condition, fit a revised predictor and compare it with the baseline. Botie then checks whether the overall gain survives group and runtime requirements on a fresh shift.

15 sections · 4 questions · 1 coding challenge

1 / 15

The arrow keys move between sections too.

A Fitted Equation Needs an Evaluation Record

BOTIE

Least-squares fitting provides coefficients that minimise squared training discrepancies for a specified linear model. It does not decide whether the inputs are appropriate or whether a revised model is useful for its intended population.

This project brings fitting together with explicit evaluation and reporting. The greenhouse is invented, and the small exact training relation makes the mechanics visible. We still keep the distinction between fitting a relation and collecting enough evidence to use it.

Sources for this history

The Same Heater Setting Produces Different Nights

We have enough evidence to try a specific repair. Botie’s temperature baseline uses the heater setting, but its night readings are consistently underpredicted. The night indicator is available before prediction, so we can test whether adding it helps.

Keep the original model as a reference. We will fit both versions using the same training records, compare their predictions on development data, and write down what the candidate must satisfy before a limited trial. The next shift will challenge those requirements.

Workshop predictions and runtime are supplied measurements for an invented comparison. The Python challenge actually fits both least-squares models on training data.

Step 1 of 3

The missing condition is visible in matched settings

The heater setting alone cannot distinguish these day and night records.

The night indicator is known at prediction time. These four exact values are a teaching construction, not a field dataset.
Training row	Heater	Night indicator	Temperature
A	0	0	10
B	1	0	12
C	0	1	14
D	1	1	16

A
Heater
0
Night indicator
0
Temperature
10
B
Heater
1
Night indicator
0
Temperature
12
C
Heater
0
Night indicator
1
Temperature
14
D
Heater
1
Night indicator
1
Temperature
16

The night indicator is known at prediction time. These four exact values are a teaching construction, not a field dataset.

Step 2 of 3

Carry a specific hypothesis into the comparison

Each boundary answers a different question.

1
Hypothesis
- Night conditions shift the mean
- Add the available night indicator
2
Development
- Fit both versions on training only
- Inspect shared-case and group errors
3
Assessment
- Freeze candidate and requirements
- Check fresh cases before a limited trial

A failed assessment that prompts another repair becomes development evidence for that repair.

Step 3 of 3

A better average can still fail a requirement

Both candidates are compared with baseline absolute errors of 2 on each of four readings.

The second candidate has the same average improvement but breaks the night requirement.
Candidate	Errors: day, day, day, night	Mean gain	Worst group increase
Local repair	1, 1, 1, 1	1	-1
Day-only gain	0, 0, 0, 4	1	2

Local repair
Errors: day, day, day, night
1, 1, 1, 1
Mean gain
1
Worst group increase
-1
Day-only gain
Errors: day, day, day, night
0, 0, 0, 4
Mean gain
1
Worst group increase
2

The second candidate has the same average improvement but breaks the night requirement.

A repair needs a record of what changed

The residual branches are reused, and the comparison preserves the group the average could conceal. Independent requirements remain independent at the final gate.

Give the revision a fair trial decision

Compute both error arrays from shared observations, inspect group changes, then enforce the declared policy.

BOTIE

The candidate must improve the task without hiding a group regression or missing its runtime requirement.

Choose an example

Observed temperatures

Baseline predictions

Candidate predictions

Shift per observation

Declared trial policy

minGain

maxIncrease

maxMs

Recorded candidate runtime in ms

Candidate decision and reasons

{
  "improvement": 1,
  "worstSliceIncrease": -1,
  "decision": "eligible for trial",
  "reasons": []
}

Follow the operations

1. Observed temperatures
1. 010
2. 110
3. 210
4. 310
2. Baseline predictions
1. 08
2. 18
3. 28
4. 38
3. Observed minus predicted
1. 02
2. 12
3. 22
4. 32
4. Keep the size of each error
1. 02
2. 12
3. 22
4. 32
5. Candidate predictions
1. 09
2. 19
3. 29
4. 39
6. Observed minus predicted
1. 01
2. 11
3. 21
4. 31
7. Keep the size of each error
1. 01
2. 11
3. 21
4. 31
8. Shift per observation
1. 0day
2. 1day
3. 2day
4. 3night

9. Compare overall and group losses

{
  "improvement": 1,
  "worstSliceIncrease": -1
}

10. Declared trial policy

{
  "minGain": 0.5,
  "maxIncrease": 0.25,
  "maxMs": 10
}

11. Recorded candidate runtime in ms
```
8
```

12. Apply the declared trial requirements

{
  "improvement": 1,
  "worstSliceIncrease": -1,
  "decision": "eligible for trial",
  "reasons": []
}

13. Result

{
  "improvement": 1,
  "worstSliceIncrease": -1,
  "decision": "eligible for trial",
  "reasons": []
}

The workshop checks recorded predictions and one runtime value. The Python task fits models; real deployment evidence also needs uncertainty, workload and monitoring.

1. Write the hypothesis before changing the code

Our hypothesis is that a missing night condition contributes to the residual pattern. The candidate adds a night indicator and keeps the rest of the fitting procedure unchanged. Save the dataset version, split IDs and baseline first.

An indicator can capture an association without identifying its physical cause. It may stand in for ventilation or other scheduled changes. If that schedule changes, the association may fail. The diagnosis gives us a useful experiment, not a causal proof.

Confirm the new feature is available at prediction time and uses the same definition in training and serving.

2. Represent both models with visible columns

The baseline has a constant column and heater setting. The candidate adds the night column. Each row is an observation; each coefficient multiplies its matching column.

NumPy column_stack combines the one-dimensional columns into those matrices. The leading ones let the fitted coefficient represent an intercept. The candidate design must have independent columns to identify all three coefficients in this exercise.

Baseline row:  [1, heater]
Candidate row: [1, heater, night]
Prediction = design row @ coefficients

Keep column names and order with the saved model. A two-column input matrix should not silently accept a swapped schema.

3. Fit on training records, then freeze the coefficients

linalg.lstsq returns a least-squares solution. We use its first returned item, the coefficient vector, and multiply a separate evaluation design matrix by those coefficients. Evaluation outcomes enter only after the predictions exist.

The model is fitted by squared error while this task reports MAE. That is deliberate: the fitting objective and the operational report need not be identical, but their difference should be stated.

For the exact training table:
Baseline coefficients: [12, 2]
Candidate coefficients: [10, 2, 4]
New rows: [[2, 0], [2, 1]]
Baseline predictions: [16, 16]
Candidate predictions: [14, 18]

The exact candidate fit was designed for teaching. Real data include noise and may not support this simple relation.

4. Use a small fitting program before the full comparison

The example below fits one intercept-and-slope model. Inspect the design matrix and prediction before building both versions in the challenge.

The reusable challenge returns predictions as well as scores. That allows a test to change evaluation labels and verify that the predictions do not move. If they do, the fitting step has leaked evaluation information.

Rank checks here produce a clear teaching failure instead of silently fitting an unidentified candidate.

Fit one frozen rule

This example uses one input feature and a different dataset. The challenge adds a candidate feature and evaluation checks.

Python

import numpy as np
x = np.array([0., 1., 2.])
y = np.array([3., 5., 7.])
design = np.column_stack([np.ones(len(x)), x])
weights = np.linalg.lstsq(design, y, rcond=None)[0]
print(weights.round(4).tolist())
print(round(float(np.array([1.,3.]) @ weights),4))

Output

[3.0, 2.0]
9.0

The new input uses the fitted coefficients. No outcome for that new input was needed to produce its prediction.

5. Compare the repair where it was meant to help

Calculate both absolute-error arrays on the same development observations. Report mean gain and group changes. A night repair that damages daytime performance deserves another look even if it meets an overall target.

Keep a ledger of attempts: hypothesis, changed feature or setting, observations used, outcome and next decision. Use paired uncertainty analysis at the appropriate independent unit when the data support it. Do not select one lucky seed and omit the other runs.

Group increase = candidate group MAE - baseline group MAE
Worst group increase = maximum over declared groups
Overall improvement = baseline MAE - candidate MAE

A negative worst-group increase means every observed group improved. Small or absent groups still limit the evidence.

6. Declare the policy before the fresh shift

For the workshop, require at least the specified overall gain, no group increase above its allowance, and runtime no greater than its limit. Preserve every failure reason. A faster candidate does not compensate for a failed error requirement unless the policy explicitly allows that trade.

Real runtime assessment needs a defined hardware and workload distribution, often including tail latency. A single supplied runtime in this fixture only teaches how an independent requirement enters a decision.

Teaching policy:
minimum MAE gain: 0.5
maximum group MAE increase: 0.25
maximum recorded runtime: 10 ms

Passing this small gate means eligible for a limited trial under these rules. It does not establish safety or justify automatic broad deployment.

7. Let a fresh shift challenge the repair

The investigation keeps the overall improvement but moves the candidate’s errors into the night group. Predict which requirement will reject it before running the construction. Then test the slow candidate, where errors improve but the runtime limit fails.

Retain both failures as regression cases. If either changes the model or policy, obtain new independent assessment data for the resulting candidate. Repeatedly repairing against one “fresh” set eventually makes it another development set.

The learner’s actual comparison graph must explain which requirement failed; a memorised decision cannot handle the changed cases.

8. Keep the evidence with the model that enters a trial

Save the feature schema, fitted parameters, data version, evaluation IDs and metric definitions. Record the trial limits and how to revert to the baseline. Monitor delayed outcomes as well as input changes, since a running service can look healthy while predictions degrade.

For language-model and agent revisions, the same discipline applies but task outcomes and stochastic runs need their own evaluators. The agent iteration section continues that problem with tool contracts, traces and reviewed cases.

A revision is easier to improve when we can reconstruct what changed and why we thought it would help.