Better for Which Cases?

Most readings come from the day shift. A model can improve its overall score while getting worse at night. Keep group sizes beside group errors before calling that a repair.

13 sections · 3 questions · 1 coding challenge

1 / 13

The arrow keys move between sections too.

A Model Report Needs to Say Who Was Measured

BOTIE

A single reported score can hide differences across populations and operating conditions. Model Cards proposed structured documentation of intended uses, evaluation conditions and disaggregated performance. That documented reporting proposal helps us ask where a model’s evidence actually applies.

We will use shifts in an invented greenhouse. The calculation is small enough to inspect, while the reporting question also applies to devices, locations and carefully defined population groups.

Sources for this history

Mitchell and colleagues: Model Cards for Model Reporting, 2019

The Day Shift Is Drowning Out the Night Shift

Botie’s revised temperature model improves on most readings. The night team disagrees: their predictions are worse. Both observations can be true if the evaluation contains far more daytime readings.

Let’s keep the same error calculation and change how we group its results. We want to see who benefits from the revision and where we still need evidence.

Invented readings. One mutually exclusive shift label per observation is used here; real evaluation slices can overlap.

Step 1 of 2

A smaller overall error can conceal a regression

The same four readings are used for both versions.

The candidate improves three daytime readings but doubles the night error.
Reading	Shift	Baseline absolute error	Candidate absolute error
A	day	2	1
B	day	2	1
C	day	2	1
D	night	2	4

A
Shift
day
Baseline absolute error
2
Candidate absolute error
1
B
Shift
day
Baseline absolute error
2
Candidate absolute error
1
C
Shift
day
Baseline absolute error
2
Candidate absolute error
1
D
Shift
night
Baseline absolute error
2
Candidate absolute error
4

The candidate improves three daytime readings but doubles the night error.

Step 2 of 2

Keep the sample size beside the metric

The aggregate weights each reading equally, so daytime receives most of the weight.

One night reading is a warning worth investigating, not a precise estimate of all nights.
Report	Count	Baseline MAE	Candidate MAE
All readings	4	2	1.75
Day	3	2	1
Night	1	2	4

All readings
Count
4
Baseline MAE
2
Candidate MAE
1.75
Day
Count
3
Baseline MAE
2
Candidate MAE
1
Night
Count
1
Baseline MAE
2
Candidate MAE
4

One night reading is a warning worth investigating, not a precise estimate of all nights.

The denominator determines whose errors dominate

The earlier residual calculation survives. Grouping changes which errors are averaged together.

Keep the night readings visible

Calculate one absolute error per reading, then aggregate within the supplied shift names.

BOTIE

The night team needs to see its mistakes even when daytime dominates the data.

Choose an example

Observed temperatures

Predicted temperatures

Shift for each reading

Errors by shift

[
  {
    "group": "day",
    "n": 2,
    "mae": 1
  },
  {
    "group": "night",
    "n": 2,
    "mae": 1
  }
]

Follow the operations

1. Observed temperatures
1. 010
2. 110
3. 210
4. 310
2. Predicted temperatures
1. 09
2. 19
3. 29
4. 39
3. Observed minus predicted
1. 01
2. 11
3. 21
4. 31
4. Keep the size of each error
1. 01
2. 11
3. 21
4. 31
5. Shift for each reading
1. 0day
2. 1night
3. 2day
4. 3night

6. Keep the error beside its group

[
  {
    "group": "day",
    "n": 2,
    "mae": 1
  },
  {
    "group": "night",
    "n": 2,
    "mae": 1
  }
]

7. Result

[
  {
    "group": "day",
    "n": 2,
    "mae": 1
  },
  {
    "group": "night",
    "n": 2,
    "mae": 1
  }
]

Counts and means alone do not quantify uncertainty, establish fairness or explain the cause of a group difference.

1. Keep the comparison on the same observations

Retain an ID, an observed outcome and both predictions for every evaluation case. Attach the shift from the original record. Compute absolute error before aggregating.

If the candidate’s report excludes a failed night prediction, its lower average may come from removing a difficult case. Count missing outputs separately and require the declared evaluation set to remain accounted for.

The workshop assumes complete aligned predictions. In a production report, missing results need their own explicit status.

2. Choose slices for a reason

A slice is a subset defined by a condition, such as night readings or a particular sensor model. Start with conditions that matter to the task and are available in the evaluation data. Record the definitions before comparing candidates.

Exploring additional slices can reveal a problem you did not anticipate. Mark that as a discovery to confirm on new evidence. Searching hundreds of tiny slices and presenting only the worst or best result can create misleading certainty.

Night slice: shift == "night"
Day slice: shift == "day"

For people-related groups, the labels, collection process and reporting choices need particular care. An error comparison alone does not establish a complete fairness claim.

3. Distinguish equal readings from equal groups

Overall MAE weights every observation equally. A macro average of group MAEs weights every group equally, regardless of how many observations it contains. Both can be useful, but they describe different target populations.

For disjoint groups covering the dataset, weighting each group MAE by its count recovers the overall MAE. Overlapping slices cannot simply be added this way because some observations would be counted more than once.

Candidate group MAEs: day = 1, night = 4
Observation-weighted MAE = (3 × 1 + 1 × 4) / 4 = 1.75
Equal-group MAE = (1 + 4) / 2 = 2.5

Name the averaging rule. Changing the weights is a decision about which population or cost the report represents.

4. Small slices need visible uncertainty

A group with one observation can produce a dramatic number without telling us how stable it is. Report the count and, where appropriate, an uncertainty interval based on the actual independent sampling units. Repeated readings from one greenhouse do not become independent buildings because they occupy separate rows.

Do not silently suppress a small slice’s failure. Mark the evidence as limited and decide what additional observations would help distinguish a persistent problem from one unusual case.

No observations means no measured performance. Display “not evaluated” rather than a perfect score.

5. Use a mask to preserve group membership

The groups array names the shift at each position. Comparing it with one group name creates a Boolean mask. Apply that mask to the error array, then reduce the selected values.

A Python dictionary preserves first-seen key order. We use that to keep reports predictable without hard-coding that only day and night can occur.

The reusable challenge must handle a new group name as well as a new number of readings.

Inspect the night slice directly

The example extracts a group before calculating its mean.

Python

import numpy as np
errors = np.array([1., 2., 5.])
groups = np.array(["day", "night", "night"])
selected = errors[groups == "night"]
print(selected.tolist())
print(float(selected.mean()))

Output

[2.0, 5.0]
3.5

The daytime error never enters the night denominator.

6. Turn a group regression into a testable repair

Inspect the night cases. Perhaps the heater behaves differently after sunset, or the sensor measures a different part of the room. A plausible missing feature gives us something specific to test.

Keep day performance in the comparison while developing the repair. Declare acceptable group regressions and minimum evidence before the final assessment. A tiny apparent gain should not automatically excuse a large error where the system is most needed.

The later model iteration project combines this report with paired gains and runtime requirements.