Is the Model Wrong, or Is the Record Wrong?

Botie repeatedly predicts a fault for a pump labelled healthy. We inspect the record and discover that its label came from an earlier visit. Changing model settings would not fix that timestamp.

Other disagreements will be genuine model mistakes. We need a review process that can tell those cases apart without treating the model’s own opinion as the answer.

Keep eligible development records before applying the disagreement threshold.

A record may join a sensor reading to the wrong pump, an inspection to the wrong day, or a label to an event that was unavailable when prediction was supposed to happen. Inspect identifiers and timestamps before diagnosing model capacity.

Write the target in operational terms. “Fault present at inspection” differs from “maintenance performed within a month”. A team’s maintenance decisions may depend on the existing model, so those labels can reflect the policy as well as the condition.

An accurate calculation on misjoined records still answers the wrong question.

Exact duplicates can place the same example in training and assessment. Near duplicates, repeated patients and multiple views of one event can do the same less obviously. Group related observations before splitting when they share the information the model could memorise.

Deduplication itself needs a definition. Two pumps with equal temperature readings are not automatically duplicate events. Preserve event identity and domain context instead of dropping equal numeric rows indiscriminately.

A data audit can change the split plan, not only the label values.

A model fitted on an example can memorise its label, making agreement uninformative. Out-of-fold predictions come from models that did not train on the evaluated fold. They provide a more useful disagreement signal for development review.

Respect group and time boundaries inside those folds. Keep label review on the development side; do not feed final test answers back into training while retaining the old test score as independent evidence.

For a recorded label 0: disagreement score = p(fault)
For a recorded label 1: disagreement score = 1 - p(fault)

An overconfident model can produce strong but mistaken flags. The queue is a prioritisation aid.

Our teaching rule flags eligible records when disagreement confidence reaches a threshold above one half. Threshold inclusion is deliberate: a score equal to the threshold enters the queue.

Review records in a stable order and keep unique IDs. A real queue might combine disagreement with group coverage or expected correction value. Sample some unflagged cases too; otherwise you may never see errors the model confidently agrees with.

Record A: label 0, score 0.95, threshold 0.90 → review
Record B: label 1, score 0.95, threshold 0.90 → no disagreement flag

The rule does not estimate how many labels are wrong, and unflagged does not mean verified.

A reviewer should see the evidence needed for the target, with a written annotation policy. Record the previous label, new label if any, reason and data version. Where feasible, independent or blinded review reduces the chance of simply agreeing with the model.

Some cases remain ambiguous. Keep that status rather than manufacturing certainty. Agreement rates between reviewers, adjudication rules and sampling design help explain what the revised labels mean.

If a final evaluation label really is wrong, correct it transparently and report the changed assessment version. Do not silently tune on it and continue calling it untouched.

The review function returns IDs only. It should not rewrite labels, reorder source records or substitute predictions. That separation makes an accidental automatic relabelling harder to hide.

The challenge adds eligibility, threshold and duplicate checks around this inspection.

record = {"id": "P4", "label": 1, "p": 0.08}
disagreement = 1 - record["p"]
print(round(disagreement, 2))
print(record["label"])

0.92
1

After corrections, refit the same procedure first so the comparison can isolate the data revision. Record any additional feature or algorithm change separately. Keep prior difficult cases in regression tests and assess on evidence outside the correction process.

Report how many records were reviewed, corrected, retained and unresolved. A model may improve because labels became more consistent, because the task definition changed, or because difficult cases were removed. Those explanations have different implications.

A cleaner development dataset is useful. It still needs an assessment that represents the intended use.