Naive Bayes

The warning light was only one clue. Botie can also inspect whether a sensor's casing is warm and whether its readings fluctuate. A faulty sensor might produce both signs, but a working sensor can sometimes produce them too.

We could count every possible combination of signs in each group. As the list of signs grows, many combinations would have very few examples. Let's start with a simpler proposal: learn how often each sign appears within each group, then combine those contributions. We need to understand the assumption that makes that possible.

Faulty: 0.8 × 0.7 = 0.56
Working: 0.1 × 0.2 = 0.02

Each row describes the same observed signs under a different possible cause. Change one likelihood and follow its effect through the class weight and the final normalisation.

A training record contains the observed signs and a known class. For our sensor, a sign is either present or absent. Within the faulty group, count how many sensors show a warm casing; repeat for unstable readings. Then do the same within the working group.

The class frequencies supply priors. Each feature's frequency within a class supplies a conditional probability. Keep the feature order and class order fixed when these values become arrays.

Class order: faulty, working
Feature order: warm, unstable
Priors: [0.2, 0.8]
P(sign present | class):
  faulty  [0.8, 0.7]
  working [0.1, 0.2]

These are supplied illustrative rates. Estimating them from labelled observations belongs entirely inside the training split.

Let's make an assumption we can calculate with. Within each class, the model treats the features as conditionally independent. Once we specify faulty or working, learning that the casing is warm does not change the model's probability of an unstable reading. Across the whole population, the two signs can still be related because both may be more common in faulty sensors.

With that assumption, we can multiply the feature likelihoods for each class. Bayes' rule then brings in the prior class probabilities and lets us normalize the results. Later we will look at what happens when the signs are related even within a class.

Faulty contribution = 0.2 × 0.8 × 0.7 = 0.112
Working contribution = 0.8 × 0.1 × 0.2 = 0.016
Faulty posterior = 0.112 / (0.112 + 0.016) = 0.875

Correlated clues can be counted too strongly. A classifier can still rank classes usefully while giving probabilities that are too confident.

In the Bernoulli version, every feature has a present-or-absent outcome. If the casing is warm but the reading is stable, use the warm probability and the complement of the unstable probability. Simply dropping absent features changes the model.

A multinomial text model instead describes counts of events such as word occurrences. It does not reuse the Bernoulli formula unchanged.

Observed signs: warm = 1, unstable = 0
Faulty likelihood = 0.8 × (1 - 0.7) = 0.24
Working likelihood = 0.1 × (1 - 0.2) = 0.08

In the workshop, the supplied table already contains the probabilities of the outcomes actually observed. You combine its rows rather than guessing which features were present.

A feature absent from a small class-specific training sample would otherwise receive a zero estimated rate. One such factor can erase that class's entire likelihood. Smoothing adds a small count to each possible outcome before dividing.

For a binary feature with one added count for presence and one for absence, both the numerator and denominator change. This is an explicit estimation choice, not evidence that the unseen event really occurred.

Present count = 0; class examples = 4
Smoothed present rate = (0 + 1) / (4 + 2) = 1/6

With many features, add log likelihoods to avoid multiplying very small numbers. Fit smoothing choices on training and validation data, not the final test.

Compare this classifier with a simple class-frequency baseline. Inspect mistakes on new sensors, including cases where the two signs come from the same underlying measurement. Duplicating a feature can amplify its influence without adding information.

If we need reliable probabilities for inspection decisions, evaluate calibration as well as classification. A high posterior from an independence approximation is not a guarantee of correctness.

The calculation remains useful because every contribution is inspectable. Its simplicity also makes its assumptions visible.

import numpy as np
prior = np.array([0.2, 0.8])
present = np.array([[0.8, 0.7], [0.1, 0.2]])
x = np.array([1, 0])
terms = np.where(x == 1, present, 1 - present)
scores = prior * terms.prod(axis=1)
print(np.round(scores / scores.sum(), 4).tolist())

[0.4286, 0.5714]

Fitting produces a class prior and a table of feature probabilities. Prediction receives a new sign vector and reads those saved tables. It must not count the new batch to redefine what a faulty sensor looks like.

For binary signs, every coordinate contributes either the probability of presence or the probability of absence. Add log probabilities to form one score per class. Subtract the largest score before exponentiating and normalising, just as we did for stable probability calculations in NumPy.

The training table below uses warm casing and unstable readings as its two columns. Labels zero and one mean working and faulty. We add one count for each binary outcome inside each class.

The example shows both learned tables and a new prediction. Inspect those tables before trusting the label: a feature order error can preserve every shape while changing the meaning of the evidence.

import numpy as np
X = np.array([[0,0], [1,0], [1,1], [1,1]])
y = np.array([0,0,1,1])
rate = np.array([(X[y == label].sum(axis=0) + 1) / ((y == label).sum() + 2) for label in [0,1]])
prior = np.array([(y == label).mean() for label in [0,1]])
new = np.array([1,1])
score = np.log(prior) + (new*np.log(rate) + (1-new)*np.log1p(-rate)).sum(axis=1)
probability = np.exp(score-score.max())
probability /= probability.sum()
print(rate.tolist())
print(np.round(probability, 4).tolist())

[[0.5, 0.25], [0.75, 0.75]]
[0.1818, 0.8182]

If the unstable-reading feature is copied into a second column, the model multiplies its evidence twice. It treats the copy as another conditionally independent observation even though no new measurement occurred.

Try that change and compare probabilities before labels. The predicted class may stay the same while confidence becomes more extreme. This is why a useful classifier can still need calibration, and why a high probability is not a substitute for inspecting the feature construction.