Which Designs Are Actually Acceptable?

Botie has several candidate models for the inspection camera. One responds quickly but misses more faults. Another catches more faults but cannot respond before the part leaves the station. Which should we use?

We first need to separate requirements from preferences. A response deadline is a limit the design must meet. Among designs that meet it, we might prefer a lower operating cost. Mixing those two decisions into one score can hide an unacceptable compromise.

Change the latency or miss limits and inspect which candidate rows remain eligible.

Suppose the part leaves the camera station after a fixed time. That gives a response deadline. The team also chooses an acceptable miss limit for the evaluation. These requirements should be justified from the task, not adjusted merely so a favourite model passes.

Specify the metric itself. Mean latency and a high latency percentile answer different questions. A miss count needs an evaluation size and population. Without those definitions, a threshold can appear precise while comparing unlike evidence.

Our table uses the same benchmark statistic and the same 100 faulty parts for every candidate. That makes this small comparison interpretable.

For each candidate, compare latency with its maximum and misses with their maximum. A boolean mask records the candidates that pass one requirement. Combining the masks with AND keeps candidates that pass both.

In the workshop, use the limits as the first inputs of Threshold gates. Each gate returns 1 when its limit is at least the measured value. Multiply the resulting masks to require both conditions.

Latency = [60, 90, 120, 80] ms
Misses = [5, 3, 1, 4]
Latency at most 100: [1, 1, 0, 1]
Misses at most 4: [0, 1, 1, 1]
Feasible: [0, 1, 0, 1]

B and D remain. Returning a mask keeps their positions aligned with the candidate names.

import numpy as np
names = np.array(["small", "medium", "large"])
latency = np.array([40., 70., 130.])
print(names[latency <= 80].tolist())

['small', 'medium']

An empty feasible set is useful information. We may need to improve the implementation, collect better training data, change the operating process, or negotiate a requirement with the person responsible for it.

Do not quietly return the least bad candidate as if it passed. If a limit is a preference rather than an absolute requirement, label it that way and show the consequence of relaxing it.

A design review should be able to say “none of these yet”. That is more useful than a ranking that hides a failed requirement.

Among feasible candidates, say B dominates E if B is no worse on every objective and strictly better on at least one. Here, smaller latency and fewer misses are both preferred. A dominated candidate offers no improvement on either reported quantity.

Candidates that are not dominated form a Pareto set. B and D both remain because one is faster while the other misses fewer faults. This identifies the trade; it does not decide how much a miss is worth.

B = [90 ms, 3 misses]
E = [95 ms, 4 misses]
B is no worse in both columns and better in at least one
D = [80 ms, 4 misses]
D is faster than B; B misses fewer than D
Neither B nor D dominates the other

Equal rows do not strictly dominate each other. A duplicate may still differ in an unreported property such as maintainability.

Adding milliseconds to a miss count does not produce a naturally meaningful measure. A weighted score needs a stated preference model, including scales and the meaning of each weight. Changing units can change an unjustified score’s ranking.

An explicit consequence model, such as the inspection costs from the earlier lesson, gives a more interpretable comparison when the costs are defensible. Some consequences should remain hard limits instead.

A Pareto comparison can remove clearly inferior choices without inventing a conversion between units.

Latency varies, and a sample miss count does not reveal the exact future failure rate. Two close candidates may exchange places on another representative sample. Report uncertainty and relevant subgroups before treating small differences as settled.

Also check whether an omitted property changes the choice. Memory, a disconnected camera, or a requirement to explain decisions can make an apparently dominated model worth keeping.

Dominance is relative to the objectives and evidence we supplied. Python will calculate it exactly for a table, while we remain responsible for choosing the right table.

Save the candidate versions, benchmark procedure, requirements, and reasons for the choice. If the deadline changes, someone should be able to repeat the comparison rather than infer your preferences from a final model filename.

Start with feasibility, then examine the remaining compromises. Botie’s workshop builds the first step. The second Python challenge implements dominance on a small matrix so the comparison stays inspectable.

The functions deliberately keep all candidate positions. They return masks instead of silently renaming or reordering the designs.