Threshold Dial: is 96% accurate any good?

A thousand units came off the line today; fifty are faulty. A vision model scores every unit for fault-likelihood, and you choose the score at which a unit gets pulled for inspection. The headline accuracy number is watching you. Ignore it.

Today's thousand units, by fault score

0 · obviously fine100 · obviously faulty
good units, passed flagged for inspection the fifty real faults

The confusion matrix (the truth)

every unit lands in exactly one box
–
flagged & faulty · caught
–
flagged & good · false alarm
–
passed & faulty · shipped to a customer
–
passed & good · correct pass

The scorecard (the summary)

Headline accuracy–
Doing nothing would score95.0%
Faults caught–
Of every flag, this many are real–

What each mistake costs

A shipped fault (recall, warranty, trust) $
A false alarm (manual re-inspection) $
Today's total cost at this threshold–
–

Accuracy is a summary; the matrix is the truth. With 50 faults in 1,000 units, a dead detector scores 95% and the best possible headline is only 1.5 points better, so the headline cannot tell working from dead. The number that decides where the dial belongs is not on the scorecard at all: it is the price of each kind of mistake, and that is a business judgement, not a model property. When someone quotes you one accuracy figure, ask for the matrix, ask for the slice, and ask what a miss costs.