Overfit: the exam the model wrote itself

Sixty labelled points to learn from, sixty fresh ones held back. One slider makes the model more flexible. Training accuracy will love you for dragging it right. Watch what the fresh points think.

The map · two classes, one boundary

class A (circles) class B (squares) misclassified hollow = held-out test point

The dial and the two exams

rigidflexible
–
train accuracy · data it has seen
–
test accuracy · data it hasn't

–

Train accuracy is a rigged exam: the model is marking questions it memorised. At full flexibility the boundary grows an island around every stray point and scores 100% on data it has seen while dropping to 73% on data it hasn't; and the rigged exam's advice ("more complexity!") points exactly the wrong way. This is why the first move of any honest evaluation is a held-out split, why Orange keeps Data Sampler upstream of Test & Score, and why one quoted accuracy figure should always get the question: measured on what? The dial here is k in a nearest-neighbour model; in a tree it is depth, in a network it is size and epochs. Same curve, different name.