Generalization and Overfitting in Plain Language

The goal of training a model is not to score well on the data it was trained on — it is to score well on data it has never seen. That property is called generalization. Overfitting is its opposite: the model learns patterns specific to the training set, including its noise, that do not hold on new data. The gap between training and validation performance is the clearest signal of which one is happening.

What generalization means

A model generalizes when its performance on unseen data is close to its performance on training data. That means it has captured patterns that are real — that exist in the underlying process generating the data — rather than accidents of the particular sample. Generalization is never guaranteed; it is an estimate based on a held-out set, and it depends on the new data resembling the training data in distribution.

A worked example: the memorizing model

Suppose you have 100 tracks with a 14-day stream label. A deep tree that splits until each leaf has one example achieves near-zero training error: it has memorized each row. On 50 held-out tracks, its error is much higher, because the new tracks do not share the exact feature values of the training rows. The model has not learned the relationship between features and streams; it has learned the training set itself. A shallower tree that achieves moderate training error but similar validation error is generalizing better.

How to detect overfitting

The signal is a large gap between training and validation performance: low training error, much higher validation error. The scikit-learn learning-curve plot shows training and validation scores as data size changes, making the gap visible (scikit-learn: learning curves). If the gap is large, the model is overfitting; if both errors are high, it is underfitting.

Common mistakes

One mistake is reporting training accuracy as if it were a measure of generalization. Another is adding complexity until training error is near zero, which usually worsens validation error. A third is assuming generalization holds when the new data distribution differs from training — a model can generalize within its training distribution and still fail under distribution shift.

An exercise

Train a model and record both training and validation error. Increase model complexity step by step and plot both curves. Find the point where validation error stops improving while training error keeps dropping — that is where overfitting begins. For the broader learning path, see the Applied AI program previews or the data analysis program.