Training, Validation and Test Sets: A Worked Example Without Data Leakage

There is a moment many applied AI learners hit: you train a model, it scores well on your test set, and then it disappoints when it meets real, new data. One possible cause is data leakage — when information that should be hidden from the model during training sneaks in, making your test score overly optimistic. Leakage is not the only reason models disappoint in production: distribution shift, sampling bias, overfitting, and differences between the training and deployment pipelines also matter. But leakage is one of the most preventable, and understanding the three-way split and a few common traps goes a long way.

The three sets and what each is for

The training set is what the model learns from — it sees these examples and their labels repeatedly. The validation set is what you use to tune decisions: which model type, which hyperparameters, which features to keep or drop. The model never trains on it, but you look at it many times while deciding how to build the model. The test set is used at the very end, to estimate how the model will perform on unseen data. The discipline is simple in principle: the test set is set aside until the final measurement, and you avoid adapting model decisions to its results.

A worked example: splitting a streaming dataset

Suppose you have 1,000 track releases, each with a label of whether it exceeded 10,000 streams in 14 days. A common illustrative split is 600 for training, 200 for validation, and 200 for test. You train on the 600, check on the 200 validation examples, adjust hyperparameters, retrain, check again — repeating as needed. Only when you are satisfied do you run the model once on the 200 test examples and report that score. Because the test set was not used to make decisions about the model, its score is a more honest estimate. The scikit-learn train_test_split function handles the mechanics of a split; the official documentation shows the exact usage (scikit-learn: train_test_split). Note that a simple random split is illustrative only. If your goal is to predict future releases, you need chronological boundaries, features available at prediction time, mature 14-day outcomes, and appropriate grouping so that repeated tracks or the same artist do not appear in both training and test — a random train_test_split does not resolve that time or group leakage on its own.

Where leakage actually hides

One common leakage is scaling before splitting. If you compute the mean and standard deviation of a feature across all 1,000 rows and then split, the training set's scaling has been informed by the test set's values. The fix is to fit the scaler on the training set only, then transform the validation and test sets with those training-set parameters. A second common leakage is duplicated rows that straddle the split: if the same track appears twice and one copy lands in training and one in test, the model has effectively seen the test example. Deduplicate before splitting. A third, subtler leakage is time: if your data has a time order and you split randomly, the training set may contain information that would not have been available at the test period. For time-ordered data, split chronologically — train on the past, test on the future.

Common mistakes and how to catch them

A red flag is a validation or test score that is suspiciously close to the training score — real models usually drop a little on unseen data, and a near-perfect match can be a sign of leakage or an unrealistically easy split. But similar train and test scores alone are not proof of leakage; they can also reflect an easy task or a small dataset. Another mistake is reusing the test set after seeing its score: "the test score was low, so let me try a different model and test again" turns the test set into a second validation set and weakens its independence. A practical safeguard is to write down, before you look at the test set, exactly what model and settings you are measuring — then run it once and accept the result. Re-running a deterministic report is fine; adapting model decisions to test results compromises independence and, if it happens, requires a fresh evaluation on new data.

An exercise: build a leakage-aware pipeline

Take a dataset and, in code, split off the test set first and store it in a separate variable you do not touch. From the remainder, split off a validation set. Fit any preprocessing (scaling, imputation) on the training portion only, and apply it to validation and test. Train a model, tune on validation, then run once on test. Write down the test score and do not re-run. If your data has a time order, split chronologically instead of randomly. The discipline of not touching the test set a second time is the whole point of the exercise.

Your next step

Once your split is honest, the next skill is reading the results correctly — especially when one class is rare and accuracy becomes a misleading number. That is the subject of the next article, on reading a confusion matrix with imbalanced classes. For the broader context of how this fits a structured AI curriculum, see the Applied AI program previews or the data analysis program.