Keep a Useful Experiment Log for Your First AI Project

Here is a common experience in applied AI: you run an experiment, get a good result, change something, get a worse result, and weeks later you cannot reconstruct what produced the good result in the first place. The model that scored 0.87 is gone, replaced by one that scores 0.83, and you have no record of what was different. This is not a memory problem — it is a logging problem. A useful experiment log is a cheap, high-leverage habit, and few people start one early enough.

What a single experiment entry should contain

Every experiment is one row in your log. The minimum useful fields are: a unique run name or number, the date, the question you were trying to answer, the exact model and hyperparameters, the data split used (including split identifiers and random seeds), the preprocessing applied, the metrics on validation, and a one-line note on what you learned. Include references to data and code versions, the environment, and any artifact paths so a run can be reproduced. The "question" field is the one most people skip and the one that matters most — it forces you to state, before running, what you expect to learn. "Does adding artist-followers as a feature improve 14-day stream prediction?" is a real experiment; "try random forest" is not. During iteration, log validation metrics; reserve the test set for a single final evaluation, not for every row.

A worked example: a controlled comparison

Imagine a log kept as a simple spreadsheet. Row 1: run-001, question "baseline," model majority-class, split 800/200, validation accuracy 80%, note "baseline to beat." Row 2: run-002, question "does track tempo help," model logistic regression with tempo added, same split and seed, validation accuracy 81%, note "tempo adds one point." Row 3: run-003, question "does genre help," same logistic regression with genre one-hot added, same split and seed, validation accuracy 83%, note "genre adds two more points." Row 4: run-004, question "does a larger random forest beat the logistic regression," model random forest 200 trees, same split and seed, validation accuracy 82.5%, note "forest slightly worse; more complexity, no gain — stick with logistic." The key is that each row changes one thing in the same model family, so the gain can be attributed to that change — though you should still describe observed differences with sampling and seed uncertainty in mind, since small deltas can be noise. Keep the final test evaluation separate from this 800/200 train/validation illustration. Four rows, and the project's story is legible weeks later: which features helped, which model did not, and why.

Common mistakes that make logs useless

One mistake is logging only the wins. A log without the failed runs cannot teach you what does not work, which is most of what you need to know. Another is logging metrics without the configuration: "0.86 accuracy" means little if you did not record which features and hyperparameters produced it. A third is changing more than one thing between runs — if you add genre and switch models simultaneously and the score moves, you cannot tell which change caused it. The discipline of one change per run is what makes a log interpretable. A fourth is relying on memory or scattered notebook cells instead of a single, durable record; notebooks are for exploration, not for memory. Finally, do not log private records, personal data, or secrets in a log you intend to share — keep the log to reproducible configuration and aggregate metrics only.

An exercise: log your next three runs

Before your next three experiments, write down the question, the single change you are making, and your prediction of what will happen. After each run, record the actual validation metrics and whether your prediction held. The prediction column is the secret ingredient: it turns the log from a passive archive into an active tool for calibrating your own intuition about what works. You will quickly notice which kinds of changes you consistently over- or under-estimate, which is itself a valuable finding.

Tools versus discipline

There are good tools for experiment tracking, and they are worth adopting as projects grow. But the value comes from the discipline of recording, not from the tool. A spreadsheet kept faithfully beats a sophisticated tracking system used inconsistently. Start with the simplest format that captures the fields above, and only move to a heavier tool when the spreadsheet becomes painful to maintain. For a broader view of reproducible research practices, the official scikit-learn cross-validation documentation is a useful reference on how evaluation should be structured (scikit-learn: cross-validation).

Your next step

An experiment log turns a pile of runs into a legible story; the next habit that compounds with it is versioning your data and code together, so each log entry points at an exact, reproducible state. That is the subject of a later article in this series. For the structured learning path these habits belong to, see the Applied AI program previews or the data analysis program.