pandas DataFrame Basics for AI Projects

Almost every applied AI project begins with loading data into a pandas DataFrame and inspecting it. Before any modelling, you need to see the shape, the column types, the first rows, and basic statistics. These operations are the foundation of every later step — cleaning, feature engineering, splitting, and modelling — and doing them carefully catches problems early.

Loading and first inspection

A DataFrame is a table with named columns and indexed rows. You load one with pd.read_csv for CSV files, then inspect with .head() to see the first rows, .shape to see the number of rows and columns, .dtypes to see column types, and .describe() for summary statistics of numeric columns. The official pandas getting started guide walks through these basics (pandas: getting started).

A worked example

Suppose you load a CSV of 1,000 track releases. df.shape returns (1000, 8) — 1,000 rows, 8 columns. df.dtypes shows that "tempo" is a float, "genre" is an object (string), and "release_date" is also an object — a sign you may need to convert it to a datetime. df.head() reveals that some rows have "NaN" in the prior_streams column, meaning missing values. df.describe() shows that the 14-day streams column has a mean of 6,200 and a maximum of 150,000 — a sign of a long-tailed distribution. Each of these observations leads to a concrete next step: convert the date, handle missing values, and consider a log transform for the target.

Selecting and filtering

You select a single column with df["tempo"] and multiple columns with df[["tempo", "genre"]]. You filter rows with a condition: df[df["genre"] == "pop"] returns only pop tracks. You combine conditions with & and |. These operations let you inspect subsets, which is how you find data quality issues that aggregate statistics hide.

Common mistakes

One mistake is skipping inspection and going straight to modelling, then discovering much later that a column was the wrong type or had widespread missing values. Another is using .describe() only on numeric columns and missing that a categorical column has 500 unique values where you expected 10. A third is modifying a slice of a DataFrame without .copy(), leading to a SettingWithCopyWarning and silent data corruption. The pandas user guide on indexing explains how to avoid this (pandas: indexing).

An exercise

Load a CSV and run .shape, .dtypes, .head(), and .describe(). Write down one observation from each that changes your plan. Then select one column and filter rows by a condition, and inspect the result. For the broader learning path, see the Applied AI program previews or the data analysis program.