Python Data Cleaning Basics

In the last lesson, Reading CSV Files with Pandas, you learned to load real data from a file. So now you can get data in. But here is the hard truth about real data: it is almost never clean. There are blank cells, the same row entered twice, ages saved as text, and column names with random capital letters. If you analyze messy data, you get wrong answers. So before any real analysis, you clean. This lesson covers the cleaning moves you will use every single time.

🤔 Why clean data first?

There is an old saying in data work: “garbage in, garbage out”. If the data going in is messy, the result coming out is wrong, no matter how clever your code is.

Common problems hide in almost every real dataset:

• Missing values, which are empty cells where a number or word should be. Pandas shows these as NaN, short for “Not a Number”.
• Duplicate rows, where the same record got entered more than once.
• Wrong types, like an age stored as the text "25" instead of the number 25, so math fails.
• Messy column names, with spaces or odd capitals that make the columns annoying to use.

Cleaning means finding these and fixing them before you trust the numbers. Let’s set up a messy table and fix it piece by piece.

🧱 Our messy table

We will use this small, deliberately messy DataFrame for every example. Notice the None values and the repeated row.

import pandas as pd
import numpy as np

data = {
    "name": ["Alex", "Riya", "Arjun", "Maria", "Riya"],
    "age": [25, np.nan, 22, 35, np.nan],
    "city": ["London", "Mumbai", None, "London", "Mumbai"]
}

df = pd.DataFrame(data)
print(df)

    name   age    city
0   Alex  25.0  London
1   Riya   NaN  Mumbai
2  Arjun  22.0    None
3  Maria  35.0  London
4   Riya   NaN  Mumbai

See the trouble already. Riya’s age is NaN, Arjun’s city is None, and the Riya row appears twice. The age column also shows 25.0 with a decimal, because the missing value forced the column to become float. We will fix all of this.

🔍 Finding missing values

Before you fix blanks, you need to find them. isnull() marks every cell as True if it is missing. Adding .sum() counts the missing cells per column.

This counts the blanks in each column:

print(df.isnull().sum())

name    0
age     2
city    1
dtype: int64

So now you know exactly where the holes are. The “name” column is complete, “age” has 2 blanks, and “city” has 1. This single line is usually the first thing you run on any new dataset. It tells you how much cleaning is ahead.

🩹 Handling missing values

Once you find blanks, you have two choices: drop the rows that have them, or fill them with a sensible value.

The first option, dropna(), removes any row that has a missing value:

clean = df.dropna()
print(clean)

    name   age    city
0   Alex  25.0  London
3  Maria  35.0  London

Dropping is simple but it throws away whole rows, even the good parts. So use it when you have plenty of data and the messy rows are few.

The other option, fillna(), replaces blanks with a value you choose. A common choice for a number column is the average of that column:

average_age = df["age"].mean()
df["age"] = df["age"].fillna(average_age)
print(df)

    name   age    city
0   Alex  25.0  London
1   Riya  27.5  Mumbai
2  Arjun  22.0    None
3  Maria  35.0  London
4   Riya  27.5  Mumbai

Now the two blank ages are filled with 27.5, the average of the ages we did have. Filling keeps every row, which is why it is often preferred over dropping. For a text column you might fill with something like "Unknown" instead.

🧹 Removing duplicate rows

Duplicate rows quietly throw off your counts and averages. drop_duplicates() keeps the first copy of each repeated row and removes the rest.

This removes the repeated Riya row:

df = df.drop_duplicates()
print(df)

    name   age    city
0   Alex  25.0  London
1   Riya  27.5  Mumbai
2  Arjun  22.0    None
3  Maria  35.0  London

The second Riya row is gone, and the first one stays. Notice the index now skips 4, because that row was removed. By default Pandas compares whole rows, so two rows must match in every column to count as duplicates.

🔧 Fixing data types

Sometimes a number column comes in as text, usually because the file had a stray word or symbol in it. Math will not work until you fix the type. astype() converts a column to the type you want.

Say age came in as text. This turns it back into numbers:

import pandas as pd

scores = pd.DataFrame({"name": ["Alex", "Riya"], "age": ["25", "30"]})
print(scores["age"].dtype)        # object means text

scores["age"] = scores["age"].astype(int)
print(scores["age"].dtype)        # now int64

print(scores["age"].mean())

object
int64
27.5

The first print shows object, which is how Pandas labels text. After astype(int), the type is int64, a whole number, so .mean() works and gives 27.5. If you had tried .mean() while it was still text, it would not add up the way you expect.

✏️ Renaming columns

Messy column names, like "First Name" or "AGE " with a space, are annoying to type and easy to get wrong. rename gives them clean names.

This renames two columns:

import pandas as pd

df = pd.DataFrame({"First Name": ["Alex"], "AGE": [25]})
df = df.rename(columns={"First Name": "name", "AGE": "age"})

print(df.columns)

Index(['name', 'age'], dtype='object')

You pass a dictionary to columns, where each key is the old name and each value is the new name. Clean, lowercase names with no spaces make every later step easier to type and read.

🌍 Quick reference

The cleaning moves from this lesson in one place.

⚠️ Common Mistakes

A few traps that catch people:

• Expecting these methods to change the DataFrame in place. Most return a new, cleaned DataFrame and leave the original alone. You must save the result back.

# ❌ This computes a clean version but throws it away
df.dropna()

# ✅ Save the result back into df
df = df.dropna()

• Filling a number column with text, or the other way around. If “age” is numbers, fill its blanks with a number like the average, not with "Unknown".
• Forgetting that == does not find missing values. df["age"] == np.nan is always False. Use df["age"].isnull() to find blanks.

✅ Best Practices

Habits that keep cleaning under control:

• Start every dataset with df.isnull().sum() and df.head(), so you see the mess before you decide how to fix it.
• Prefer filling over dropping when you can, so you do not lose good data along with the bad.
• Fix column names and types early, right after loading. Clean names and correct types make everything after easier.
• Save the cleaned DataFrame back to a new CSV with to_csv, so you do not have to redo the cleaning every time.

🧩 What You’ve Learned

✅ Real data is messy, with missing values (NaN), duplicates, wrong types, and untidy column names.

✅ df.isnull().sum() counts the blanks in each column, and it is the first thing to run on new data.

✅ dropna() removes rows with blanks, while fillna() replaces them, often with an average or “Unknown”.

✅ drop_duplicates() removes repeated rows, and astype() fixes a column’s type so math works.

✅ rename gives columns clean names, and most cleaning methods return a new DataFrame, so save the result back.

🚀 What’s Next?

Your data is now clean and trustworthy. That is the hard part done. Next we put it all together and walk through a small, complete analysis, from loading a file to answering real questions about it.

Data Analysis Examples