Python Pandas DataFrames

In the last lesson, Introduction to Pandas, you built your first DataFrame and learned that it is a labeled table. So you can make a table now. But a table is only useful if you can look inside it and pull out the parts you care about. Real tables have thousands of rows, and you almost never want all of them at once. This lesson teaches you the everyday moves: inspect the table, pick the rows and columns you want, and filter for the data that matters.

🤔 Why these DataFrame skills matter

Imagine a table with ten thousand customers. You cannot print the whole thing and read it. You need to peek at the top few rows, ask how big it is, and grab just the columns and rows you care about, like “customers older than 30 in London”.

So the core skills are:

• Look at the table without printing all of it.
• Select specific columns and rows.
• Filter rows that match a condition.
• Add a new column built from the others.

These four moves cover most of what you do with data every day. Let’s set up one small table and use it for all the examples.

🧱 Our sample table

We will use this same DataFrame throughout the lesson. It is a small table of people with a name, age, city, and salary.

import pandas as pd

data = {
    "name": ["Alex", "Riya", "Arjun", "Maria", "Sam"],
    "age": [25, 30, 22, 35, 28],
    "city": ["London", "Mumbai", "Toronto", "London", "Mumbai"],
    "salary": [50000, 60000, 45000, 80000, 52000]
}

df = pd.DataFrame(data)
print(df)

    name  age     city  salary
0   Alex   25   London   50000
1   Riya   30   Mumbai   60000
2  Arjun   22  Toronto   45000
3  Maria   35   London   80000
4    Sam   28   Mumbai   52000

Calling the variable df is a common habit, short for DataFrame. We will keep using this df below.

👀 Looking at the table

When a table is large, you do not print all of it. Pandas gives you quick ways to peek and to ask about its shape.

This shows the most-used inspection tools:

print(df.head(2))      # first 2 rows
print(df.shape)        # (rows, columns)
print(df.columns)      # the column names

   name  age    city  salary
0  Alex   25  London   50000
1  Riya   30  Mumbai   60000

(5, 4)

Index(['name', 'age', 'city', 'salary'], dtype='object')

Here is what each one does:

• df.head(2) shows the first 2 rows. There is also df.tail(2) for the last rows. With no number, head() shows the first 5.
• df.shape gives (rows, columns). Our table is 5 rows by 4 columns.
• df.columns lists the column names, so you know what you have to work with.

There is also df.describe(), which gives a quick summary of the number columns: count, average, smallest, largest, and more. It is a fast way to get a feel for the data.

print(df.describe())

             age        salary
count   5.000000      5.000000
mean   28.000000  57400.000000
std     4.949747  13740.451594
min    22.000000  45000.000000
25%    25.000000  50000.000000
50%    28.000000  52000.000000
75%    30.000000  60000.000000
max    35.000000  80000.000000

🎯 Selecting columns

You already saw that picking one column gives a Series. To pick several columns, you pass a list of names inside the brackets, and you get a smaller DataFrame back.

This grabs one column, then two columns:

print(df["city"])                # one column, a Series

print(df[["name", "salary"]])    # two columns, a DataFrame

0     London
1     Mumbai
2    Toronto
3     London
4     Mumbai
Name: city, dtype: object

    name  salary
0   Alex   50000
1   Riya   60000
2  Arjun   45000
3  Maria   80000
4    Sam   52000

The key detail is the double brackets. df["city"] uses single brackets and returns one column as a Series. df[["name", "salary"]] uses a list inside the brackets, so you ask for several columns and get a DataFrame back. So double brackets mean “give me a table with these columns”.

🔢 Selecting rows with loc and iloc

To pick rows, Pandas gives you two tools that look similar but work differently.

• iloc selects by position number, starting at 0. The i stands for “integer position”.
• loc selects by label, the row’s index value.

This shows both picking a row:

print(df.iloc[0])       # the first row, by position
print(df.loc[2])        # the row whose index label is 2

name        Alex
age           25
city      London
salary     50000
Name: 0, dtype: object

name        Arjun
age            22
city      Toronto
salary      45000
Name: 2, dtype: object

In our table the index labels happen to be 0, 1, 2, so loc and iloc look the same here. The difference matters once you set custom labels, like names, as the index. Then iloc[0] still means “the first row by position”, while loc["Alex"] means “the row labelled Alex”. A simple way to remember: iloc is position, loc is label.

🔍 Filtering rows with a condition

This is the move you will use the most. You write a condition, and Pandas keeps only the rows where it is true.

Say you want everyone older than 28. You put the condition inside the brackets:

older = df[df["age"] > 28]
print(older)

    name  age    city  salary
1   Riya   30  Mumbai   60000
3  Maria   35  London   80000

Let’s unpack it, because it looks strange the first time:

• df["age"] > 28 checks every age and builds a column of True and False values. True for Riya and Maria, False for the rest.
• df[ ... ] then keeps only the rows where the value is True.

So the whole line reads as “from df, give me the rows where age is over 28”. You can combine conditions too. Use & for “and” and | for “or”, and wrap each part in its own brackets:

# People in Mumbai AND earning more than 55000
result = df[(df["city"] == "Mumbai") & (df["salary"] > 55000)]
print(result)

   name  age    city  salary
1  Riya   30  Mumbai   60000

➕ Adding a new column

You often build a new column from the ones you have. You just assign to a new name, and Pandas fills it down every row.

This adds a column for a 10% raise:

df["new_salary"] = df["salary"] * 1.1
print(df[["name", "salary", "new_salary"]])

    name  salary  new_salary
0   Alex   50000     55000.0
1   Riya   60000     66000.0
2  Arjun   45000     49500.0
3  Maria   80000     88000.0
4    Sam   52000     57200.0

The math df["salary"] * 1.1 runs on the whole salary column at once, the same vectorized style you saw with NumPy. Then df["new_salary"] = ... stores the result as a brand new column. No loop needed.

🌍 Quick reference

Here are the DataFrame moves from this lesson in one place.

⚠️ Common Mistakes

A few traps to avoid:

• Using single brackets for many columns. df["name", "salary"] raises an error. You need a list inside, so use double brackets: df[["name", "salary"]].
• Using and / or in a filter. Pandas needs & and | for row-by-row conditions, and each part needs its own brackets.

# ❌ The words and / or do not work here
df[df["age"] > 28 and df["city"] == "London"]   # ValueError

# ✅ Use & with brackets around each condition
df[(df["age"] > 28) & (df["city"] == "London")]

• Mixing up loc and iloc. iloc is for position numbers, loc is for index labels. Reach for iloc when you mean “the first row”, and loc when you mean “the row called X”.

✅ Best Practices

Habits that make DataFrame work smooth:

• Start with df.head() and df.shape on any new table, so you know what you are holding before you do anything else.
• Build filters one condition at a time and print the result, then combine them with & and | once each piece works.
• Use clear names for new columns, like new_salary, so the table stays readable.
• Reach for vectorized math like df["salary"] * 1.1 instead of looping over rows. It is faster and shorter.

🧩 What You’ve Learned

✅ head(), tail(), shape, columns, and describe() let you inspect a table without printing all of it.

✅ Single brackets pick one column as a Series, double brackets pick several columns as a DataFrame.

✅ iloc selects rows by position number, and loc selects rows by their index label.

✅ Filtering with df[df["age"] > 28] keeps only the rows where the condition is True.

✅ Combine conditions with & and |, wrapping each one in brackets, and add new columns by assigning to a new name.

🚀 What’s Next?

You can now inspect, select, and filter a table with confidence. But so far we typed the data by hand. Real data usually lives in a file. Next we learn to load a CSV file straight into a DataFrame, which is how most data projects begin.

Reading CSV Files