Python API Data Fetcher

In the last lesson, To-Do Application, you built a polished everyday tool. Now we look at a skill that powers a huge part of real work: fetching data from an online service and saving it. Companies pull data from APIs all the time, to gather news, prices, user info, and more. The problem is, when you try this the first time, the program either hangs forever, crashes on a bad response, or blows up because one field was missing. So in this project we build a clean data fetcher that does it the safe way. It asks an API for some records, checks the reply carefully, picks out the useful fields, and saves them to a file.

🤔 Why fetch and save data?

A lot of useful data lives on other people’s servers, and you reach it through an API. An API is just a web address you can send a request to, and it sends data back, usually as JSON. You rarely want to fetch it once and forget it. You want to grab it, keep the parts you care about, and store them for later.

This fetch-and-save pattern shows up everywhere:

• A price tracker that grabs product prices each day and saves them.
• A news app that pulls the latest headlines and stores them to show offline.
• A report tool that collects data overnight so it is ready in the morning.

So the job has three steps. Fetch the data, pick out the fields you want, then save them to a file. We will use a free practice API that needs no key, so you can run this right away.

📦 Step 1: Fetching the data

We will use JSONPlaceholder, a free fake API made for practice. It returns sample data like users and posts, with no sign-up and no key needed. So you can run every line here yourself. We fetch a list of users.

This function gets the data and checks it worked:

import requests

def fetch_users():
    url = "https://jsonplaceholder.typicode.com/users"
    response = requests.get(url, timeout=10)

    if response.status_code != 200:
        print("Could not fetch data. Status:", response.status_code)
        return None

    return response.json()

Let’s read it line by line, because every line here is doing something real:

• url is just the web address we want data from. This one is the users endpoint of the practice API.
• requests.get(url, timeout=10) sends the request and waits for the reply. get means “please give me data from this address”. It hands back a Response object, which is the box that holds everything the server sent.
• response.status_code is a number the server sends to say how it went. 200 means success. So we check it is 200 before we trust the data. If it is something else, we print it and return None.
• response.json() reads the JSON text in the reply and turns it into normal Python data. Here that is a list of dictionaries, one dictionary per user.

So what is this Response object, really? Think of it like a parcel that came back. The status code is the note on the outside saying “delivered fine” or “address not found”. The body inside is the actual data. We open the box with .json().

And what is that timeout=10? That is important. It tells requests to wait at most 10 seconds for the server. Without it, if the server goes quiet, your program just sits there forever waiting. So always set a timeout. It is the difference between “the program is slow” and “the program is stuck”.

🧹 Step 2: Doing it the safe way with error handling

The version above works when everything is fine. But the internet is not always fine, right? The wifi drops. The server is down. The address has a typo. So a real fetcher has to expect trouble.

This is the same fetch, but written defensively:

import requests

def fetch_users():
    url = "https://jsonplaceholder.typicode.com/users"
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        return response.json()
    except requests.exceptions.RequestException as error:
        print("Request failed:", error)
        return None

Here is what each safety piece does:

• response.raise_for_status() is a shortcut. Instead of you checking status_code by hand, this line raises an error automatically if the status is a bad one, like 404 or 500. So you don’t have to write the if check yourself.
• The try block wraps the whole network call. If anything goes wrong inside, Python jumps straight to except instead of crashing.
• except requests.exceptions.RequestException catches the network problems. This is the parent error for the whole requests library. So a timeout, a dropped connection, a bad status, a DNS failure, they all land here. One catch covers them all.
• When it fails, we print the reason and return None, so the rest of the program can check for None and stop calmly.

So the rule is simple. Any time you talk to the network, wrap it in try / except, set a timeout, and check the status. These three habits stop almost every nasty crash.

✂️ Step 3: Picking out the fields you want

The API returns a lot of fields per user, more than we need. A good fetcher keeps only the useful ones. So we loop through the users and build a smaller, clean list.

This keeps just the name, email, and city of each user:

def clean_users(users):
    cleaned = []
    for user in users:
        cleaned.append({
            "name": user.get("name", "Unknown"),
            "email": user.get("email", "Unknown"),
            "city": user.get("address", {}).get("city", "Unknown")
        })
    return cleaned

Here is what happens:

• We start with an empty list, cleaned.
• For each user, we build a small dictionary with only the three fields we want.
• Notice we use user.get("name", "Unknown") instead of user["name"]. This matters a lot. If you write user["name"] and that key is missing, Python crashes with a KeyError. But .get("name", "Unknown") says “give me the name, and if it is not there, give me ‘Unknown’ instead”. So a single missing field will not blow up the whole run.
• user.get("address", {}).get("city", "Unknown") reaches into the nested address to grab just the city. The address is a dictionary inside the user. If address is missing, we fall back to an empty dictionary {}, then ask that for city, which is also missing, so we get "Unknown". No crash either way.
• We append each small dictionary and return the trimmed list.

So why bother with .get() instead of square brackets? Because real API data is messy. One record out of a thousand will have a missing field, and [] will crash on it while .get() just shrugs and moves on. Trimming the data is also good practice on its own. You save space and keep only what matters, instead of storing every field the API happened to send.

🔑 Step 4: Query parameters, headers, and API keys

JSONPlaceholder needs no key, which is great for learning. But most real APIs want two extra things: query parameters and an API key. Let me show both cleanly.

Query parameters are little settings you add to the request, like “give me page 2” or “only 5 results”. You could glue them onto the URL by hand, but that gets ugly fast. So requests lets you pass them as a dictionary with params:

import requests

def fetch_posts(user_id, limit):
    url = "https://jsonplaceholder.typicode.com/posts"
    params = {
        "userId": user_id,
        "_limit": limit
    }
    response = requests.get(url, params=params, timeout=10)
    response.raise_for_status()
    return response.json()

What is going on here:

• params is a plain dictionary of the settings we want.
• requests takes that dictionary and builds the final web address for you. So this quietly becomes .../posts?userId=1&_limit=5. You don’t have to glue the ? and & together by hand, which is where typos love to hide.
• _limit and userId are settings this particular API understands. Every API has its own list, so you check its docs.

Now, the API key. Many APIs want you to prove who you are, and you usually send that proof in a header. A header is extra info attached to the request, separate from the data. The catch is, an API key is a secret. You must never type it straight into your code, because then anyone who sees the file sees your key.

So we read it from an environment variable instead. An environment variable is a value stored outside your code, in your system. Here is the clean way:

import os
import requests

def fetch_with_key():
    api_key = os.environ.get("MY_API_KEY")
    if not api_key:
        print("No API key found. Set MY_API_KEY first.")
        return None

    headers = {
        "Authorization": f"Bearer {api_key}"
    }
    url = "https://api.example.com/data"
    response = requests.get(url, headers=headers, timeout=10)
    response.raise_for_status()
    return response.json()

Let’s read it:

• os.environ.get("MY_API_KEY") reads the key from the environment, not from the code. So the secret lives on your machine, never in the file you might share or push to GitHub.
• We check it is actually set. If not, we stop early with a clear message.
• headers is a dictionary. The common shape is Authorization: Bearer <your-key>. The word “Bearer” is just a standard label many APIs expect before the key.
• We pass headers=headers to the request, and requests attaches them.

Before running, you set the variable in your terminal. On Mac or Linux it is export MY_API_KEY="abc123". On Windows PowerShell it is $env:MY_API_KEY = "abc123". So the key is set for that session, and your code reads it without ever showing it.

💾 Step 5: Looping over results and saving to a file

Once you have a clean list, two everyday jobs are left. Print the records neatly so a human can scan them, and save them to a file so they are there tomorrow.

First, printing. We loop and pull out selected fields:

def print_users(users):
    for user in users:
        print(f"{user['name']:<25} {user['email']:<30} {user['city']}")

Here is the idea:

• We loop over every cleaned user.
• The :<25 part pads each name to 25 characters wide, so the columns line up neatly instead of being jagged. It just makes the output readable.

Now saving. We write the clean list to a JSON file:

import json

def save_to_file(data, filename):
    with open(filename, "w") as f:
        json.dump(data, f, indent=2)
    print(f"Saved {len(data)} records to {filename}")

The detail that matters here is indent=2. It tells json.dump to write the file with neat indentation, so a human can open and read it easily. Without it, the whole thing would be one long line. The len(data) in the message tells you how many records were saved, a nice confirmation.

🖥️ Step 6: The complete program

Here is the full fetcher, tying every step together with the safe error handling. Save it as fetcher.py and run it. You need pip install requests first if you have not already.

import requests
import json

def fetch_users():
    url = "https://jsonplaceholder.typicode.com/users"
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        return response.json()
    except requests.exceptions.RequestException as error:
        print("Request failed:", error)
        return None

def clean_users(users):
    cleaned = []
    for user in users:
        cleaned.append({
            "name": user.get("name", "Unknown"),
            "email": user.get("email", "Unknown"),
            "city": user.get("address", {}).get("city", "Unknown")
        })
    return cleaned

def print_users(users):
    for user in users:
        print(f"{user['name']:<25} {user['email']:<30} {user['city']}")

def save_to_file(data, filename):
    with open(filename, "w") as f:
        json.dump(data, f, indent=2)
    print(f"Saved {len(data)} records to {filename}")

def main():
    print("API Data Fetcher\n")
    users = fetch_users()
    if users is None:
        return
    cleaned = clean_users(users)
    print_users(cleaned)
    save_to_file(cleaned, "users.json")

main()

Here is a sample run:

API Data Fetcher

Leanne Graham            Sincere@april.biz              Gwenborough
Ervin Howell             Shanna@melissa.tv             McKenziehaven
Clementine Bauch         Nathan@yesenia.net            McKenziehaven
...
Saved 10 records to users.json

So the program fetched 10 users from the internet, kept only the name, email, and city of each, printed them in neat columns, then saved them to users.json. Open that file and you will see clean, readable records. That is a complete fetch-and-save pipeline, the same shape used in real data jobs.

🙏 Step 7: Being polite to the API

Here is a thing beginners forget. The API is on someone else’s server, and they pay for it. So be a good guest.

• Respect rate limits. A rate limit is the cap on how many requests you may send in some time window. If you hit a 429 status, you asked too fast. Slow down, add a small wait between requests.
• Cache when you can. Caching means saving a copy of data you already fetched, so you don’t ask again for the same thing. If the user list barely changes, fetch it once an hour, not once a second.
• Set a timeout, which you already do. It protects you and frees the server’s connection.

Plain version: don’t hammer the server, and don’t ask for the same data over and over. Save what you got and reuse it.

🛠️ Practice Challenge

Try these two extensions before peeking. They turn the toy fetcher into something closer to real work.

Challenge 1: Save the cleaned users to a CSV file instead of JSON, so it opens in a spreadsheet.

import csv

def save_to_csv(data, filename):
    fieldnames = ["name", "email", "city"]
    with open(filename, "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(data)
    print(f"Saved {len(data)} records to {filename}")

Challenge 2: Fetch several pages of posts and combine them into one list.

import requests
import time

def fetch_all_pages(pages):
    all_posts = []
    url = "https://jsonplaceholder.typicode.com/posts"
    for page in range(1, pages + 1):
        params = {"_page": page, "_limit": 10}
        try:
            response = requests.get(url, params=params, timeout=10)
            response.raise_for_status()
            all_posts.extend(response.json())
        except requests.exceptions.RequestException as error:
            print(f"Page {page} failed:", error)
            break
        time.sleep(0.5)
    return all_posts

⚠️ Common Mistakes

A few traps in this project:

• Forgetting the timeout. Without timeout=, a quiet server makes your program hang forever. Always set one.
• Using response["data"] style square brackets on data that might be missing a key. That throws a KeyError. Use .get() with a fallback instead.
• Not handling a failed request. If the service is down or the address is wrong, the status will not be 200. Use raise_for_status() or check the code before reading the data.
• Forgetting network errors. The connection can drop. Wrapping the call in try / except requests.exceptions.RequestException keeps the program from crashing.
• Hardcoding an API key in the file. Read it from an environment variable with os.environ so the secret never lives in your code.

✅ Best Practices

Habits this project teaches:

• Split the job into fetch, clean, print, and save functions. Each does one thing, so the pipeline is easy to follow and change.
• Always set a timeout= and wrap network calls in try / except.
• Check the response, either with raise_for_status() or by reading status_code, before using the data.
• Use .get() with a fallback when pulling fields out of JSON, so one missing key does not crash the run.
• Keep secrets out of code. Read API keys from os.environ.
• Be polite to the API. Respect rate limits and cache data you already have.

🧩 What You’ve Learned

✅ You built a fetcher that pulls data from an online API, checks it carefully, trims it, prints it, and saves it.

✅ requests.get(url, timeout=10) fetches the data and hands back a Response object, with status_code to confirm it worked and response.json() to turn the reply into Python data.

✅ Safe error handling means a timeout, raise_for_status(), and a try / except requests.exceptions.RequestException around the call.

✅ .get("field", fallback) pulls fields out of JSON without crashing on a missing key.

✅ params={...} adds query parameters cleanly, headers carry an API key, and you read that key from os.environ instead of hardcoding it.

🚀 What’s Next?

You can now fetch and store data from the web the safe way, the heart of many automation jobs. Next we build a file organizer script that tidies a messy folder automatically, sorting files into neat subfolders.

File Organizer Script