Python Web Scraper Project

Table of Contents +

In the last lesson, File Organizer Script, you automated a chore on your own computer. Now we collect information from the web. Not every website has an API. Sometimes the data you want is just sitting on a page, meant for human eyes. Web scraping means writing a program that reads a web page and pulls out the data automatically. In this project we build a scraper that fetches a page and extracts the bits we want, using requests and a friendly tool called BeautifulSoup.

🤔 What is web scraping?

A web page is written in HTML, a language of tags that describes the page’s structure: headings, paragraphs, links, and so on. Your browser reads that HTML and draws the pretty page you see. A scraper reads the same HTML, but instead of drawing it, it digs out the data.

So web scraping is useful when:

  • A site shows data you want, but offers no API to get it cleanly.
  • You want to collect the same kind of info from many pages, like titles or prices.
  • You want to track something over time, like a list of headlines each day.

The job has two halves: fetch the page’s HTML with requests, then search that HTML for the parts you want with BeautifulSoup. Let’s meet the second tool.

🍲 The tool: BeautifulSoup

Raw HTML is messy and hard to search by hand. BeautifulSoup is a library that reads HTML and lets you find things in it easily, by tag name or other clues. It turns a tangle of tags into something you can search with simple commands.

It is third-party, so install it once. It comes in a package named beautifulsoup4:

Terminal window
pip install requests beautifulsoup4

Here is the idea on a tiny piece of HTML, so you see how it works before the real thing:

from bs4 import BeautifulSoup


html = "<html><body><h1>Hello</h1><p>Welcome to my page</p></body></html>"
soup = BeautifulSoup(html, "html.parser")


print(soup.h1.text)      # the text inside the first <h1>
print(soup.p.text)       # the text inside the first <p>

Output 

Hello
Welcome to my page

So BeautifulSoup(html, "html.parser") reads the HTML into a searchable object called soup. Then soup.h1.text reaches the first heading and pulls out its text. That is the whole idea: read the HTML, then ask for the parts you want.

🧩 Step 1: Fetching the page

First we fetch the page’s HTML with requests, the same way we fetched API data, and check it worked.

This function gets the HTML of a page:

import requests


def fetch_page(url):
    response = requests.get(url)
    if response.status_code != 200:
        print("Could not load the page.")
        return None
    return response.text

Notice we use response.text here, not response.json(). A web page is HTML text, not JSON, so we want the raw text. As always, we check status_code is 200 before trusting it. This returns the page’s HTML as a long string, ready for BeautifulSoup to read.

🔍 Step 2: Extracting the data

Now we feed the HTML to BeautifulSoup and pull out what we want. Say we want every heading and every link on the page. BeautifulSoup’s find_all finds every tag of a kind.

This function reads the HTML and collects the headings and links:

from bs4 import BeautifulSoup


def extract_data(html):
    soup = BeautifulSoup(html, "html.parser")


    headings = []
    for tag in soup.find_all("h2"):
        headings.append(tag.text.strip())


    links = []
    for tag in soup.find_all("a"):
        href = tag.get("href")
        if href:
            links.append(href)


    return headings, links

Let’s read it:

  • soup.find_all("h2") finds every <h2> heading tag on the page. We loop over them and keep each one’s text. .strip() removes extra spaces around it.
  • soup.find_all("a") finds every link tag. A link’s web address is in its href attribute, which we read with tag.get("href").
  • We check the href exists before adding it, because some link tags have none.
  • We return both lists.

So find_all is the workhorse: name a tag, get every one of them. From there you pull the text or an attribute like href.

🖥️ Step 3: The complete program

Here is the full scraper, tying it together with error handling. We use a site made for scraping practice so it is safe and allowed. Save it as scraper.py and run it.

import requests
from bs4 import BeautifulSoup


def fetch_page(url):
    response = requests.get(url)
    if response.status_code != 200:
        print("Could not load the page.")
        return None
    return response.text


def extract_data(html):
    soup = BeautifulSoup(html, "html.parser")
    titles = [tag.text.strip() for tag in soup.find_all("h1")]
    links = [tag.get("href") for tag in soup.find_all("a") if tag.get("href")]
    return titles, links


def main():
    url = "https://example.com"
    print("Web Scraper")
    try:
        html = fetch_page(url)
        if html is None:
            return
        titles, links = extract_data(html)
        print("\nHeadings found:")
        for t in titles:
            print("-", t)
        print(f"\nFound {len(links)} links.")
    except requests.RequestException:
        print("Network problem. Please try again.")


main()

Here is a sample run against example.com, a simple test page:

Output 

Web Scraper


Headings found:
- Example Domain


Found 1 links.

The scraper fetched the page, read its HTML, and pulled out the heading and the links, all on its own. Point it at a richer page and adjust the tags, and it will gather whatever data you need. That is the core skill of web scraping.

⚖️ Scraping responsibly

Web scraping comes with responsibility. A web page belongs to someone, so be a good guest.

Keep these rules in mind:

  • Check the site’s rules. Many sites have a robots.txt file and terms of service that say what scraping is allowed. Respect them.
  • Do not hammer a site. Sending hundreds of fast requests can overload the server. Go slow, and pause between requests.
  • Prefer an API when one exists. If the site offers an API, use that instead. It is cleaner and clearly allowed.
  • Do not steal content or personal data. Use scraping for learning and for data you are allowed to collect.

Caution

Always scrape politely and legally. Use practice sites like example.com or sites that clearly allow scraping while you learn. Sending too many requests too fast can get you blocked, and scraping private or copyrighted data can break the rules. When in doubt, look for an API or ask permission.

⚠️ Common Mistakes

A few traps in this project:

  • Using response.json() on a web page. A page is HTML, not JSON, so use response.text. json() would crash.
  • Assuming a tag always exists. If a page has no <h2>, find_all returns an empty list, and soup.h2.text on a missing tag crashes. Loop over find_all results, which is safe even when empty.
  • Scraping too fast or against the rules. Flooding a site or grabbing data you are not allowed to can get you blocked. Scrape slowly and respect each site.

✅ Best Practices

Habits this project teaches:

  • Fetch with requests and read response.text for HTML, then parse with BeautifulSoup.
  • Use find_all to collect every tag of a kind, and .get() to read attributes like href safely.
  • Check the status and handle network errors, since the web is unreliable.
  • Always scrape politely: respect the site’s rules, go slow, and prefer an API when one is available.

🧩 What You’ve Learned

✅ Web scraping reads a page’s HTML and pulls out the data you want, useful when a site has no API.

✅ requests fetches the page, and you read the HTML with response.text, not response.json().

✅ BeautifulSoup parses the HTML so you can search it; find_all("h2") returns every matching tag.

✅ You get a tag’s text with .text and its attributes, like a link’s address, with .get("href").

✅ Scrape responsibly: respect each site’s rules, do not overload servers, and prefer an API when one exists.

Check Your Knowledge

Test what you learned. Pick an answer for each question, then click Check.

  1. 1

    What is web scraping?

    Why: Web scraping means reading a page's HTML with code and extracting the data you want, useful when the site offers no clean API.

  2. 2

    Why do you use response.text instead of response.json() for a web page?

    Why: Web pages are HTML, so you want the raw text with response.text. response.json() expects JSON and would fail on HTML.

  3. 3

    What does soup.find_all("a") return?

    Why: find_all returns every tag of the given kind. find_all('a') gives all the link tags, which you can then read the href from.

  4. 4

    Which is a rule of responsible scraping?

    Why: Scrape politely: follow the site's rules and robots.txt, avoid overloading the server, and use an official API when one is available.

Score: 0 / 4

🚀 What’s Next?

You can now gather data from the open web. For the final project, we bring the course full circle and combine your skills with AI: building an AI chatbot in Python that talks back using a language model.

AI Chatbot with Python

Previous Python File Organizer Script Next AI Chatbot with Python

Share & Connect

Share on LinkedIn