Python Web Scraping Basics

Table of Contents +

In the last lesson you learned how to send mail straight from your code in Sending Emails with Python. Now let’s go the other way and read data from the web. A lot of useful information lives on web pages. But it does not come in a tidy file you can download. So how do you get it into your program? That is what web scraping is for.

🤔 Why Web Scraping?

Say Alex wants the title of every article on a news page. The data is sitting right there on the screen. But there is no button that says “give me this as a list.”

Copy-pasting by hand works for one page. It stops working the moment you have fifty pages, or the page updates every hour.

Web scraping fixes that. Your program reads a web page the same way your browser does. Then it picks out just the pieces you want.

These tools do most of the work:

requests fetches the page. It downloads the raw HTML, which is the code that describes the page.
BeautifulSoup reads that HTML and lets you pull out tags and text by name.

🧩 What HTML Looks Like

Before scraping, it helps to see what you are actually pulling apart. A web page is just text wrapped in tags. A tag is a label like <h1> or <p> that marks what a piece of content is.

Here is a tiny page:

<html>
  <body>
    <h1>Top Stories</h1>
    <p class="headline">Markets rise today</p>
    <p class="headline">New phone launches</p>
  </body>
</html>

See the pattern? The title sits inside an <h1> tag. Each headline sits inside a <p> tag with class="headline". Scraping is mostly about saying “find me the h1” or “find me every p with this class,” then reading the text inside.

📦 Installing the Tools

Both libraries are third-party, so they do not come with Python. You install them once with pip. Run this in your terminal:

pip install requests beautifulsoup4

Notice that the package is called beautifulsoup4. But inside your code you import it from bs4. That trips up a lot of people, so keep it in mind.

🚀 Fetching a Page

Let’s start by just downloading a page. This code asks a website for its HTML and prints the first part of it.

import requests

url = "https://example.com"
response = requests.get(url)

print(response.status_code)
print(response.text[:200])

Reading it line by line:

requests.get(url) sends a request to the site and waits for the reply. The reply is stored in response.
response.status_code is a number that says how it went. 200 means OK. 404 means the page was not found.
response.text is the full HTML of the page as one big string. We slice [:200] to print just the first 200 characters. That way it does not flood the screen.

Here is the sort of thing it prints:

Output

200
<!doctype html>
<html>
<head>
    <title>Example Domain</title>
    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html;

Note

That is example output. The exact HTML depends on the live site, so what you see will not match character for character. The shape is what matters.

Always check the status code before you trust the data. If it is not 200, the page did not load. Then there is nothing useful to read.

🔎 Parsing With BeautifulSoup

Right now response.text is one long messy string. You could search it by hand, but that is painful and breaks easily. BeautifulSoup turns that string into something you can search by tag.

This code parses the HTML and grabs the title of the page:

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
response = requests.get(url)

soup = BeautifulSoup(response.text, "html.parser")

title = soup.find("h1")
print(title.text)

Walking through the new parts:

BeautifulSoup(response.text, "html.parser") takes the raw HTML and builds a soup object you can search. The "html.parser" part just tells it which parser to use. This one comes built into Python.
soup.find("h1") returns the first <h1> tag it finds on the page.
.text pulls out the readable text inside the tag, without the angle brackets.

Here is the output:

Output

Example Domain

So <h1>Example Domain</h1> becomes just Example Domain. That is the clean piece Alex actually wanted.

🗂️ Getting Many Elements With find_all

find gives you one tag. But usually you want all of them, like every headline on the page. For that you use find_all, which returns a list of every matching tag.

Imagine a page with several <p> tags marked class="headline". This code grabs them all and prints each one’s text:

import requests
from bs4 import BeautifulSoup

url = "https://news.example.com"
response = requests.get(url)

soup = BeautifulSoup(response.text, "html.parser")

headlines = soup.find_all("p", class_="headline")

for headline in headlines:
    print(headline.text)

The key parts here:

soup.find_all("p", class_="headline") finds every <p> tag whose class is headline. It hands back a list.
Notice it is class_ with an underscore, not class. That is because class is already a reserved word in Python, so BeautifulSoup spells it class_.
We loop over the list and print .text for each tag. So we get one clean line per headline.

Here is example output for a news page like that:

Output

Markets rise today
New phone launches
City council votes on new park

Note

Again, this is representative output. A real news site changes its headlines all the time, so your run will show whatever is live right then.

That loop is the heart of most scraping jobs. Fetch the page, parse it, find the tags you care about, then read their text.

⏳ Scrape Slowly and Politely

This part matters as much as the code. When you scrape, you are using someone else’s server. If your script asks for hundreds of pages as fast as it can, you put load on them. And you might get blocked.

Caution

Always respect the site you are scraping. Keep these in mind:

Read the rules. Many sites have a robots.txt file (for example https://example.com/robots.txt) and terms of service that say what you may and may not collect. Check them first.
Go slow. Do not request page after page with no pause. Add a small delay between requests so you are not hammering the server.

Adding a pause is easy. This loop waits one second between pages using time.sleep:

import requests
import time
from bs4 import BeautifulSoup

pages = [
    "https://example.com/page1",
    "https://example.com/page2",
    "https://example.com/page3",
]

for url in pages:
    response = requests.get(url)
    soup = BeautifulSoup(response.text, "html.parser")

    title = soup.find("h1")
    print(title.text)

    time.sleep(1)  # wait 1 second before the next page

time.sleep(1) simply pauses the program for one second. So between each page, your script takes a short breath instead of firing requests back to back. It is a small thing that keeps you a good guest on someone else’s server.

⚠️ Common Mistakes

A few traps catch almost everyone the first time:

Importing the wrong name. You install beautifulsoup4 but import from bs4. import beautifulsoup4 will fail.

# ❌ Avoid: this name does not exist
import beautifulsoup4

# ✅ Good: import from bs4
from bs4 import BeautifulSoup

Using class instead of class_. Plain class is a Python keyword, so it errors out.

# ❌ Avoid: class is a reserved word
soup.find_all("p", class="headline")

# ✅ Good: use class_ with an underscore
soup.find_all("p", class_="headline")

Calling .text on nothing. If find does not match anything, it returns None, and None.text crashes. Check first.

# ❌ Avoid: crashes if there is no h1
title = soup.find("h1")
print(title.text)

# ✅ Good: make sure you found something
title = soup.find("h1")
if title is not None:
    print(title.text)

Forgetting to check the status code. If the page returned 404, parsing it gives you junk. Look at response.status_code before you parse.

✅ Best Practices

Small habits that make scraping smoother:

Check response.status_code is 200 before you parse the page.
Always test that find actually returned a tag before reading .text from it.
Add a delay between requests with time.sleep so you do not overload the server.
Read the site’s robots.txt and terms of service first, and only collect what you are allowed to.
When a site offers an official API, prefer that over scraping. An API gives you clean data and is the intended way in.

🧩 What You’ve Learned

You can now read data straight off a web page:

✅ Web scraping means your program reads a web page and pulls out the parts you want.
✅ requests.get(url) downloads a page, and response.status_code tells you if it worked.
✅ BeautifulSoup(response.text, "html.parser") turns raw HTML into something you can search.
✅ find returns the first matching tag, and find_all returns a list of every match.
✅ .text gives you the clean readable text inside a tag.
✅ Scrape politely: respect robots.txt and the terms, and add a delay so you do not hammer the server.

Check Your Knowledge

Test what you learned. Pick an answer for each question, then click Check.

🚀 What’s Next?

Web pages are one source of data. Another big one is documents, and PDFs are everywhere. Next you will learn how to read and create them from Python.

Working with PDFs

Previous Sending Emails with Python Next Python Working with PDFs

Share & Connect

Share on LinkedIn