Python Generators

In the last lesson you learned about Iterators and how Python walks through items one at a time using __iter__ and __next__. Writing a full iterator class is a lot of typing for something so simple. Generators are the shortcut. They give you an iterator with a normal-looking function and one new keyword.

🤔 Why Generators?

Say you want all the square numbers from 1 to ten million. The obvious way is to build a list and return it. But that list now sits in memory all at once. Every single number stays there, even if you only needed the first few.

That is the pain. Building the whole sequence up front wastes memory and time.

A generator fixes this by being lazy. It makes one value, hands it to you, then waits. It only makes the next value when you actually ask for it. Nothing is stored ahead of time.

🧩 What Is a Generator?

A generator is a function that produces a series of values one at a time, instead of returning them all together. You write it like a normal function. But you use the yield keyword instead of return.

Think of a vending machine. It does not drop all the snacks at once. You press a button, you get one snack, then it waits for the next press. A generator works the same way. Each request gives you one value, and the machine stays ready for the next one.

The key idea is that yield pauses the function. It hands back a value and freezes everything in place. When you ask for the next value, the function wakes up right where it stopped and keeps going.

🔁 yield vs return

This is the part that trips people up, so let us go slowly.

return ends the function. The function runs, gives back one result, and it is done. Call it again and it starts over from the top.

yield does not end the function. It pauses it. The function remembers its line number, its variables, everything. The next request continues from that exact spot.

Here is a tiny generator function that yields three values.

def count_up():
    print("starting")
    yield 1
    print("after first yield")
    yield 2
    print("after second yield")
    yield 3

gen = count_up()
print(next(gen))
print(next(gen))
print(next(gen))

starting
1
after first yield
2
after second yield
3

Notice what happened there:

• Calling count_up() did not run the body. It just made a generator object.
• The first next(gen) ran the code until the first yield 1, printed starting, then paused.
• The second next(gen) woke up after yield 1, printed after first yield, ran to yield 2, then paused again.
• Each next() resumes where the last one stopped. The function never restarts.

That pausing and resuming is what makes generators special.

🔂 Looping Over a Generator

You rarely call next() by hand. A for loop does it for you and stops cleanly when the values run out.

Here is a generator that yields the first n even numbers, used in a loop.

def first_evens(n):
    number = 0
    count = 0
    while count < n:
        yield number
        number += 2
        count += 1

for value in first_evens(5):
    print(value)

0
2
4
6
8

The loop asks the generator for one value, gets it, runs the loop body, then asks again. When the while finishes, the generator ends and the for loop stops on its own. You never had to track when to stop.

⚡ Why Laziness Saves Memory

This is the real payoff. Let us compare a list against a generator for a big range.

The list version builds every value first. The generator version makes them one at a time.

import sys

# ❌ Avoid: builds all ten million numbers in memory at once
big_list = [x * x for x in range(10_000_000)]
print("list size in bytes:", sys.getsizeof(big_list))

# ✅ Good: makes each square only when asked
big_gen = (x * x for x in range(10_000_000))
print("generator size in bytes:", sys.getsizeof(big_gen))

list size in bytes: 89095160
generator size in bytes: 200

The list holds ten million numbers, so it eats tens of megabytes. The generator holds almost nothing. It only remembers where it is, not every value. For huge sequences, this is a massive difference.

🪄 Generator Expressions

You just saw (x * x for x in range(...)). That is a generator expression. It looks exactly like a list comprehension, but with round brackets instead of square ones.

• List comprehension: [x * x for x in range(5)] builds a full list right away.
• Generator expression: (x * x for x in range(5)) builds a lazy generator instead.

Same words, different brackets, very different memory behavior.

Here is a generator expression summed up directly, which never builds a list at all.

total = sum(x * x for x in range(1, 6))
print(total)

55

When a generator expression is the only argument to a function like sum(), you can even drop the extra brackets. Python feeds values to sum() one at a time, adds them up, and no list is ever created.

📄 A Real Example: Reading a Big Log File

Generators shine when you process data that is too big to hold all at once. Imagine Riya has a server log with millions of lines, and she wants only the error lines.

This generator reads the file line by line and yields just the error lines, so the whole file is never loaded into memory.

def error_lines(filename):
    with open(filename) as file:
        for line in file:
            if "ERROR" in line:
                yield line.strip()

for line in error_lines("server.log"):
    print(line)

ERROR 2026-06-12 disk space low on node-3
ERROR 2026-06-12 failed login from 10.0.4.2
ERROR 2026-06-12 timeout reaching payment service

Because the file is read one line at a time and only error lines are yielded, this works the same whether the log has a thousand lines or a billion. Memory use stays flat. This pattern is how real data tools stream through files that are far larger than your computer’s memory.

⚠️ Common Mistakes

A few things catch people off guard with generators.

• A generator is used up once. After you loop over it fully, it is empty. Loop again and you get nothing.

gen = (x for x in range(3))
print(list(gen))   # [0, 1, 2]
# ❌ Avoid: the generator is already exhausted
print(list(gen))   # []  empty, not [0, 1, 2]

• Calling the function does not run it. Forgetting that count_up() only makes the object, not the values, leads to confusion.

# ❌ Avoid: expecting this line to print anything
count_up()
# ✅ Good: actually pull values out
for value in count_up():
    print(value)

• Mixing up the brackets. Square brackets give a list and use full memory. Round brackets give a lazy generator.

✅ Best Practices

Keep these habits in mind and generators stay easy.

• Reach for a generator when the sequence is large, or when you only need the values once.
• Use a generator expression instead of a list comprehension when you are just passing the values to sum(), max(), any(), or a for loop.
• If you need the values more than once, turn the generator into a list with list(gen) and reuse that.
• Give generator functions clear, action-style names like read_rows or error_lines so callers know they get a stream of values.

🧩 What You’ve Learned

✅ A generator is an easy way to make an iterator using a function with yield.

✅ yield pauses the function and hands back one value, then resumes where it left off, while return ends the function for good.

✅ Generators are lazy. They produce values one at a time, only when asked.

✅ This laziness saves a lot of memory on large or infinite sequences.

✅ A generator expression (x * x for x in range(n)) is the lazy cousin of a list comprehension.

✅ Generators are perfect for streaming through big files and data without loading everything at once.

🚀 What’s Next?

Now that you can make lazy iterators with yield, the next step is learning how to wrap and extend functions cleanly with one of Python’s most loved features.

Decorators