Introduction to Python Generators

Python provides powerful ways to work with data sequences. Generators are one of the best tools for handling large datasets and creating patterns of data.

Unlike normal functions that give you all results simultaneously, generators hand you values one at a time. This saves memory even when working with massive amounts of data.

This article will show you how to understand and use Python generators to write more efficient, cleaner code.

Prerequisites

Before you start, make sure you have a recent version of Python (version 3.8 or higher) on your computer. You should already know the basics of Python functions and how loops work.

Step 1 — Understanding the problem with standard iteration

To get the most from this tutorial, set up a new Python project to try the examples yourself.

First, create a new folder and set up a virtual environment:

mkdir python-generators && cd python-generators

python3 -m venv venv

Activate the virtual environment:

source venv/bin/activate

Let's start by examining a common programming task - processing a sequence of numbers. Create a file named main.py with the following content:

def get_numbers(n):
    """Returns a list containing numbers from 0 to n-1."""
    result = []
    for i in range(n):
        result.append(i)
    return result

if __name__ == "__main__":
    import sys

    numbers = get_numbers(10)
    print(f"Numbers: {numbers}")
    print(f"Type: {type(numbers)}")
    print(f"Memory size: {sys.getsizeof(numbers)} bytes")

    # Try with a larger number
    large_numbers = get_numbers(1000000)
    print(f"Large numbers memory size: {sys.getsizeof(large_numbers)} bytes")

In this code, you define a function that builds a list of numbers from 0 to n - 1. It adds each number to the list individually, then returns the whole list.

This approach works fine for small numbers, but when you call it with something like one million, Python creates a complete list in memory. That can use up a lot of memory—especially if you don’t actually need all the numbers at once.

To see it in action, run the script:

python main.py

You'll see output like this:

Numbers: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Type: <class 'list'>
Memory size: 184 bytes
Large numbers memory size: 8448728 bytes

Look at how the memory usage jumps when you increase the list size. With just 10 numbers, the list uses only 184 bytes. But with a million numbers, memory usage shoots up to over 8 MB.

Creating the full result at once causes problems. It uses more memory, delays execution until the list is complete, and doesn’t handle large or infinite data well.

It's also inefficient if you only need part of the result. Generators solve these issues.

Step 2 — Introduction to generators with the yield keyword

Generators solve all these problems elegantly. Instead of creating a complete collection, generators make values one at a time, only when you ask for them.

Let's change our example to use a generator. Open the main.py file and modify it:

def generate_numbers(n):
    """Generates numbers from 0 to n-1."""
    for i in range(n):
        yield i

def get_numbers(n):
    """Returns a list containing numbers from 0 to n-1."""
    result = []
    for i in range(n):
        result.append(i)
    return result

if __name__ == "__main__":
    import sys

    # Test the list function
    numbers_list = get_numbers(10)
    print(f"List: {numbers_list}")
    print(f"List type: {type(numbers_list)}")
    print(f"List memory: {sys.getsizeof(numbers_list)} bytes")

    # Test the generator function
    numbers_gen = generate_numbers(10)
    print(f"Generator: {numbers_gen}")
    print(f"Generator type: {type(numbers_gen)}")
    print(f"Generator memory: {sys.getsizeof(numbers_gen)} bytes")

    # Create large versions
    large_list = get_numbers(1000000)
    large_gen = generate_numbers(1000000)
    print(f"Large list memory: {sys.getsizeof(large_list)} bytes")
    print(f"Large generator memory: {sys.getsizeof(large_gen)} bytes")

    # Demonstrate iteration
    print("\nIterating through generator:")
    for num in generate_numbers(5):
        print(f"Got number: {num}")

Here's what makes generators different from lists:

1. Your function uses yield instead of return to produce values
2. When you call the function, you get a generator object, not the actual values
3. The generator always uses the same tiny amount of memory (112 bytes), no matter if you generate 10 or 1,000,000 numbers
4. Values are created only when you ask for them during iteration

The yield keyword is the magic behind generators. When Python sees yield, it turns your regular function into a generator function. When you call this function, you get a generator object that you can loop through. Each time you ask for a value, the function runs until it hits a yield, gives you that value, then pauses until you ask for the next one.

The best part is how memory-friendly generators are. Whether you're working with 10 or 1,000,000 numbers, the generator itself always uses the same small amount of memory. It only calculates values when you need them, one at a time.

Run this script:

python main.py

You'll see output like:

List: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
List type: <class 'list'>
List memory: 184 bytes
Generator: <generator object generate_numbers at 0x10239ea80>
Generator type: <class 'generator'>
Generator memory: 200 bytes
Large list memory: 8448728 bytes
Large generator memory: 200 bytes

Iterating through generator:
Got number: 0
Got number: 1
Got number: 2
Got number: 3
Got number: 4

The output shows that the list stores all values at once, which increases its memory usage as the number of items grows. In contrast, the generator uses the same small amount of memory regardless of how many values it can produce.

When you loop through it gives you one value at a time on demand, making it much more efficient for large or infinite sequences.

Step 3 — Understanding generator state and execution flow

A key feature of generators is how they remember their state between yield statements. This allows them to pause execution and resume exactly where they left off.

The pause and resume behavior

Let's first look at a simple example that shows the basic pause-resume behavior. Create a file named basic_flow.py:

def simple_generator():
    print("First execution point")
    yield 1

    print("Second execution point")
    yield 2

    print("Third execution point")
    yield 3

    print("Generator complete")

# Create the generator
gen = simple_generator()
print(f"Generator created: {type(gen)}")

# Get first value
print("\nGetting first value...")
value = next(gen)
print(f"Received: {value}")

# Get second value
print("\nGetting second value...")
value = next(gen)
print(f"Received: {value}")

In this simple example, we create a generator that yields three values. Each yield statement is surrounded by print statements that show exactly when different parts of the code execute.

First, we create the generator object with gen = simple_generator(). This doesn't run any code inside the function yet - it just creates a generator object ready to be used.

When we call next(gen) the first time, the code runs until it reaches the first yield statement. It prints "First execution point", then yields the value 1 and pauses.

The second time we call next(gen), execution resumes right where it left off - after the first yield. It prints "Second execution point", then yields the value 2 and pauses again.

Run this script:

python basic_flow.py

You'll see output like:

Generator created: <class 'generator'>

Getting first value...
First execution point
Received: 1

Getting second value...
Second execution point
Received: 2

Notice how the "Second execution point" message only appears when we request the second value. This shows that the generator pauses execution after each yield and resumes from that exact point when you ask for the next value.

Generator exhaustion

Now let's see what happens when we try to get more values than the generator produces. Create a file named exhausted.py:

def three_values():
    yield "First"
    yield "Second"
    yield "Third"

gen = three_values()

# Get all values and one more
for _ in range(4):
    try:
        value = next(gen)
        print(f"Got: {value}")
    except StopIteration:
        print("Generator exhausted!")

This example shows what happens when a generator runs out of values. We create a generator that yields exactly three values, then try to get four values from it.

For the first three calls to next(gen), we get the expected values. But on the fourth call, there are no more yield statements to execute, so the generator raises a StopIteration exception. This is the standard way Python signals that an iterator has no more values.

Our code catches this exception and prints a message. This is how Python's for loops know when to stop iterating over a generator - they catch this exception internally.

Run this script:

python exhausted.py

You'll see:

Got: First
Got: Second
Got: Third
Generator exhausted!

Maintaining state between calls

The most powerful aspect of generators is their ability to maintain state between calls. Let's create a simple counter to demonstrate this:

def counter():
    count = 0
    while True:
        count += 1
        yield count

# Create counter
c = counter()

# Get first 5 values
for _ in range(5):
    print(next(c))

print("\nThe counter remembers its state!")

# Get next 3 values
for _ in range(3):
    print(next(c))

This example demonstrates how generators maintain their internal state between calls. The counter generator has a local variable count that keeps track of the current count. Each time we call next(c), it increments count and yields the new value.

This is powerful because the count variable persists between calls. After yielding a value, the generator pauses, but it remembers all its local variables. When we call next(c) again, it resumes with the same variable values it had when it paused.

Notice how we use while True to create an infinite generator. This pattern is common when creating generators that can produce an unlimited sequence of values.

Run this script:

python counter.py

The output shows the counter maintaining its state:

1
2
3
4
5

The counter remembers its state!
6
7
8

Notice how the counter continues from where it left off after the first loop. After yielding 5, it picks up and yields 6, 7, and 8. The count variable retains its value between calls to next(). This state preservation is what makes generators so powerful for many applications.

Let's see a practical example of using this state-preserving behavior with two-way communication:

def running_average():
    """Calculate running average of numbers."""
    total = 0
    count = 0

    # Prime the generator - first yield is just to start
    new_value = yield

    while True:
        # None is our signal to reset
        if new_value is None:
            total = 0
            count = 0
            new_value = yield "Reset complete"
            continue

        # Update running average
        total += new_value
        count += 1
        average = total / count

        # Yield the current average and get next value
        new_value = yield f"Average: {average:.2f}"

# Create and initialize the generator
avg = running_average()
next(avg)  # Prime the generator

# Send some values
print(avg.send(10))
print(avg.send(20))
print(avg.send(30))

# Reset
print(avg.send(None))

# Send more values
print(avg.send(100))
print(avg.send(200))

This example introduces the send() method, which allows you to pass values into a generator. It’s a more advanced pattern that enables two-way communication, where the generator both receives input and sends output.

The running_average generator maintains two variables: total and count, which it uses to calculate the average of the values you send in. The first yield statement (with nothing after it) is used to prime the generator. When you call next(avg), it runs up to that first yield and pauses, waiting to receive a value.

Once primed, the generator resumes where it left off each time you call avg.send(value). The sent value is assigned to new_value. If the value is None, the generator resets its state and yields a message confirming the reset. Otherwise, it updates the total and count, calculates the new average, and yields it.

The key behavior here is how new_value = yield ... works: the generator first yields a value, then pauses and waits for the next input, which is then assigned to new_value. This back-and-forth flow lets you interact with the generator like a stateful object, without needing a class.

This pattern is great for streaming calculations or incremental state tracking—anywhere you need to update values over time without resetting everything each time.

When you run the script, you’ll see:

python running_avg.py

You'll see:

Average: 10.00
Average: 15.00
Average: 20.00
Reset complete
Average: 100.00
Average: 150.00

The generator holds onto the total and count variables between calls, updating the running average each time you send in a new value. After sending None, it resets and starts fresh with new input.

This pattern is handy for things like real-time statistics, parsing large or streamed data, or elegantly managing internal state.

Here are the key takeaways from this section:

1. A generator pauses at each yield and resumes from that exact spot when continued.
   
2. When no more values are yielded, it raises a StopIteration exception.
   
3. It maintains its local variables between calls, preserving state across iterations.
   
4. With send(), you can pass data into the generator, enabling two-way communication.

These features make generators powerful tools for building efficient, stateful logic in a clean and compact way.

Final thoughts

This article demonstrates how Python generators are a powerful tool for writing clean, efficient, and memory-friendly code. With yield and send(), generators allow you to work with large or infinite sequences without loading everything into memory. They pause and resume execution while maintaining internal state, making them ideal for a wide range of applications.

For more details, check the official Python documentation on generators and PEP 255: Simple Generators.

Article by

Stanley Ulili

Stanley Ulili is a technical educator at Better Stack based in Malawi. He specializes in backend development and has freelanced for platforms like DigitalOcean, LogRocket, and AppSignal. Stanley is passionate about making complex topics accessible to developers.

Got an article suggestion? Let us know

Next article

Practical Guide to Asynchronous Programming in Python

Learn how to use Python's `asyncio` library to write efficient, concurrent code. This guide covers async functions, async generators, and semaphores, helping you handle multiple tasks concurrently for improved performance. Ideal for I/O-bound tasks and large datasets.

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.