Yield in Python: How Generators Work Under the Hood

⏱️ Lectura: 13 min

A list of a million numbers loads all million into memory before you can use the first one. A function with yield doesn’t: it delivers a value, freezes right there, and only continues when someone asks for more. That controlled pause is the foundation of generators in Python, one of the most used and least understood mechanisms in the language.

📑 En este artículo

1. TL;DR
2. What Python Generators Are and Why They Matter
3. How Yield Works in Detail
4. Practical Examples: From Simple to Real
   1. Minimal Generator
   2. Unbounded Fibonacci
   3. Reading a Large File Line by Line
   4. Generator Expression: The Compact Version
5. Getting Started: Turning a Function into a Generator Step by Step
6. Real-World Use Cases
7. Common Mistakes and Best Practices
8. Comparison: List, Generator Expression, and Function with Yield
9. Going Deeper: send(), yield from, and Async Generators
10. Frequently Asked Questions
    1. What’s the difference between yield and return?
    2. Does a generator always use less memory?
    3. Can I convert a generator into a list?
    4. What is a generator expression?
    5. When does an async generator make sense?
    6. Are generators thread-safe?
11. References

This article explains exactly what the yield keyword does, how generators work under the hood in Python, and when it makes sense to use one instead of a list or a regular function.

TL;DR

• You’ll understand how yield pauses a function and resumes it exactly where it left off, without losing its state.
• You’ll be able to build your own generators to read huge files line by line without loading them fully into RAM.
• You’ll know how to use next() and send() to manually control a generator and the iteration protocol.
• You’ll be able to tell when a list, a generator expression, or a function with yield is the right choice.
• You’ll learn to avoid the mistake of iterating an exhausted generator and why it can only be used once.
• You’ll get to know yield from for delegating to subgenerators and simplifying data pipelines.
• You’ll understand the difference between a synchronous generator and an asynchronous one with async def and async for.

What Python Generators Are and Why They Matter

A generator is a function that, instead of returning a single result with return and finishing, delivers values one at a time with yield and suspends between each delivery. The function doesn’t run again from the top: it picks up exactly at the line after the last yield, with all its local variables intact.

The practical difference is memory and time. A function that builds a list with return has to compute and store every element before returning anything. An equivalent generator with yield produces the first element immediately and computes the rest only if someone asks for it. This is called lazy evaluation.

Generators in Python have existed since version 2.2, defined in PEP 255: Simple Generators. Today they underpin entire standard library modules, like itertools, and the iteration protocol itself that every for loop uses.

flowchart TD
    A["Function with return []"] --> B["Builds the full list"]
    B --> C[("Entire list in RAM")]
    D["Function with yield"] --> E["Generates one value at a time"]
    E --> F[("Only the current value in RAM")]

How Yield Works in Detail

When Python finds a yield inside a function’s body, that function stops being a regular function: it becomes a generator function. Calling it doesn’t execute a single line of its body; it only creates and returns a generator object.

def count_to(n):
    print("starting the generator")
    i = 1
    while i <= n:
        yield i
        i += 1

g = count_to(3)
print(g)

Running this code prints something like <generator object count_to at 0x...>. The print("starting the generator") hasn’t run yet: calling count_to(3) only creates the generator object, it doesn’t run the function body.

The body only starts running when someone requests the first value with the next() function:

print(next(g))  # starting the generator -> 1
print(next(g))  # 2
print(next(g))  # 3
print(next(g))  # StopIteration

Each call to next(g) runs the body until the next yield, delivers that value, and suspends there. When the function finishes, Python automatically raises the StopIteration exception, the same one a for loop uses internally to know when to stop.

💭 Key point: a for loop never calls next() manually: internally it does exactly next(iterator) on each pass and exits when it catches StopIteration. Generators aren’t a special case, they’re the mechanism that makes for work.

stateDiagram-v2
    [*] --> Created
    Created --> Suspended : first next()
    Suspended --> Running : next() or send()
    Running --> Suspended : yield delivers a value
    Running --> Exhausted : function ends
    Exhausted --> [*]

Practical Examples: From Simple to Real

Minimal Generator

The simplest possible generator delivers a single value and finishes:

def greeting():
    yield "hello from a generator"

for message in greeting():
    print(message)

The for loop calls next() for you, prints the message, and on the second pass gets StopIteration and exits the loop without raising an error.

Unbounded Fibonacci

A generator can represent an infinite sequence because it never computes it all at once:

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib = fibonacci()
first_10 = [next(fib) for _ in range(10)]
print(first_10)  # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

With a list this would be impossible: while True never ends, so a list would never be completed. With yield, each number exists only at the instant it’s requested.

Reading a Large File Line by Line

The most common production use case is processing files that don’t fit in memory:

def lines_containing(path, word):
    with open(path, encoding="utf-8") as file:
        for line in file:
            if word in line:
                yield line.strip()

for match in lines_containing("logs.txt", "ERROR"):
    print(match)

This function never holds the entire file in memory. Even if logs.txt weighs several gigabytes, the generator only keeps one line at a time, because iterating the file object with for line in file is already lazy by itself.

Generator Expression: The Compact Version

When the logic is simple, you don’t need to define a full function. A generator expression does the same thing between parentheses instead of brackets:

squares = (x * x for x in range(1_000_000))
total = sum(squares)
print(total)

Swapping the parentheses () for brackets [] would turn this into a list of a million integers built in memory before summing anything. With parentheses, sum() requests one value at a time and there’s never more than one integer computed at once.

💡 Tip: if you only need to iterate over the data once and the logic is simple, prefer a generator expression over defining a full function with yield: less code for the same result.

Getting Started: Turning a Function into a Generator Step by Step

To go from a regular function to a generator you don’t need to install anything; yield has been part of the language since Python 2.2. The steps are always the same:

1. Identify where your function builds a list with return, for example result.append(x) followed by return result.
2. Remove the intermediate list (result = [] and the .append).
3. Replace each result.append(x) with yield x.
4. Remove the final return result; a generator ends on its own once the function finishes executing.

# before: builds the entire list in memory
def evens_up_to(n):
    result = []
    for x in range(n):
        if x % 2 == 0:
            result.append(x)
    return result

# after: delivers the evens one at a time
def evens_up_to_gen(n):
    for x in range(n):
        if x % 2 == 0:
            yield x

To consume either version, the calling code doesn’t change: both work inside a for, with list(...), or with sum(...). You can confirm that evens_up_to_gen is a generator with:

import types
print(isinstance(evens_up_to_gen(10), types.GeneratorType))  # True

Real-World Use Cases

Generators show up anywhere the volume of data is large, unknown ahead of time, or potentially infinite:

• Data pipelines: chaining several generators (read file, filter, transform, aggregate) processes each row exactly once without materializing full intermediate results.
• API pagination: a generator can request the next page from an API only when the consumer asks for the next item, without fetching every page upfront.
• ORMs and querysets: frameworks like Django return objects that behave like generators to avoid loading an entire database table into RAM when iterating over it.
• itertools: the standard library’s itertools module is built almost entirely on generators, with functions like itertools.count(), itertools.chain(), and itertools.islice().
• Event streaming: systems that read from a socket or a message queue use generators to process events as they arrive, without waiting to have all the events.

Common Mistakes and Best Practices

The most frequent mistake is trying to iterate over a generator twice. Unlike a list, a generator is single-use:

numbers = (x for x in range(3))
print(list(numbers))  # [0, 1, 2]
print(list(numbers))  # [] , already exhausted

If you need to iterate over the same data multiple times, the fix isn’t to patch the generator: it’s to decide whether a list is the right choice in that case, or to recreate the generator from scratch by calling the function again.

⚠️ Watch out: calling next() on an already exhausted generator doesn’t reset anything, it keeps raising StopIteration forever. There’s no way to rewind a generator.

Another common mistake is mixing in return value inside a generator, expecting that value to come out through a for. In a generator, return value doesn’t deliver that value to whoever is iterating: it only ends the function, and that value becomes available as the .value attribute of the StopIteration exception, something designed specifically for yield from, not for regular use.

Best practices: name generator functions so it’s clear they return a lazy sequence, avoid opening expensive resources outside the generator if it might never be consumed, and use with inside the generator’s body so the resource closes even if the consumer stops iterating halfway through.

Comparison: List, Generator Expression, and Function with Yield

Option	When to use it	Advantage	Limitation
List	Small data iterated multiple times	Can be indexed, iterated N times, and used with len()	Entire content lives in memory at once
Generator expression	Single pass over large data, simple logic	Compact and lazy syntax	Exhausted after one use, doesn’t support indexing
Function with yield	Complex logic, multiple steps, or state across iterations	Lazy and reusable as a generator factory	Each call creates a new generator, no state shared between calls
Async generator	Data arriving from async I/O, sockets, or APIs	Doesn’t block the event loop while waiting for data	Can only be iterated with async for, not regular for

Going Deeper: send(), yield from, and Async Generators

A generator doesn’t just deliver values outward: it can also receive values from outside with send(). This was added in PEP 342: Coroutines via Enhanced Generators, included in Python 2.5, and it turned generators into the foundation of coroutines before async/await existed.

def accumulator():
    total = 0
    while True:
        value = yield total
        total += value

acc = accumulator()
next(acc)             # starts the generator, delivers total=0
print(acc.send(10))   # total is now 10, delivers 10
print(acc.send(5))    # total is now 15, delivers 15

Here yield total does two things at once: it delivers total outward, and when the generator resumes, that same expression evaluates to whatever value came through send(). That two-way channel is what sets yield as an expression apart from yield as a plain statement.

sequenceDiagram
    participant Main as Main program
    participant Gen as Generator
    Main->>Gen: next(acc)
    Gen-->>Main: yield total = 0
    Main->>Gen: acc.send(10)
    Gen-->>Main: yield total = 10
    Main->>Gen: acc.send(5)
    Gen-->>Main: yield total = 15

When a generator needs to delegate work to another generator, yield from avoids writing a manual forwarding loop. It was added in PEP 380, part of Python 3.3:

def inner_generator():
    yield 1
    yield 2

def outer_generator():
    yield from inner_generator()
    yield 3

print(list(outer_generator()))  # [1, 2, 3]

Finally, Python extended the same concept to the async world with PEP 525: Asynchronous Generators, available since Python 3.6. An async generator is defined with async def and iterated with async for instead of for:

import asyncio

async def read_pages(client, url):
    page = 1
    while True:
        data = await client.get(f"{url}?page={page}")
        if not data:
            break
        yield data
        page += 1

async def main():
    async for page in read_pages(client, "https://api.example.com/items"):
        print(page)

The difference from a regular generator is that each yield can be preceded by an await that hands control back to the event loop while waiting for a network response, without blocking the rest of the program.

To confirm in practice what kind of object you’re dealing with, the inspect module offers specific functions:

import inspect
print(inspect.isgeneratorfunction(fibonacci))    # True
print(inspect.isasyncgenfunction(read_pages))    # True

📖 Summary on Telegram: View summary

Your next step: open a Python terminal, write a generator that reads one of your own text files line by line and filters only the lines containing a keyword, and confirm with isinstance(generator, types.GeneratorType) that it’s genuinely lazy.

Frequently Asked Questions

What’s the difference between yield and return?

Return ends the function and delivers a single final value. Yield delivers a value and suspends the function without ending it, keeping all local variables intact for the next call to next().

Does a generator always use less memory?

It uses less memory when you don’t need all the values at once or when the sequence is very large or infinite. If you’re going to iterate over the same data multiple times or need to index it, a list is usually more practical.

Can I convert a generator into a list?

Yes, by wrapping it with list(generator). That consumes the generator entirely and stores all its values in memory, losing the benefit of lazy evaluation.

What is a generator expression?

It’s the compact version of a generator, written between parentheses instead of brackets, for example (x*x for x in data). It behaves the same as a function with yield but without needing to define a separate function.

When does an async generator make sense?

When each value depends on an input/output operation that takes time, like a network call or a database read, and you don’t want to block the rest of the program while waiting for that response.

Are generators thread-safe?

They’re not designed to be shared between threads: a generator maintains a single point of execution, and calling next() from multiple threads at the same time produces undefined behavior. Each thread should have its own generator.

References

• Stack Overflow: What does the yield keyword do in Python?: the question with thousands of votes that popularized the explanation of the generator protocol.
• Python Docs: Glossary, term generator: the official definition of generator and generator function in the Python documentation.
• PEP 255: Simple Generators: the original proposal that introduced yield in Python 2.2.
• PEP 525: Asynchronous Generators: the proposal that added async generators in Python 3.6.

📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day. @programacion

Imagen destacada: Foto de Safar Safarov en Unsplash