Comprehensions, iterators, and generators

An iterable can produce an iterator. An iterator produces successive values and remembers its position. A generator is an iterator created by a generator function or generator expression. These distinctions determine whether data is replayable, lazy, finite, and safe to consume more than once.

The iteration protocol

iter(value) asks for an iterator. next(iterator) returns a value or raises StopIteration. A for loop performs this protocol for you.

values = ["a", "b"]
iterator = iter(values)

assert next(iterator) == "a"
assert next(iterator) == "b"

try:
    next(iterator)
except StopIteration:
    pass

A list is a replayable iterable because each iter(values) returns a fresh list iterator. An iterator returns itself from __iter__ and is normally one-shot.

iterator = iter([1, 2, 3])
assert iter(iterator) is iterator
assert list(iterator) == [1, 2, 3]
assert list(iterator) == []

Accept Iterable[T] when a function only loops once. Require a Collection[T] or Sequence[T] when it needs length, membership, indexing, or repeated passes. Materialize deliberately when replay is required.

Writing a custom iterable

from collections.abc import Iterator


class Countdown:
    def __init__(self, start: int) -> None:
        if start < 0:
            raise ValueError("start must be non-negative")
        self._start = start

    def __iter__(self) -> Iterator[int]:
        return iter(range(self._start, -1, -1))

Countdown is replayable because __iter__ returns a new iterator each time. Making the container itself track iteration state would prevent nested loops and surprise callers.

Generator functions

Any function body containing yield is a generator function. Calling it creates a generator without executing the body. Each next() resumes execution until the next yield:

from collections.abc import Iterator


def read_chunks(data: bytes, size: int) -> Iterator[bytes]:
    if size <= 0:
        raise ValueError("size must be positive")

    for offset in range(0, len(data), size):
        yield data[offset : offset + size]


assert list(read_chunks(b"abcdef", 2)) == [b"ab", b"cd", b"ef"]

Validation inside the generator body is delayed until iteration begins. If an invalid argument must fail at call time, use a normal outer function that validates and returns an inner generator.

yield from iterable delegates iteration and can receive a subgenerator's return value. Advanced generator methods—send, throw, and close—support coroutine-like communication, but asyncio is usually clearer for modern concurrency.

Lazy pipelines

from collections.abc import Iterable, Iterator


def active_names(rows: Iterable[dict[str, object]]) -> Iterator[str]:
    for row in rows:
        if row.get("active") is True:
            name = row.get("name")
            if isinstance(name, str):
                yield name.strip()

This pipeline processes one row at a time. Laziness can reduce memory and begin producing results early. It also defers computation, side effects, and errors until consumption. The generator may depend on a file or transaction that must remain open while it is consumed.

Do not return a generator tied to a resource that has already left its context:

from collections.abc import Iterator
from pathlib import Path


def read_lines(path: Path) -> Iterator[str]:
    with path.open(encoding="utf-8") as stream:
        for line in stream:
            yield line.rstrip("\n")

Here the with block remains active while the generator runs and closes when it finishes or is closed. Consumers should close partially consumed generators when prompt cleanup matters; wrapping consumption in contextlib.closing can make that lifecycle explicit.

Comprehensions

Comprehensions express mapping and filtering:

squares = [number * number for number in range(10) if number % 2 == 0]
by_length = {word: len(word) for word in {"pear", "fig", "plum"}}
initials = {word[0] for word in ["alpha", "atom", "beta"]}

Their evaluation order follows nested loops from left to right:

pairs = [
    (left, right)
    for left in range(3)
    for right in range(left)
]

This is equivalent to an outer loop over left and an inner loop over right. Comprehensions have their own scope in modern Python, so their loop variable does not leak into the containing scope.

A generator expression is lazy:

total = sum(number * number for number in range(1_000_000))

The surrounding function consumes values as produced. Parentheses may be omitted when the generator expression is the sole function argument.

Readability boundaries

Use a comprehension for one recognizable transformation with an optional simple filter. Switch to a named loop or generator when logic includes:

• Multiple side effects.
• Complex nested conditions.
• Error handling.
• State carried between iterations.
• More than two levels of nesting.
• A transformation that deserves a domain name.

Avoid comprehensions used only for side effects, such as [send(item) for item in items]. They allocate a throwaway list and disguise an imperative operation. Use a for loop.

Generator cleanup and returns

A generator's return value terminates iteration by raising StopIteration(value). Ordinary for loops discard that value. Letting StopIteration accidentally escape from inside a generator is converted to RuntimeError, preventing subtle early termination.

A generator's finally blocks run when it exhausts, is closed, or is finalized, but prompt finalization is not a portable resource guarantee. Use explicit contexts for resources with important release timing.

Tradeoffs and pitfalls

• Lists support repeated access but allocate all results immediately.
• Generators bound memory usage but are one-shot and defer failures.
• itertools.tee creates independent views by buffering; a lagging consumer can cause large memory growth.
• Membership checks consume an iterator up to the match and alter future results.
• any() and all() short-circuit, which is efficient but means later side effects do not run.
• Mutating a collection while iterating over it can skip values or raise errors; iterate over a snapshot or build a new collection.
• Infinite iterables require bounded consumers such as itertools.islice.

Practice

1. Implement a replayable iterable without making it its own iterator.
2. Write a lazy pipeline that parses, filters, and transforms lines from a file.
3. Rewrite a three-level comprehension as a generator with named intermediate decisions.
4. Demonstrate how in, next, and list change the state of one iterator.