Collections and itertools

Built-in list, dict, set, and tuple should remain the default. collections adds containers for recurring patterns; itertools adds fast, lazy iterator combinators.

Specialized containers

Counter

from collections import Counter

counts = Counter("mississippi")
print(counts.most_common(2))
counts.update("miss")

Missing keys return zero. Counter arithmetic can discard zero and negative results depending on the operator, so inspect its semantics before using it as a general ledger.

defaultdict

from collections import defaultdict

groups: defaultdict[str, list[int]] = defaultdict(list)
for name, score in [("red", 3), ("blue", 4), ("red", 5)]:
    groups[name].append(score)

Reading a missing key mutates the mapping by creating a value. Use dict.setdefault() or dict.get() when that side effect is undesirable.

deque

from collections import deque

pending = deque(["a", "b"])
pending.append("c")
item = pending.popleft()

recent = deque(maxlen=100)

deque supports fast appends and pops at both ends. A list is still better for random indexing. A bounded deque silently discards items from the opposite end.

namedtuple, ChainMap, and abstractions

namedtuple() provides lightweight immutable tuple records, though dataclasses.dataclass is often clearer for evolving domain models. ChainMap overlays mappings without copying and is useful for layered configuration. Import abstract interfaces such as Mapping, Iterable, and Callable from collections.abc for runtime checks.

Iterator fundamentals

An iterable can produce an iterator. An iterator is stateful, returns itself from iter(), and is usually single-use. Materializing with list() consumes it.

values = (number * number for number in range(5))
print(sum(values))
print(list(values))  # empty: already consumed

Lazy pipelines reduce memory use, but errors occur during consumption rather than construction.

itertools building blocks

from itertools import batched, chain, islice

numbers = chain(range(3), range(10, 13))
first_four = list(islice(numbers, 4))
chunks = list(batched(range(7), 3))

Python 3.12 added itertools.batched(). It yields a shorter final tuple by default.

Common groups:

• selection: islice, takewhile, dropwhile, filterfalse;
• transformation: starmap, accumulate;
• combination: chain, zip_longest;
• grouping: groupby;
• combinatorics: product, permutations, combinations;
• infinite sources: count, cycle, repeat.

Always bound infinite iterators.

groupby groups adjacent values

from itertools import groupby
from operator import itemgetter

rows = [
    {"team": "blue", "score": 2},
    {"team": "red", "score": 5},
    {"team": "blue", "score": 7},
]
rows.sort(key=itemgetter("team"))

for team, group in groupby(rows, key=itemgetter("team")):
    print(team, [row["score"] for row in group])

groupby() does not aggregate all matching values globally; matching values must be adjacent. Each group shares the underlying iterator, so consume it before advancing the outer loop.

tee and shared consumption

itertools.tee() creates independent-looking iterators by buffering values consumed at different rates. The buffer can grow without bound. It is not thread-safe and is often worse than building a list when one consumer may run far ahead.

Selection tradeoffs

• Prefer comprehensions for a short, eager transformation.
• Prefer generators for a readable custom lazy process.
• Prefer itertools when standard combinators express the pipeline clearly.
• Prefer a loop when it makes state, exceptions, or early exits easier to understand.

Dense one-liners are not inherently more Pythonic.

Common mistakes and practice

• Reusing an exhausted iterator.
• Mutating a collection while iterating over it.
• Applying combinatoric functions to large inputs without estimating output size.
• Assuming groupby() performs database-style grouping.
• Forgetting that lazy file-backed iterators outlive the file context that created them.

Practice: stream (customer, amount) records already sorted by customer, group them, and emit totals in batches of 100. Then change the premise to unsorted input and explain the memory and sorting tradeoffs.