What does yield do in Python, and when is a generator better than returning a list?

A function with yield anywhere in its body is a generator function. Calling it runs none of the body; it returns a generator object. Each time something asks that object for the next value, whether a for loop, next(gen) or list(gen), the body runs until it reaches a yield, hands that value out and freezes, with all local variables and the current position preserved. The next request resumes on the line after the yield.

When the body finishes, the generator raises StopIteration, which a for loop treats as the end. Only one record exists at a time, which is why the memory use dropped.

The difference in numbers: sum([x * x for x in range(10_000_000)]) first builds a list of ten million results. The list alone holds ten million 8-byte pointers, about 80 MB, before counting the integer objects. sum(x * x for x in range(10_000_000)), a generator expression with parentheses instead of brackets, holds one value at a time, and the generator object stays tiny however many items it will produce.

Generators also chain into pipelines: lines = (l for l in f if l.strip()), then recs = (parse(l) for l in lines), then bad = (r for r in recs if r.error). Nothing is read until the final loop pulls items through, one line at a time. File objects are lazy line iterators themselves, so for line in f never loads the whole file.

Your empty second loop is expected: a generator is single-use. Once exhausted it stays exhausted, and there is no rewind. For two passes, call the generator function again to get a fresh generator, or materialise the data once with data = list(gen) if it fits in memory.

For the same reason len(gen) and gen[5] do not work; values do not exist until they are produced. itertools.islice(gen, 10) takes the first ten items. itertools.tee can split one generator into two iterators, but it buffers everything one consumer has seen and the other has not, so it saves nothing if you run one pass to completion before starting the other.

Use a generator when the data is large, when it is unbounded (reading a serial port forever with while True: yield ser.readline()), or when the consumer may stop early, since nothing beyond what is consumed gets computed. Return a list when the caller needs to index, sort, take len() or iterate several times and the data is small.

One caveat about timing: because the body runs lazily, errors and side effects happen during iteration, not at the call. A generator that opens a file does not open it until the first next(). If the consumer breaks out early, a with block inside the generator is only exited when the generator is closed or garbage-collected; call gen.close() or wrap it in contextlib.closing when clean-up must be prompt.

Two further features are worth knowing. yield from other delegates to another iterable or generator and yields every item of it. That keeps recursive generators short: a function that walks nested lists can yield from flatten(item) when an item is itself a list and yield item otherwise.

Also, yield is an expression: a caller can push a value in with gen.send(v), which turns the generator into a coroutine. That mechanism is what async and await grew out of; for new concurrent code use asyncio rather than hand-written send protocols. A return value inside a generator simply ends it, and the value becomes the result of the enclosing yield from expression.