Generator Functions with yield - Python Programming

Generator functions use the yield keyword to produce values one at a time, creating iterators automatically without needing to implement __iter__() and __next__().

Basic Generator

                            limit
                            limit = 5 (default)limit = 3limit = 7
                        

                            basic.py
                        
Replay: real traced execution (multi-file project)

                            # Basic generator


def count_up(n):
    """Generate numbers from 0 to n-1"""
    i = 0
    while i < n:
        yield i
        i += 1


# Use generator in for loop
limit = 5
for num in count_up(limit):
    print(num, end=" ")

print("\n")

# Manual iteration
gen = count_up(3)
print(next(gen))  # 0
print(next(gen))  # 1
print(next(gen))  # 2

# Basic generator


def count_up(n):
    """Generate numbers from 0 to n-1"""
    i = 0
    while i < n:
        yield i
        i += 1


# Use generator in for loop
limit = 3
for num in count_up(limit):
    print(num, end=" ")

print("\n")

# Manual iteration
gen = count_up(3)
print(next(gen))  # 0
print(next(gen))  # 1
print(next(gen))  # 2

# Basic generator


def count_up(n):
    """Generate numbers from 0 to n-1"""
    i = 0
    while i < n:
        yield i
        i += 1


# Use generator in for loop
limit = 7
for num in count_up(limit):
    print(num, end=" ")

print("\n")

# Manual iteration
gen = count_up(3)
print(next(gen))  # 0
print(next(gen))  # 1
print(next(gen))  # 2


                        
1limit ← 5⋯12# Use generator in for loop13limit→ 5 = 514#@limit=3, 7⋯
2i ← 0pass 1 of 2⋯4def count_up(n5):5    """Generate numbers from 0 to n-1"""6    i→ 0 = 07    while i < n:⋯
3while i < n:pass 1 of 8⋯6i = 07while i0 < n5:8    yield i09    i += 1⋯All 8 passes — pass 1 is the card abovepassin
105
215
325
435
545
603
713
823
4i ← 1pass 1 of 5⋯7    while i < n:8        yield i09        i→ 1 += 1101112# Use generator in for loop13limit = 514#@limit=3, 715for num0 in count_up(limit5):16    print(num0, end=" ")⋯output0All 5 passes — pass 1 is the card abovepassnumi
100 → 1
211 → 2
322 → 3
433 → 4
544 → 5
5gen ← ⟨generator object count_up A⟩⋯18print("\n")1920# Manual iteration21gen→ ⟨generator object count_up A⟩ = count_up(3)22print(next(gen⟨generator object count_up A⟩))  # 023print(next(gen))  # 1⋯output
6i ← 0pass 2 of 2⋯4def count_up(n3):5    """Generate numbers from 0 to n-1"""6    i→ 0 = 07    while i < n:⋯
7i ← 1⋯7    while i < n:8        yield i09        i→ 1 += 1101112# Use generator in for loop13limit = 514#@limit=3, 715for num in count_up(limit):16    print(num, end=" ")1718print("\n")1920# Manual iteration21gen = count_up(3)22print(next(gen⟨generator object count_up A⟩))  # 023print(next(gen⟨generator object count_up A⟩))  # 124print(next(gen))  # 2⋯output0
0
8i ← 2⋯7    while i < n:8        yield i19        i→ 2 += 1101112# Use generator in for loop13limit = 514#@limit=3, 715for num in count_up(limit):16    print(num, end=" ")1718print("\n")1920# Manual iteration21gen = count_up(3)22print(next(gen))  # 023print(next(gen⟨generator object count_up A⟩))  # 124print(next(gen⟨generator object count_up A⟩))  # 2⋯output1
1
9print(next(gen))  # 2⋯23print(next(gen))  # 124print(next(gen⟨generator object count_up A⟩))  # 2⋯output2
1limit ← 3⋯12# Use generator in for loop13limit→ 3 = 314for num in count_up(limit):⋯
2i ← 0pass 1 of 2⋯4def count_up(n3):5    """Generate numbers from 0 to n-1"""6    i→ 0 = 07    while i < n:⋯
3while i < n:pass 1 of 6⋯6i = 07while i0 < n3:8    yield i09    i += 1⋯All 6 passes — pass 1 is the card abovepassi
10
21
32
40
51
62
4i ← 1pass 1 of 3⋯7    while i < n:8        yield i09        i→ 1 += 1101112# Use generator in for loop13limit = 314for num0 in count_up(limit3):15    print(num0, end=" ")⋯output0All 3 passes — pass 1 is the card abovepassnumi
100 → 1
211 → 2
322 → 3
5gen ← ⟨generator object count_up A⟩⋯17print("\n")1819# Manual iteration20gen→ ⟨generator object count_up A⟩ = count_up(3)21print(next(gen⟨generator object count_up A⟩))  # 022print(next(gen))  # 1⋯output
6i ← 0pass 2 of 2⋯4def count_up(n3):5    """Generate numbers from 0 to n-1"""6    i→ 0 = 07    while i < n:⋯
7i ← 1⋯7    while i < n:8        yield i09        i→ 1 += 1101112# Use generator in for loop13limit = 314for num in count_up(limit):15    print(num, end=" ")1617print("\n")1819# Manual iteration20gen = count_up(3)21print(next(gen⟨generator object count_up A⟩))  # 022print(next(gen⟨generator object count_up A⟩))  # 123print(next(gen))  # 2⋯output0
0
8i ← 2⋯7    while i < n:8        yield i19        i→ 2 += 1101112# Use generator in for loop13limit = 314for num in count_up(limit):15    print(num, end=" ")1617print("\n")1819# Manual iteration20gen = count_up(3)21print(next(gen))  # 022print(next(gen⟨generator object count_up A⟩))  # 123print(next(gen⟨generator object count_up A⟩))  # 2⋯output1
1
9print(next(gen))  # 2⋯22print(next(gen))  # 123print(next(gen⟨generator object count_up A⟩))  # 2⋯output2
1limit ← 7⋯12# Use generator in for loop13limit→ 7 = 714for num in count_up(limit):⋯
2i ← 0pass 1 of 2⋯4def count_up(n7):5    """Generate numbers from 0 to n-1"""6    i→ 0 = 07    while i < n:⋯
3while i < n:pass 1 of 10⋯6i = 07while i0 < n7:8    yield i09    i += 1⋯All 10 passes — pass 1 is the card abovepassin
107
217
327
437
547
657
767
803
913
1023
4i ← 1pass 1 of 7⋯7    while i < n:8        yield i09        i→ 1 += 1101112# Use generator in for loop13limit = 714for num0 in count_up(limit7):15    print(num0, end=" ")⋯output0All 7 passes — pass 1 is the card abovepassnumi
100 → 1
211 → 2
322 → 3
433 → 4
544 → 5
655 → 6
766 → 7
5gen ← ⟨generator object count_up A⟩⋯17print("\n")1819# Manual iteration20gen→ ⟨generator object count_up A⟩ = count_up(3)21print(next(gen⟨generator object count_up A⟩))  # 022print(next(gen))  # 1⋯output
6i ← 0pass 2 of 2⋯4def count_up(n3):5    """Generate numbers from 0 to n-1"""6    i→ 0 = 07    while i < n:⋯
7i ← 1⋯7    while i < n:8        yield i09        i→ 1 += 1101112# Use generator in for loop13limit = 714for num in count_up(limit):15    print(num, end=" ")1617print("\n")1819# Manual iteration20gen = count_up(3)21print(next(gen⟨generator object count_up A⟩))  # 022print(next(gen⟨generator object count_up A⟩))  # 123print(next(gen))  # 2⋯output0
0
8i ← 2⋯7    while i < n:8        yield i19        i→ 2 += 1101112# Use generator in for loop13limit = 714for num in count_up(limit):15    print(num, end=" ")1617print("\n")1819# Manual iteration20gen = count_up(3)21print(next(gen))  # 022print(next(gen⟨generator object count_up A⟩))  # 123print(next(gen⟨generator object count_up A⟩))  # 2⋯output1
1
9print(next(gen))  # 2⋯22print(next(gen))  # 123print(next(gen⟨generator object count_up A⟩))  # 2⋯output2

All 8 passes — pass 1 is the card above
pass	`i`	`n`
1	0	5
2	1	5
3	2	5
4	3	5
5	4	5
6	0	3
7	1	3
8	2	3

All 5 passes — pass 1 is the card above
pass	`num`	`i`
1	0	0 → 1
2	1	1 → 2
3	2	2 → 3
4	3	3 → 4
5	4	4 → 5

All 6 passes — pass 1 is the card above
pass	`i`
1	0
2	1
3	2
4	0
5	1
6	2

All 3 passes — pass 1 is the card above
pass	`num`	`i`
1	0	0 → 1
2	1	1 → 2
3	2	2 → 3

All 10 passes — pass 1 is the card above
pass	`i`	`n`
1	0	7
2	1	7
3	2	7
4	3	7
5	4	7
6	5	7
7	6	7
8	0	3
9	1	3
10	2	3

All 7 passes — pass 1 is the card above
pass	`num`	`i`
1	0	0 → 1
2	1	1 → 2
3	2	2 → 3
4	3	3 → 4
5	4	4 → 5
6	5	5 → 6
7	6	6 → 7

yield - a keyword that pauses the function and produces a value, resuming execution on the next iteration

How Generators Work

                            state.py
                        
Replay: real traced execution (multi-file project)

                            # State preservation


def fibonacci(n):
    """Generate first n Fibonacci numbers"""
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b  # state preserved across yields


# State is preserved
print("Fibonacci sequence:")
for num in fibonacci(10):
    print(num, end=" ")


                        
1print("Fibonacci sequence:")⋯12# State is preserved13print("Fibonacci sequence:")14for num in fibonacci(10):⋯outputFibonacci sequence:
2a ← 0, b ← 1⋯4def fibonacci(n10):5    """Generate first n Fibonacci numbers"""6    a→ 0, b→ 1 = 0, 17    for _ in range(n):⋯
3for _ in range(n):pass 1 of 10⋯6a, b = 0, 17for _0 in range(n10):8    yield a09    a, b = b, a + b  # state preserved across yields⋯All 10 passes — pass 1 is the card abovepass_a
100
211
321
432
543
655
768
8713
9821
10934
4a ← 1, b ← 1pass 1 of 10⋯7    for _ in range(n):8        yield a09        a→ 1, b→ 1 = b, a + b  # state preserved across yields101112# State is preserved13print("Fibonacci sequence:")14for num0 in fibonacci(10):15    print(num0, end=" ")⋯output0All 10 passes — pass 1 is the card abovepassnumab
100 → 11
2111 → 2
311 → 22 → 3
422 → 33 → 5
533 → 55 → 8
655 → 88 → 13
788 → 1313 → 21
81313 → 2121 → 34
92121 → 3434 → 55
103434 → 5555 → 89

All 10 passes — pass 1 is the card above
pass	`_`	`a`
1	0	0
2	1	1
3	2	1
4	3	2
5	4	3
6	5	5
7	6	8
8	7	13
9	8	21
10	9	34

All 10 passes — pass 1 is the card above
pass	`num`	`a`	`b`
1	0	0 → 1	1
2	1	1	1 → 2
3	1	1 → 2	2 → 3
4	2	2 → 3	3 → 5
5	3	3 → 5	5 → 8
6	5	5 → 8	8 → 13
7	8	8 → 13	13 → 21
8	13	13 → 21	21 → 34
9	21	21 → 34	34 → 55
10	34	34 → 55	55 → 89

When called, a generator function returns a generator object (an iterator).

Lazy vs Eager Evaluation

                            lazy_vs_eager.py
                        
Replay: real traced execution (multi-file project)

                            # Generator vs list


def eager_range(n):
    """Returns a list (all values in memory)"""
    result = []
    for i in range(n):
        result.append(i)
    return result


def lazy_range(n):
    """Returns a generator (values on demand)"""
    for i in range(n):
        yield i


# Eager: all values created upfront
print("Eager (list):")
nums = eager_range(5)
print(type(nums), nums)

# Lazy: values created on demand
print("\nLazy (generator):")
gen = lazy_range(5)
print(type(gen), gen)
print("Values:", list(gen))


                        
1print("Eager (list):")⋯18# Eager: all values created upfront19print("Eager (list):")20nums = eager_range(5)21print(type(nums), nums)⋯outputEager (list):
2result ← []⋯4def eager_range(n5):5    """Returns a list (all values in memory)"""6    result→ [] = []7    for i in range(n):⋯
3result ← [0]pass 1 of 5⋯6result = []7for i0 in range(n5):8    result→ [0].append(i0)9return result⋯All 5 passes — pass 1 is the card abovepassiresult
10[] → [0]
21[0] → [0, 1]
32[0, 1] → [0, 1, 2]
43[0, 1, 2] → [0, 1, 2, 3]
54[0, 1, 2, 3] → [0, 1, 2, 3, 4]
4return result⋯8    result.append(i)9return result[0, 1, 2, 3, 4]⋯
5nums ← [0, 1, 2, 3, 4], gen ← ⟨generator object lazy_range A⟩⋯19print("Eager (list):")20nums→ [0, 1, 2, 3, 4] = eager_range(5)21print(type(nums[0, 1, 2, 3, 4]), nums)2223# Lazy: values created on demand24print("\nLazy (generator):")25gen→ ⟨generator object lazy_range A⟩ = lazy_range(5)26print(type(gen⟨generator object lazy_range A⟩), gen)27print("Values:", list(gen⟨generator object lazy_range A⟩))⋯output<class 'list'> [0, 1, 2, 3, 4]

Lazy (generator):
<class 'generator'> ⟨generator object lazy_range A⟩
6def lazy_range(n):⋯12def lazy_range(n5):13    """Returns a generator (values on demand)"""14    for i in range(n):⋯
7for i in range(n):pass 1 of 5⋯13"""Returns a generator (values on demand)"""14for i0 in range(n5):15    yield i0⋯All 5 passes — pass 1 is the card abovepassi
10
21
32
43
54
8print("Values:", list(gen))⋯26print(type(gen), gen)27print("Values:", list(gen⟨generator object lazy_range A⟩))⋯outputValues: [0, 1, 2, 3, 4]

All 5 passes — pass 1 is the card above
pass	`i`	`result`
1	0	[] → [0]
2	1	[0] → [0, 1]
3	2	[0, 1] → [0, 1, 2]
4	3	[0, 1, 2] → [0, 1, 2, 3]
5	4	[0, 1, 2, 3] → [0, 1, 2, 3, 4]

All 5 passes — pass 1 is the card above
pass	`i`
1	0
2	1
3	2
4	3
5	4

lazy evaluation - computing values only when needed, rather than all at once upfront

Generator Expressions

                            expression.py
                        
Replay: real traced execution (multi-file project)

                            # Generator expressions


# Generator expression (lazy)
gen = (x * x for x in range(5))
print("Type:", type(gen))
print("Values:", list(gen))

# Compare with list comprehension (eager)
lst = [x * x for x in range(5)]
print("\nList:", type(lst), lst)

# Pipeline with generator expressions
numbers = range(10)
squares = (x * x for x in numbers)
evens = (x for x in squares if x % 2 == 0)
print("\nEven squares:", list(evens))


                        
1gen ← <generator object <genexpr> at ⟨addr A⟩>, lst ← [0, 1, 4, 9, 16]⋯4# Generator expression (lazy)5gen→ <generator object <genexpr> at ⟨addr A⟩> = (x * x for x in range(5))6print("Type:", type(gen<generator object <genexpr> at ⟨addr A⟩>))7print("Values:", list(gen<generator object <genexpr> at ⟨addr A⟩>))89# Compare with list comprehension (eager)10lst→ [0, 1, 4, 9, 16] = [x * x for x in range(5)]11print("\nList:", type(lst[0, 1, 4, 9, 16]), lst)1213# Pipeline with generator expressions14numbers→ range(0, 10) = range(10)15squares→ <generator object <genexpr> at ⟨addr B⟩> = (x * x for x in numbersrange(0, 10))16evens→ <generator object <genexpr> at ⟨addr C⟩> = (x for x in squares<generator object <genexpr> at ⟨addr B⟩> if x % 2 == 0)17print("\nEven squares:", list(evens<generator object <genexpr> at ⟨addr C⟩>))⋯outputType: <class 'generator'>
Values: [0, 1, 4, 9, 16]

List: <class 'list'> [0, 1, 4, 9, 16]

Even squares: [0, 4, 16, 36, 64]

generator expression - a compact syntax `(expr for x in iterable)` that creates a generator inline

Infinite Generators

                            infinite.py
                        
Replay: real traced execution (multi-file project)

                            # Infinite generators


def infinite_count(start=0):
    """Infinite counter"""
    n = start
    while True:
        yield n
        n += 1


# Use with break or takewhile
counter = infinite_count(1)
for num in counter:
    print(num, end=" ")
    if num >= 10:
        break

print("\n")

# With itertools.islice
from itertools import islice

counter = infinite_count(100)
print("First 5:", list(islice(counter, 5)))


                        
1counter ← ⟨generator object infinite_count A⟩⋯12# Use with break or takewhile13counter→ ⟨generator object infinite_count A⟩ = infinite_count(1)14for num in counter:⋯
2n ← 1pass 1 of 2⋯4def infinite_count(start1=0):5    """Infinite counter"""6    n→ 1 = start17    while True:⋯
3while True:pass 1 of 15⋯6n = start7while True:8    yield n19    n += 1⋯15 passes — pass 1 is the card abovepassnumn
1—1
2—2
3—3
4—4
5—5
6—6
7—7
8—8
9—9
⋯ 4 more passes ⋯
14—103 → 104
15—104
4n ← 2pass 1 of 10⋯7    while True:8        yield n19        n→ 2 += 1101112# Use with break or takewhile13counter = infinite_count(1)14for num1 in counter⟨generator object infinite_count A⟩:15    print(num1, end=" ")16    if num >= 10:⋯output1All 10 passes — pass 1 is the card abovepassnumn
111 → 2
222 → 3
333 → 4
444 → 5
555 → 6
666 → 7
777 → 8
888 → 9
999 → 10
1010—
5if num >= 10:⋯15print(num, end=" ")16if num10 >= 10:17    break⋯
6counter ← ⟨generator object infinite_count B⟩⋯19print("\n")2021# With itertools.islice22from itertools import islice2324counter→ ⟨generator object infinite_count B⟩ = infinite_count(100)25print("First 5:", list(islice(counter⟨generator object infinite_count B⟩, 5)))⋯output
7n ← 100pass 2 of 2⋯4def infinite_count(start100=0):5    """Infinite counter"""6    n→ 100 = start1007    while True:⋯
8print("First 5:", list(islice(counter, 5)))⋯24counter = infinite_count(100)25print("First 5:", list(islice(counter⟨generator object infinite_count B⟩, 5)))⋯outputFirst 5: [100, 101, 102, 103, 104]

15 passes — pass 1 is the card above
pass	`num`	`n`
1	—	1
2	—	2
3	—	3
4	—	4
5	—	5
6	—	6
7	—	7
8	—	8
9	—	9
⋯ 4 more passes ⋯
14	—	103 → 104
15	—	104

All 10 passes — pass 1 is the card above
pass	`num`	`n`
1	1	1 → 2
2	2	2 → 3
3	3	3 → 4
4	4	4 → 5
5	5	5 → 6
6	6	6 → 7
7	7	7 → 8
8	8	8 → 9
9	9	9 → 10
10	10	—

yield vs return

return ends the function and returns a value
yield pauses the function and produces a value, resuming later

Key Benefits

Lazy evaluation: values produced on demand, not all at once
State preservation: local variables preserved between yields
Memory efficient: only one value in memory at a time
Automatic iterator: no need to write __iter__ and __next__

Exercise: practical.py

Build a file reader that yields lines matching a pattern