Iterators and Iterables in Python

Iterators and iterables are foundational concepts in Python, forming the basis of its looping mechanisms. They allow you to traverse through collections like lists, tuples, dictionaries, and even custom objects. Let’s explore them in detail.

What is an Iterable?

An iterable is any Python object that can be iterated over (i.e., you can go through its elements one at a time). This includes built-in data types such as lists, tuples, dictionaries, sets, and strings.

Characteristics of an Iterable:

• It implements the __iter__() method, which returns an iterator.
• Iterables can be passed to Python’s iter() function to get an iterator.

# Example 1: List
my_list = [1, 2, 3, 4]
for item in my_list:
    print(item)

# Example 2: String
my_string = "Hello"
for char in my_string:
    print(char)

What is an Iterator?

An iterator is an object that represents a stream of data. It enables you to traverse through the data one element at a time.

Characteristics of an Iterator:

• It implements two methods:
  • __iter__(): Returns the iterator object itself.
  • __next__(): Returns the next item in the sequence. If there are no more items, it raises the StopIteration exception.
• Iterators are stateful, meaning they remember where they are in the sequence.

# Example 1: Creating an iterator from a list
my_list = [1, 2, 3]
iterator = iter(my_list)

print(next(iterator))  # 1
print(next(iterator))  # 2
print(next(iterator))  # 3
# print(next(iterator))  # Raises StopIteration

Making Objects Iterable

You can make a custom object iterable by implementing the __iter__() method. If you also implement __next__(), the object can behave as its own iterator.

class MyIterable:
    def __init__(self, data):
        self.data = data

    def __iter__(self):
        return MyIterator(self.data)

class MyIterator:
    def __init__(self, data):
        self.data = data
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.index < len(self.data):
            result = self.data[self.index]
            self.index += 1
            return result
        else:
            raise StopIteration

# Using the custom iterable
my_iterable = MyIterable([1, 2, 3])
for item in my_iterable:
    print(item)

StopIteration: To prevent the iteration from going on forever, we can use the StopIteration statement.

Generators as Iterators

A generator is a simpler way to create iterators. Instead of implementing __iter__() and __next__(), you use the yield keyword in a function.

def my_generator():
    yield 1
    yield 2
    yield 3

gen = my_generator()
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # 3

Generators are memory-efficient because they yield items one at a time instead of generating all items at once.

Benefits of Iterators and Iterables

• Memory Efficiency: Iterators do not require storing the entire data in memory.
• Lazy Evaluation: Iterators fetch items as needed, which is useful for large datasets.
• Custom Traversal: You can customize the way an object is traversed.

Iterable vs Iterator

Something interesting

Case 1 Not all iterables are iterators

An object that implements __iter__() but not __next__() is iterable but not an iterator.

my_list = [1, 2, 3]  # A list is an iterable
my_set = {1,2,3,4,5}

print(hasattr(my_list, '__iter__'))  # True, it has __iter__()
print(hasattr(my_list, '__next__'))  # False, it doesn't have __next__()


print(hasattr(my_set, '__iter__'))  # True, it has __iter__()
print(hasattr(my_set, '__next__'))  # False, it doesn't have __next__()

You can create an iterator from this iterable using iter():

iterator = iter(my_list)  # Creates an iterator from the list
print(next(iterator))  # 1
print(next(iterator))  # 2
print(next(iterator))  # 3

How Does a Loop Work on a List?

If list, set etc are not iterators , then how we are accessing them through loop?

The for loop works on a list by first converting it into an iterator using iter().

The for loop in Python works by implicitly calling iter() on the iterable (like a list) to get an iterator. Then it uses the next() function on the iterator to fetch items one by one, until a StopIteration exception is raised.

Step-by-Step Explanation of a for Loop:

• The for loop checks if the object is iterable by trying to call iter() on it.
• The iter() function returns an iterator for the iterable (e.g., a list iterator for a list).
• The for loop repeatedly calls next() on the iterator to fetch the next element.
• When the iterator is exhausted (i.e., no more elements to return), it raises a StopIteration exception, which the for loop handles internally to terminate.

Example: How Python Handles a for Loop Internally

Let’s manually simulate how a for loop processes a list:

my_list = [1, 2, 3]  # A list (iterable)

# Internally, the for loop does the following:
iterator = iter(my_list)  # Create an iterator from the list

while True:
    try:
        # Get the next item
        item = next(iterator)
        print(item)  # Process the item
    except StopIteration:
        # If no more items, break the loop
        break

Case 2: Iterator and Iterable

An object that implements both __iter__() and __next__() is both an iterator and an iterable. Most iterators fall into this category.

class MyIterator:
    def __init__(self, data):
        self.data = data
        self.index = 0

    def __iter__(self):
        return self  # Returns itself as an iterator

    def __next__(self):
        if self.index < len(self.data):
            result = self.data[self.index]
            self.index += 1
            return result
        else:
            raise StopIteration

my_iterator = MyIterator([1, 2, 3])
print(hasattr(my_iterator, '__iter__'))  # True
print(hasattr(my_iterator, '__next__'))  # True

# Iterating over the iterator
for item in my_iterator:
    print(item)  # Outputs 1, 2, 3

Case 3: Iterator but not Iterable

An object that implements next() but not iter() is an iterator but not an iterable. This is uncommon and not typical in Python.

class NonIterableIterator:
    def __next__(self):
        return 42  # Always returns 42

non_iterable_iterator = NonIterableIterator()

print(hasattr(non_iterable_iterator, '__iter__'))  # False
print(hasattr(non_iterable_iterator, '__next__'))  # True

# You can still call next() on it:
print(next(non_iterable_iterator))  # 42
print(next(non_iterable_iterator))  # 42

Case 4: Neither Iterable nor Iterator

An object that implements neither __iter__() nor __next__() is neither an iterable nor an iterator.

class NonIterableNonIterator:
    pass

obj = NonIterableNonIterator()

print(hasattr(obj, '__iter__'))  # False
print(hasattr(obj, '__next__'))  # False

Summary Table

Practical Use Cases

File Reading

with open('example.txt', 'r') as file:
    for line in file:
        print(line.strip())

Infinite Sequences:

def infinite_counter():
    count = 0
    while True:
        yield count
        count += 1

counter = infinite_counter()
print(next(counter))  # 0
print(next(counter))  # 1

Common Mistakes

1. Exhausting Iterators: Once an iterator is exhausted, it cannot be reused.

iterator = iter([1, 2, 3])
for item in iterator:
    print(item)
for item in iterator:
    print(item)  # Nothing happens

2. Confusing Iterables and Iterators: Not all iterables are iterators, and not all iterators are inherently iterables.

Conclusion

• Iterables provide a way to access elements sequentially.
• Iterators provide the mechanism to fetch elements one at a time.
• Together, they form the basis of Python’s for loop and many other constructs.

Understanding iterables and iterators is crucial for writing efficient and Pythonic code, especially when working with large datasets or designing custom data structures.

Resources

• Scalar
• Geeksforgeeks
• Python Interview questions