8.11 Iterators and the iter() Function
8.11.1 What Are Iterators?
Iterators were added to Python in version 2.2 to give sequence-like objects a sequence-like interface. We formally introduced sequences back in Chapter 6.
They are just data structures that you can “iterate” over by using their index starting at 0 and continuing till the final item of the sequence. Because you can do this “counting,” iterating over sequences is trivial. Iteration support in Python works seamlessly with sequences but now also allows program- mers to iterate through non-sequence types, including user-defined objects.
Iterators come in handy when you are iterating over something that is not a sequence but exhibits behavior that makes it seem like a sequence, for example, keys of a dictionary, lines of a file, etc. When you use loops to iterate over an object item, you will not be able to easily tell whether it is an iterator or a sequence. The best part is that you do not have to care because Python makes it seem like a sequence.
8.11.2 Why Iterators?
The defining PEP (234) cites that iterators:
• Provide an extensible iterator interface.
• Bring performance enhancements to list iteration.
• Allow for big performance improvements in dictionary iteration.
• Allow for the creation of a true iteration interface as opposed to overriding methods originally meant for random element access.
• Be backward-compatible with all existing user-defined classes and extension objects that emulate sequences and mappings.
• Result in more concise and readable code that iterates over non-sequence collections (mappings and files, for instance).
8.11.3 How Do You Iterate?
Basically, instead of an index to count sequentially, an iterator is any item that has a next() method. When the next item is desired, either you or a looping mechanism like for will call the iterators next()method to get
ptg 310 Chapter 8 Conditionals and Loops
the next value. Once the items have been exhausted, a StopIteration exception is raised, not to indicate an error, but to let folks know that we are done.
Iterators do have some restrictions, however. For example, you cannot move backward, go back to the beginning, or copy an iterator. If you want to iterate over the same objects again (or simultaneously), you have to create another iterator object. It isn’t all that bad, however, as there are various tools to help you with using iterators.
There is a reversed()built-in function that returns an iterator that traverses an iterable in reverse order. The enumerate() BIF also returns an iterator. Two new BIFs, any() and all(), made their debut in Python 2.5—
they will return True if any or all items traversed across an iterator have a BooleanTrue value, respectively. We saw earlier in the chapter how you can use it in a for loop to iterate over both the index and the item of an iterable.
There is also an entire module called itertools that contains various itera- tors you may find useful.
8.11.4 Using Iterators with . . .
Sequences
As mentioned before, iterating through Python sequence types is as expected:
>>> myTuple = (123, 'xyz', 45.67)
>>> i = iter(myTuple)
>>> i.next() 123
>>> i.next() 'xyz'
>>> i.next() 45.67
>>> i.next()
Traceback (most recent call last):
File "", line 1, in ? StopIteration
If this had been an actual program, we would have enclosed the code inside a try-except block. Sequences now automatically produce their own iterators, so a for loop:
for i in seq:
do_something_to(i)
ptg 8.11 Iterators and the iter() Function 311
under the covers now really behaves like this:
fetch = iter(seq) while True:
try:
i = fetch.next() except StopIteration:
break
do_something_to(i)
However, your code does not need to change because the for loop itself calls the iterator’s next() method (as well as monitors for StopIteration).
Dictionaries
Dictionaries and files are two other Python data types that received the itera- tion makeover. A dictionary’s iterator traverses its keys. The idiom for eachKey in myDict.keys() can be shortened to for eachKey in myDict as shown here:
>>> legends = { ('Poe', 'author'): (1809, 1849, 1976), ... ('Gaudi', 'architect'): (1852, 1906, 1987),
... ('Freud', 'psychoanalyst'): (1856, 1939, 1990) ... }
...
>>> for eachLegend in legends:
... print 'Name: %s\tOccupation: %s' % eachLegend ... print ' Birth: %s\tDeath: %s\tAlbum: %s\n' \ ... % legends[eachLegend]
...
Name: Freud Occupation: psychoanalyst Birth: 1856 Death: 1939 Album: 1990 Name: Poe Occupation: author
Birth: 1809 Death: 1849 Album: 1976 Name: Gaudi Occupation: architect
Birth: 1852 Death: 1906 Album: 1987
In addition, three new built-in dictionary methods have been introduced to define the iteration: myDict.iterkeys()(iterate through the keys), myDict.itervalues() (iterate through the values), and myDict.iter- items() (iterate through key/value pairs). Note that the in operator has been modified to check a dictionary’s keys. This means the Boolean expression myDict.has_key(anyKey) can be simplified as anyKey in myDict.
ptg 312 Chapter 8 Conditionals and Loops
Files
File objects produce an iterator that calls the readline() method. Thus, they loop through all lines of a text file, allowing the programmer to replace essentially for eachLine in myFile.readlines()with the more sim- plisticforeachLineinmyFile:
>>> myFile = open('config-win.txt')
>>> for eachLine in myFile:
... print eachLine, # comma suppresses extra \n ...
[EditorWindow]
font-name: courier new font-size: 10
>>> myFile.close()
8.11.5 Mutable Objects and Iterators
Remember that interfering with mutable objects while you are iterating them is not a good idea. This was a problem before iterators appeared. One popular example of this is to loop through a list and remove items from it if certain criteria are met (or not):
for eachURL in allURLs:
if not eachURL.startswith('http://'):
allURLs.remove(eachURL) # YIKES!!
All sequences are immutable except lists, so the danger occurs only there. A sequence’s iterator only keeps track of the Nth element you are on, so if you change elements around during iteration, those updates will be reflected as you traverse through the items. If you run out, then StopIteration will be raised.
When iterating through keys of a dictionary, you must not modify the dictio- nary. Using a dictionary’s keys() method is okay because keys() returns a list that is independent of the dictionary. But iterators are tied much more inti- mately with the actual object and will not let us play that game anymore:
>>> myDict = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
>>> for eachKey in myDict:
... print eachKey, myDict[eachKey]
... del myDict[eachKey]
...
a 1
Traceback (most recent call last):
File "", line 1, in ?
RuntimeError: dictionary changed size during iteration This will help prevent buggy code. For full details on iterators, see PEP 234.