Note: This article was first published the January 2008 issue of Python Magazine

Iterators, iterables, and generators are features handled so wall by Python that people programming in other languages cannot help but drool over. Fortunately for us, creating iterators, iterables and generators is a relatively simple task. This article introduces the concepts of iterators, iterables, and generators and illustrates how easy it is to add them to your code.

1. Introduction
2. Iteration in Python
3. An Initial Example
4. Creating An Iterator
5. Looking More Closely At The Iterator
6. The Upside And Downside Of Iterators
7. Generators
8. Looking Closely At The Generator
9. But What About Iterables?
10. Creating An Iterable Object
11. Conclusion

Introduction

In this article I’m going to introduce three related Python features: iterators, iterables, and generators. Generators are easy to define, they are functions that create and return an iterator. Iterators and iterables on the other hand, are easier to use than they are to define. An iterable object is a “container object capable of returning its members one at a time.”[1] An iterator object is “An object representing a stream of data. Repeated calls to the iterator’s next() method return successive items in the stream. When no more data is available a StopIteration exception is raised instead. At this point, the iterator object is exhausted and any further calls to its next() method just raise StopIteration again.”[1] You can think of the difference between the two in this way: an iterable object can be iterated over multiple times, whereas an iterator object can only be iterated over once. In general an iterable produces an iterator every time something wants to iterate over its data.

Note: Classes that define the __getitem__ function are also considered iterables, but since that falls outside the scope of this article, it will not be covered here.

In this tutorial, I will begin by discussing iterators, the most basic concept. Then I will move onto generators, and finish by discussing iterables, the most wide open topic of the three.

Iteration in Python

Iterators objects are used in Python in order to iterate over an objects data. For example, we all know how to do this in Python when we work with lists:

my_list = [1,2,3]
for num in my_list:
	print num

This code will iterate over the list object my_list and print out all of the list items , i.e., the numbers 1, 2, and 3. Iterating over sequences in this simple and transparent manner happens to be one of my favourite features of Python.

According to our definition above, lists are iterables since you can iterate over them multiple times. In fact, each time you iterate over a list you are actually using a listiterator iterator object produced by the list.

It may not be immediately clear to you when you should add an iteration support to a class. However, the more you work with Python the more you’ll find instances when doing just this is very useful, sometimes the only advantage is cleaner looking code. One nice thing about iterators (and generators too) is that the processing for each item happens as you need it. Instead of collecting all of the data into a list and then running through the list, you will collect each item as you need it. This might not seem like a large difference but imagine if there were tens of thousands of items to process? What if you were collection your data from an online source? Performing all of your processing up front may take a very long time, especially if you only wanted the first few items.

An initial example

In order to explain iteration further, let’s look at a simple example task where we might use iterators. For this example we will create a class that takes a string of characters as input and then converts each character into its byte value. If we were NOT going to use iterators we might do something like what is found in Listing 1.

Listing 1

class ByteValue(object):

	def __init__(self, data):
		self.data = data

	def to_bytes(self):
		bytes = []
		for char in self.data:
			bytes.append(ord(char))
		return bytes

This code is pretty simple. We have a data member, named data, that we use to store the string that was used to initialize the class. In the to_bytes function we loop through the string, converting each character to its byte value using the built in ord function. We store each byte value in a list and once we have collected all of the values we return that list.

When we run the following:

bv = ByteValue("abcdef")
for byte in bv.to_bytes():
	print byte

we would get this as our output:

97
98
99
100
101
102

Creating an Iterator

Let’s convert this into an iterator. Making your class into an iterator requires adding “two methods, which together form the iterator protocol.”[2] The two functions needed are: 1) the __iter__ function; and, 2) the next function. The __iter__ function will return the object itself, while the next function will return the next item. The next function is where the actual iteration work occurs. The next function iterates by returning the next item in the “sequence” each time it is called for as long as there is a “next” item. When there are no more items to iterate over, the next function must raise the StopIteration exception to halt the iteration.

To be clear, in order to make your class an iterator you need to do two things:

1) Add an __iter__ function that returns the object itself (self)

2) Add a next function that returns the next item in the sequence each time it is called. When there are no more items in the sequence, the next function raises a StopIteration exception signal the end of the iteration.

For those of you still confused, the following example will help illustrate how iterators work. If we were to convert our ByteValue class into an iterator object, it might look something like Listing 2.

Listing 2

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0

	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value

Let’s compare the code in Listings 1 and Listing 2 in detail, focusing on the iterator in Listing 2. The first difference is the addition of the data member current_item to the class, initialized in the __init__ function. current_item serves as a counter and keeps track of the current character in the string while we iterate over it. The counter must have class scope since the iterator works through successive calls to the next function. If current_item were local to the next function, its value would be reset with each subsequent call, and would not be of much use.

The second difference between the listings is the addition in Listing 2, of the __iter__ function where we return self.

The final addition to Listing 2 is the next function, where we first check to see if current_item is equal to the length of our string. If current_item is equal to the strings length we raise the StopIteration exception to signal the end of the iteration because we have no more characters left to iterate over. If there are more characters to iterate over we calculate the byte value of the current character, increase our current_item counter, and then return the byte value.

Note: Notice that current_item is only initialized when the ByteValue object is created. This happens because according to the iteration protocol, “once an iterator’s next() method raises StopIteration, it will continue to do so on subsequent calls.”[2] If we were to re-initialize current_item we would then be able to iterate over the iterator more than once breaking the iteration protocol.

Now that we have converted our class into an iterator we can use it as follows:

for byte in ByteValue("abcdef"):
	print byte

Doing so would result in:

97
98
99
100
101
102

Notice that we do not store the instance of our ByteValue class in a variable. Doing so would be useless because since ByteValue is an iterator it is only good for one pass of the data. If ByteValue were an iterable (returning an iterator object when __iter__ was called) it would make sense to keep an instance around because we could iterate over the instance more than once. We will look at creating iterables later on in this article.

Looking more closely at the Iterator

Let’s look at what is happening in more detail by examining what is happening behind the scenes during the iteration process. In order to illustrate what is happening in the for loop I will demonstrate the order in which things are being called behind the scenes.

The first step in the iteration process is to call to the __iter__ function in order to get the iterator object that will perform the iteration. Notice that this works on iterator and iterable objects, since iterators returns themselves and iterables return an iterator.

bv = ByteValue("abcdef")
iterator = bv.__iter__()

Now that we have the iterator object, we start iterating by calling the next function in order to get the next value:

print iterator.next()

This executes the next function which will return the byte value of character in the string with which we are currently working. Since this is the first call to the next function current_item will be zero and we will calculate the byte value of the first character in our string (’a') resulting in:

97

If we continue the iteration process by calling the next function six more times we would get the results shown in Listing 3. Notice that we have now made a total of seven calls to the next function, one more then the number of characters in our string. I’m running the python code from a file (iter.py found in Listing 4). Depending on how you are running it, you might get slightly different results. However, the most important thing to observe in this example is the exception that is raised.

Listing 3

97
98
99
100
101
102
Traceback (most recent call last):
  File "Listing4.py", line 34, in ?
    main()
  File "Listing4.py", line 30, in main
    print iterator.next()
  File "Listing4.py", line 13, in next
    raise StopIteration
StopIteration

Listing 4

#!/usr/bin/env python

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0
	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value
		return self.data

def main():
    for v in ByteValue("abc"):
        if v in ByteValue("abc"):
            print "We have a %d" % v

    bv = ByteValue("abcdef")
    iterator = bv.__iter__()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()

if __name__ == "__main__":
	# Someone is launching this directly
	main()

For the most part you won’t be calling an iterator object’s __iter__ or next functions manually, instead you’ll probably just let the for loop do it all for you.

The upside and downside of Iterators

Now that our class is an iterator object we can use any of the built-in functions and methods that work on iterators and iterables, such as: the sum, tuple, sorted, and list functions, to name a few.

In the first example the bytes function returned a list object that we could work with. If we want a list instead of an iterator for any reason, it will be as simple as using the list function, which takes an iterator as a parameter and returns a list:

list(ByteValue("abcdef"))

If we want a sorted list:

sorted(ByteValue("abcdef"))

If we want the sum of all of the bytes:

sum(ByteValue("abcdef"))

While very useful, iterators do have some downsides to them. The most obvious is that they are only good for one pass over the data. They also generally require you to add extra data members to your class in order to keep track of your current iterator position. Depending on what you are iterating over, this process can become quite complex. Iterators also only allow you to perform one “type” of iteration, i.e., in one direction or over one piece of internal data. You might address this by adding flags to your class but this will further clutter the class and decrease readability. So how can you simply add multiple types of iteration to a class? Enter the generator!

Generators

A generator is a function that creates, or generates, an iterator. In order for a function to become a generator it must return a value using the yield keyword.

Generators are interesting because they are functions, yet execution does not run through them as it does in a normal function. The first time execution enters a generator function it will start at the beginning of the function and continue until the yield keyword is encountered. When the iteration continues, execution will continue in the generator function on the statement immediately following the yield keyword. All local variables in the function will remain intact. If the yield statement occurs within a loop, execution will continue within the loop as though execution had not been interrupted.

Continuing with the ByteValue example, let’s add a generator function named reverse that can be used to iterate through the byte vales of the string in reverse order:

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		yield ord(self.data[current_item])

So what’s going on here? The first thing to notice is that we did not add anymore data members to our class, this generator is a self-contained unit. Secondly, since this is a generator, our counter current_item can be a local variable.

In the reverse function the first step is to initialize current_item, which represents the current character in the string, to be equal to the length our string. We initialize it to the length of the string instead of zero since we are iterating through the string in reverse. Next we have a while loop that loops while current_item is greater than zero. We then subtract one from our counter, to give us the current character to process. Finally, we yield the byte value of the current character.

Note: We subtract one from our counter the first time through the loop because Python is zero-based and the length of a list minus one gives us the position of the last item in the list. In our examples we have used the following string:

abcdef
012345

When we calculate the length of the string we get 6. We then subtract 1 from that number, leaving 5, which is the index of the last number in the string.

Making use of our new generator function, we run the following:

bv = ByteValue("abcdef")
for byte in bv.reverse():
	print byte

We get our favourite byte values in reverse:

102
101
100
99
98
97

Looking closely at the Generator

Let’s take a detailed look at what is happening in the generator in the same way that we did earlier with the iterator object. The first thing that happens when you call a generator function is NOT the execution of the actual function, rather, it is the creation of a generator object. Running the following:

bv = ByteValue("abcdef")
gen = bv.reverse()
print gen

will result in:

<generator object at 0xb7d8e04c>

This demonstrates that, as stated above, the first call to our generator function does not return the byte value of that last character in the string, instead it creates a generator object. In fact the first time a generator function is called, the actual function is not executed at all. A generator object is “what Python uses to implement generator iterators. They are normally created by iterating over a function that yields values”.[3]

Once we have a generator object we can start calling its next function (sound familiar?) to perform the action iteration. An example of this can be found in Listing 5. The results of this execution can be found in Listing 6. The results will seem very familiar to you, especially the StopIteration exception.

Listing 5

#!/usr/bin/env python

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0
	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value
		return self.data

	def reverse(self):
		current_item = len(self.data)
		while (current_item > 0):
			current_item -= 1
			yield ord(self.data[current_item])

def main():

	bv = ByteValue("abcdef")
	gen = bv.reverse()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()

if __name__ == "__main__":
	# Someone is launching this directly
	main()

Listing 6

102
101
100
99
98
97
Traceback (most recent call last):
  File "Listing5.py", line 40, in ?
    main()
  File "Listing5.py", line 36, in main
    print gen.next()
StopIteration

If we want to look at the contents of the generator object we could use the following code:

print dir(gen)

And getting the following results:

['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'gi_frame', 'gi_running', 'next']

Notice that the generator object contains the __iter__ and next functions making the generator object itself an iterator. Because the generator itself is an iterator object, the same built-in iterator functions I mentioned earlier can also be used on our generators:

print sum(bv.reverse())

It is also important to remember that your generator or iterator does not have to perform only “dumb” iteratation, simply moving through some sort of an internal list. Rather, it can make decisions just like any other block of code.

For example, let’s say that in our reverse generator we want to use the byte value 99 as an end condition. We can do something similar to the example found in Listing 7.

Listing 7

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		value = ord(self.data[current_item])
		if (value == 99):
			return
		yield value

In Listing 7 if the byte value equals 99 we return from our generator function using the return keyword. Since we didn’t yield anything this will cause the StopIteration exception to be fired halting the iteration.

Be careful about getting too smart with your iteration because you cannot return any information from a generator you can only yield it. So if you tried to execute the code in Listing 8, in an attempt to return one last value before quitting (or if you wanted to return a success or failure code) you will get the following error when the return value line of code is executed:

SyntaxError: 'return' with argument inside generator

Listing 8

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		value = ord(self.data[current_item])
		if (value == 99):
			return value
		yield value

But what about iterables?

I’m sure that by now you are wondering about the iterable objects that I mentioned at the start of this tutorial. As you have probably guessed, simply making your class an iterator isn’t that useful unless it is only performing one task like our ByteValue example. Since your classes will generally be performing more than one task, you will likely want to make your class an iterable object rather than an iterator object. To recap, iterable objects return an iterator object when their __iter__ function is called, which allows for multiple passes over their data.

Creating an Iterable object

Since the definition of an iterable object is an object that returns an iterator object when its __iter__ function is called, creating an iterable object can be done in a variety of ways. Two options that come to mind are: 1)creating an iterator helper class to perform the iteration and, 2) using a generator function. I prefer using a generator function since it keeps the functionality within the main class.

See Listing 9 for an example of creating an iterarable object using a generator. You will see that we have replaced the next function with the forward function. The forward function is a generator that iterates through the data in the “forward” direction. In the __iter__ function we return the results of a call to the forward function, a generator object. Since generator objects contain the iterator protocol and are, in fact iterators, by returning one from our __init__ function we have successfully created an iterable.

Listing 9

class ByteValue(object):

	def __init__(self, data):
		self.data = data

	def __iter__(self):
		#We are an iterable, so return our iterator
		return self.forward()

	def forward(self):
		#The forward generator
		current_item = 0
		while (current_item < len(self.data)):
			byte_value = ord(self.data[current_item])
			current_item += 1
			yield byte_value

	def reverse(self):
		#The reverse generator
		current_item = len(self.data)
		while (current_item > 0):
			current_item -= 1
			yield ord(self.data[current_item])

def main():
    bv = ByteValue("abc")
    for v in bv:
        if v in bv:
            print "We have a %d" % v

if __name__ == "__main__":
    main()

Now that we have an iterable object, we can iterate over it as many times as we want. We can even have fun with nested iteration:

bv = ByteValue("abcdef")
for value in bv:
	print value
	for second_value in bv:
		print second_value

Conclusion

This concludes our introduction to iterators, iterables, and generators. I hope I have demonstrated the immense power and flexibility that they provide. In general, making your object an iterable object and/or using generators allows for more flexibility than simple iterator objects provide. For most complex classes or complex sets of data, multiple iterations are a given.

With all of the praise that I have heaped upon iterators, iterables, and generators, it’s important to remember that they are not a panacea and should not be used in every case where a sequence of items is needed. There are many instances where a returning a list is the desired result. This being said, iterators, iterables, and generators are extremely useful and provide a great way to loop through data.

[1] http://docs.python.org/tut/node18.html
[2] http://docs.python.org/lib/typeiter.html
[3] http://docs.python.org/api/gen-objects.html

I’m still writing for Python Magazine and working on the editor in some of the free time that I get. I just added a selection cursor mode to the editor which makes it easier for me to use. You can see it in the second row of the pallet in the following screen shot.

Now with a Select Mode!

If you want access to the newest code you can grab it from the mercurial repository: http://freehg.org/u/selsine/dodger/

If anyone is at all interested in this project, whether it’s using or helping to develop, or just some suggestions please let me know. I’m always open to opinions. If I get the time I’d like to work on a short post showing you how you can use the Dodger Editor to create the level files for a game, when that will be I don’t know but hopefully soon.

The project is really simple and meant as an easy way for you to create “text buttons” for your PyGame projects. It’s not meant to be the definitive way to do this, just a simple solution for people that just want to drop a class in and have working “text buttons”. It’s LGPL so you can use it in whatever way you want. If you do decide to use it please drop me an email and let me know.

For more information on how to use the project please take a look at the initial blog post.

Note: This article was first published the December 2007 issue of Python Magazine

While the command line will never cease to be useful, nothing will impress your friends more than your latest python masterpiece wrapped up in a slick cross-platform Graphical User Interface (GUI). This tutorial will show you how to create a simple GUI in Python using PyQt4.

1. Introduction
2. Installing PyQt4
3. Your First PyQt4 Application
4. The Main Window
5. Adding Some Widgets
6. Signal Handling
7. Displaying a Message
8. Conclusion

Introduction

While writing console applications and modules using Python can be very enjoyable, I think that almost everyone learning Python eventually wants to do one thing: create a application with a full Graphical User Interface (GUI). Fortunately for Python users, there are a few options available to achieve this. One of the more interesting options is PyQt4, Python bindings for the fourth version of the famous cross platform application development API Qt.

Qt, owned by Trolltech software, is probably most famous as the foundation for the KDE window manager on Linux. PyQt4, and PyQt version 3, were created and are maintained by Riverbank software. Qt and PyQt4 are both open source and free for open source applications, but if you wish to develop commercial applications you will need to purchase the commercial versions of both Qt and PyQt4.

One of the best features of Qt is that it is a cross platform library which means that it gives you easy access to the three main desktop environments: Linux, Windows, and OS X. This is an important requirement for me because when I’m developing a small open source application I want it available on as many platforms as possible. The nice thing about Qt’s latest version, Qt4, is that it also gives your GUI a native look and feel on the different operating systems.

This article will create a simple application to introduce you to the basic features of PyQt4. The application will consist of a single window that does the following: accepts user input, responds to a button click, and displays a message using a pop-up dialog.

Installing PyQt4

Before installing PyQt4 you need to have Python 2.5 or newer and Qt version 4 installed. You can download Python 2.5 from the main Python website.[1] Qt version 4 is available from the TrollTech website’s Downloads section.[2] Installing PyQt4 on Linux, Windows, or OS X is relatively easy, and I know since I installed it on each of them. You can download the different PyQt4 packages from the Riverbank website, including an all in one (Qt version 4 included) installer for Windows.[3] There are also installation instructions in the PyQt4 online documentation that are worth a read if you are new to installing software from source.[4] If you are using a modern Linux distribution, you may want to consider using your package manager to install PyQt4 as it should being along all the necessary dependencies for you.

Your First PyQt4 Application

Now that you have PyQT4 properly installed, let’s dive right in and create an application. The first step is to start a new Python file, I called mine PyQt4Intro.py but you can call it whatever you like. Set it up using the following code:

#!/usr/bin/env python
import PyQt4

if __name__ == "__main__":
	# Someone is launching this directly
	pass

If you’ve read any of my other columns, you may recognize this code as a general template that I use for my Python projects. The major difference in this example is that we import PyQt4. After you have set your file up run the code. If all goes well nothing will happen. If you do get an error, something like the following, it probably means that PyQt4 is not installed properly:

ImportError: No module named PyQt4

Remember that PyQT4 requires Python 2.5. If you have Python 2.5 and older versions of Python installed, try specifying python2.5 when you run the script:

python2.5 PyQt4Intro.py

Now that we have everything in order, let’s create and show a window. After all this is a GUI tutorial! If you are creating a GUI application using PyQt4 (of course, you can also create a non-GUI application using PyQt4), you will need to create a QApplication instance.

The PyQt4 documentation provides a great description of the “QApplication” class: “The QApplication class manages the GUI application’s control flow and main settings. It contains the main event loop, where all events from the window system and other sources are processed and dispatched. It also handles the application’s initialization and finalization, and provides session management. It also handles most system-wide and application-wide settings. For any GUI application that uses Qt, there is precisely one QApplication object, no matter whether the application has 0, 1, 2 or more windows at any time.”[5]

What we need to do is create the one and only QApplication instance for our application. To do this, we will import two new modules:

import sys
from PyQt4 import QtGui

The first line imports the sys module. This is needed to initialize the QApplication. The second line imports the QtGui module. It allows us to reference QApplication and other QtGui components.

In our main block, we will add the following code:

if __name__ == "__main__":
	# Someone is launching this directly
	# Create the QApplication
	app = QtGui.QApplication(sys.argv)
	# Enter the main loop
	app.exec_()

This will create the QApplication instance, initialized with the command-line parameters that may be passed to our script. This is a necessary parameter to the QApplication class and is the reason that we imported the sys module. The next call is to the exec function which “enters the main event loop and waits until exit() is called or the main widget is destroyed”.[6]

If you run the entire script at this point, which I don’t necessarily recommend since it will seem to hang, you will be running a PyQt4 application. You won’t see anything since there is no visible component. However, the script will continue to run in its main loop instead of returning right after being executed. If you do decide to run this script, use CTRL+Z to exit.

The Main Window

GUI applications without a visible component are not that useful so let’s add a window to our application. We will do this by sub-classing the QMainWindow class. The QMainWindow class “provides a framework for building an application’s user interface. Qt has “QMainWindow” and its related classes for main window management. “QMainWindow” has its own layout to which you can add “QToolBars”, “QDockWindows”, a “QMenuBar”, and a “QStatusBar”.”[7]

Note: There are other ways to create a visible window. For example you could use a QWidget to accomplish much of what we are going to do. I chose to use the QMainWindow since it provides an application framework that you might want to use when building a full GUI application.

We can subclass the QMainWindow as follows:

class HelloWindow(QtGui.QMainWindow):

	def __init__(self, win_parent = None):
		#Init the base class
		QtGui.QMainWindow.__init__(self, win_parent)

Then in the main block we can create and show our main window like this:

app = QtGui.QApplication(sys.argv)
#The Main window
main_window = HelloWindow()
main_window.show()
# Enter the main loop
app.exec_()

The entire source code thus far is found in Listing 1. When you run the code, you will finally be greeted with your very first PyQt4 window. It will look something similar to Figure 1.

Listing 1

#!/usr/bin/env python
import PyQt4
import sys
from PyQt4 import QtGui

class HelloWindow(QtGui.QMainWindow):

	def __init__(self, win_parent = None):
		#Init the base class
		QtGui.QMainWindow.__init__(self, win_parent)

if __name__ == "__main__":
	# Someone is launching this directly
	# Create the QApplication
	app = QtGui.QApplication(sys.argv)
	#The Main window
	main_window = HelloWindow()
	main_window.show()
	# Enter the main loop
	app.exec_()

Figure 1

Adding Some Widgets

A blank window is only slightly more useful then an invisible oen. Therefore our next step is to add some functionality to our window by adding some interactive widgets. In order to do this, we will set the central widget of our main window and use a “geometry manager” (or “layout manager”) class to manage the positioning of our widgets.

We will add a new function to our HelloWindow class called create_widgets where we will create all of our widgets. We will call this function from the __init__ function. The first step in the create_widgets function is to create a widget that will serve as the “central widget” for our main window. As mentioned earlier, the QMainWindow class already has a built-in layout making it easy to add menus, toolbars, and other items common to applications. QMainWindow classes also have a “central widget” which you can think of as the main part of the application. More to the point, the “central widget” is not the toolbars, menu, or docking panes, but the actual functional area of the application. For a greater description of the “central widget” see the PyQt4 website.[8]

We will create and set our “central widget” using the following code:

def create_widgets(self):
	central_widget = QtGui.QWidget()
	self.setCentralWidget(central_widget)

So far this won’t do anything, since we still have not added anything interactive. To add interactivity, we must add widgets to our “central widget”. Rather than place the widgets in static positions, we will use a layout manager to automatically position and adjust our widgets. If you are at all familiar with modern GUI toolkits, the idea of a layout manager should be familiar to you. They serve as a way to “pack” widgets into an area where their positions will be relative to each other and to the window. When you resize the window (or area), geometry manager’s will automatically adjust the size and positions of their child widgets to accommodate the new size.

We will be creating three widgets in our simple application:

1. A QLabel widget to display some static text.
2. A QLineEdit widget to let the user enter some text into an edit field.
3. A QPushButton button widget that will pop up a dialog to display a message when the user clicks on it.

We will create the widgets using the following code. Notice that in the creation of the QLabel and QPushButton widgets we are also setting the text that will appear on each widget. (We can do the same for the QLineEdit, but for now we’ll leave it blank):

def create_widgets(self):
	#Widgets
	self.label = QtGui.QLabel("Say hello:")
	self.hello_edit = QtGui.QLineEdit()
	self.hello_button = QtGui.QPushButton("Push Me!")

Next, we will create a QHBoxLayout instance to serve as the central widget’s layout manager. The QHBoxLayout class is a specific type of layout manager that lays widgets out, and manage their positions, in the horizontal direction. Once we have created the QHBoxLayout instance, we will add our three widgets to it. Adding widgets to a QHBoxLayout will pack them from left to right:

#Horizontal layout
h_box = QtGui.QHBoxLayout()
h_box.addWidget(self.label)
h_box.addWidget(self.hello_edit)
h_box.addWidget(self.hello_button)

The last step in our widget creation is to set h_box as the layout manager for our central widget:

#Create central widget, add layout, and set
central_widget = QtGui.QWidget()
central_widget.setLayout(h_box)
self.setCentralWidget(central_widget)

The entire code can be seen in Listing 2. When you run this code, you will finally be greeted with a window containing actual widgets. This is illustrated in Figure 2.

Listing 2

#!/usr/bin/env python
import PyQt4
import sys
from PyQt4 import QtGui

class HelloWindow(QtGui.QMainWindow):

	def __init__(self, win_parent = None):
		#Init the base class
		QtGui.QMainWindow.__init__(self, win_parent)
		self.create_widgets()

	def create_widgets(self):
		#Widgets
		self.label = QtGui.QLabel("Say hello:")
		self.hello_edit = QtGui.QLineEdit()
		self.hello_button = QtGui.QPushButton("Push Me!")
		#Horizontal layout
		h_box = QtGui.QHBoxLayout()
		h_box.addWidget(self.label)
		h_box.addWidget(self.hello_edit)
		h_box.addWidget(self.hello_button)
		#Create central widget, add layout and set
		central_widget = QtGui.QWidget()
		central_widget.setLayout(h_box)
		self.setCentralWidget(central_widget)

if __name__ == "__main__":
	# Someone is launching this directly
	# Create the QApplication
	app = QtGui.QApplication(sys.argv)
	#The Main window
	main_window = HelloWindow()
	main_window.show()
	# Enter the main loop
	app.exec_()

Figure 2

Signal Handling

Responding to the user’s interaction with the GUI is something that must be done in all GUI applications. Therefore the next step in constructing a GUI application is signal or event handling. If you are unfamiliar with these terms you can thing of it his way: when the end user interacts with your GUI, her interaction will causes signals to be sent to your application. So if she clicks on a button the clicked() signal will be sent. If your application wants to do something when that button is clicked, then you will have to handle that widgets clicked() signal. In our example, we are going to show a pop up a dialog when the user clicks on our button widget.

I find connecting to signals in PyQt4 a bit odd because of a built in macro that needs to be called but the process is relatively easy so I shouldn’t complain too much. The idea is similar to that of other Python GUI toolkits: you connect functions that you have written to signals or events that are emitted by a widget. So whenever a widget emits a signal that has been connected to a function, that function will be called.

Note: Qt goes a bit further with the idea of built-in “slots” but for the sake of this tutorial we will be ignoring them. For more information on “slots” please see the PyQt4 documentation.

In order for the “QPushButton” to do something when pushed, we have to connect a function in our HelloWindow class with the clicked signal. To do this, we need to call the QObject classes connect static function. We must tell it three things:

1. The QWidget that will emit the signal.
2. The signal that we want to connect to.
3. The function that should be called when the QWidget emits the signal.

In our example, we will do this like so:

#connect signal
QtCore.QObject.connect(self.hello_button
	, QtCore.SIGNAL("clicked()")
	, self.on_hello_clicked)

Note: We could just as easily have called self.connect since a QMainWindow is a descendant of the QObject class.

QtCore.SIGNAL is a macro that you have to use for the second parameter. It accepts the signal name as a string (in this case clicked()) and converts it into an object that is required for the signal connection. The documentation is a bit sparse regarding exactly what the macro does, but luckily for us we don’t really need to know the details.

In order for this code to compile, we also have to import the QtCore module:

from PyQt4 import QtCore

We also have to add a new function to the HelloWindow class. This is the function on_hello_clicked that will respond to the clicked() signal:

def on_hello_clicked(self):
	print "Clicked"

When you run the code in Listing 3, you will see “Clicked” printed out to the command line every time you click the “Push Me!” button.

Listing 3

#!/usr/bin/env python
import PyQt4
import sys
from PyQt4 import QtGui
from PyQt4 import QtCore

class HelloWindow(QtGui.QMainWindow):

	def __init__(self, win_parent = None):
		#Init the base class
		QtGui.QMainWindow.__init__(self, win_parent)
		self.create_widgets()


	def create_widgets(self):
		#Widgets
		self.label = QtGui.QLabel("Say hello:")
		self.hello_edit = QtGui.QLineEdit()
		self.hello_button = QtGui.QPushButton("Push Me!")

		#connect signal
		QtCore.QObject.connect(self.hello_button
			, QtCore.SIGNAL("clicked()")
			, self.on_hello_clicked)


		#Horizontal layout
		h_box = QtGui.QHBoxLayout()
		h_box.addWidget(self.label)
		h_box.addWidget(self.hello_edit)
		h_box.addWidget(self.hello_button)
		#Create central widget, add layout and set
		central_widget = QtGui.QWidget()
		central_widget.setLayout(h_box)
		self.setCentralWidget(central_widget)

	def on_hello_clicked(self):
		print "Clicked"
		
if __name__ == "__main__":
	# Someone is launching this directly
	# Create the QApplication
	app = QtGui.QApplication(sys.argv)
	#The Main window
	main_window = HelloWindow()
	main_window.show()
	# Enter the main loop
	app.exec_()

Displaying a Message

Now that we have a function connected to the clicking of the button widget we need to read the text from the QLineEdit widget and display it to the user. Fortunately both tasks are quite easy in PyQt4.

In order to get the text from a QLineEdit widget, the QLineEdit classes member function displayText needs to be called. The displayText function simply returns the contents of the QLineEdit widget.

Showing a pop-up dialog using PyQt4 is just as easy as getting the contents of a QLineEdit widget, thanks to the QMesssageBox classes static function information. This function allows you to set the title, text and what buttons are on a simple “information dialog”. Calling it from our PyQt4 application in response to the button click is very simple:

def on_hello_clicked(self):
	QtGui.QMessageBox.information(self
		, "Hello!"
		, "Hello %s" % self.hello_edit.displayText()
		, QtGui.QMessageBox.Ok)

In this code, we call the information function, set its title text to be “Hello!”, set its main message to be “Hello ” followed by the contents of the hello_edit QInputEdit, and give it one button, an “ok” button. All of the code is found in Listing 4. What this looks like running on Linux, Windows and OS X can be seen in Figure 3, Figure 4, and Figure 5 respectively.

Listing 4

#!/usr/bin/env python
import PyQt4
import sys
from PyQt4 import QtGui
from PyQt4 import QtCore

class HelloWindow(QtGui.QMainWindow):

	def __init__(self, win_parent = None):
		#Init the base class
		QtGui.QMainWindow.__init__(self, win_parent)
		self.create_widgets()


	def create_widgets(self):
		#Widgets
		self.label = QtGui.QLabel("Say hello:")
		self.hello_edit = QtGui.QLineEdit()
		self.hello_button = QtGui.QPushButton("Push Me!")

		#connect signal
		QtCore.QObject.connect(self.hello_button
			, QtCore.SIGNAL("clicked()")
			, self.on_hello_clicked)


		#Horizontal layout
		h_box = QtGui.QHBoxLayout()
		h_box.addWidget(self.label)
		h_box.addWidget(self.hello_edit)
		h_box.addWidget(self.hello_button)
		#Create central widget, add layout and set
		central_widget = QtGui.QWidget()
		central_widget.setLayout(h_box)
		self.setCentralWidget(central_widget)

	def on_hello_clicked(self):
		QtGui.QMessageBox.information(self
			, "Hello!"
			, "Hello %s" % self.hello_edit.displayText()
			, QtGui.QMessageBox.Ok)

if __name__ == "__main__":
	# Someone is launching this directly
	# Create the QApplication
	app = QtGui.QApplication(sys.argv)
	#The Main window
	main_window = HelloWindow()
	main_window.show()
	# Enter the main loop
	app.exec_()

Figure 3

Figure 4

Figure 5

Conclusion

That’s it for this quick introduction to PyQt4. All-in-all, I was pleasantly surprised by PyQt4. Even during my first foray into the toolkit, I found it very straightforward and was able to get things going in a matter of minutes. Granted, what I was trying to accomplish was pretty simple, but it’s always a good sign when doing something simple is easy!

You should now have enough information to go and create your very own “GUIfied” applications. Of course, there is much, much more to PyQt4 than I have covered in this tutorial. But if you use the PyQt4 documentation and the fundamentals from this tutorial, you shouldn’t have much trouble moving forward.

[1] http://www.python.org/download/
[2] http://www.riverbankcomputing.co.uk/pyqt/download.php
[3] http://trolltech.com/downloads
[4] http://www.riverbankcomputing.com/Docs/PyQt4/pyqt4ref.html#installing-pyqt
[5] http://www.riverbankcomputing.com/Docs/PyQt4/html/qapplication.html
[6] http://www.riverbankcomputing.com/Docs/PyQt4/html/qapplication.html#exec
[7] http://www.riverbankcomputing.com/Docs/PyQt4/html/qmainwindow.html#details
[8] http://www.riverbankcomputing.com/Docs/PyQt4/html/qmainwindow.html#details

If I had gotten done everything that I wanted to get done then I would be releasing the full first version of the project instead of this incredibly alpha release. So if something doesn’t work don’t be surprised. That being said you should be able to use it for it’s basic functionality. Or at least that’s what I’m hoping.

Installation

Once you have met the requirements:

• Python >= 2.5 http://www.python.org/download/
• GTK+ >= 2.10 http://www.pygtk.org/downloads.html
• PyGTK >= 2.10 http://www.pygtk.org/downloads.html
• pyGObject >= 2.10 http://www.pygtk.org/downloads.html
• pyglet >= 1.1 http://www.pyglet.org/download.html
• PyYAML > 3.0 http://pyyaml.org/wiki/PyYAML
• PyOpenGL 3.x http://pyopengl.sourceforge.net/
• PyGtkGlExt? http://www.k-3d.org/gtkglext/Main_Page

You should be able to install dodger by downloading the and extracting the source tarball. Once you have extracted the source change into the source directory and install using:


python setyup.py install


If you don’t want to install and just want to test dodger out simply follow the above instructions except instead of installing dodger run the following:


python runner.py


Contributing

I still want to setup a dedicated site that will host a mercurial repository of the code but for now I’m going to use the fabulous http://freehg.org site. For now if you want you can pull the source from  http://freehg.org/u/selsine/dodger/:


hg clone http://freehg.org/u/selsine/dodger/


For now any changes that you make or fixes will have to be emailed as a patch but eventually I want to make it much much easier for people to submit fixes.

If you are really interested in contributing to this project let me know via a comment to this post and or an email. Then once all of the source issue are setup we can start working together.

New Features

Since the last release there has been a lot of changes under the hood. The most visible changes for anyone using dodger is the addition of the zoom feature:

zoomed out example

zoomed in example

General Usage

For general usage information please read the README file or General Information page on the google site. It’s pretty rough right now but hopefully over time we’ll be able to work on it.

History

If you are interested in the history of this project you can read the following blog posts to see how everything came about:

• A Simple Python Game Engine?
• More thoughts on the simple Python Game Engine
• Pyglet Level Editor
• Level Editor 0.2
• Level Editor 0.3 (Dodger)

Beware

If you do decide to use or test dodger remember that you are going to find a lot of bugs during the iteration. Please be kind and report the issue.

Thanks for you patience everyone, I’ll be back with more soon!

I’ve been busy with work, Python Magazine, my wife, trips to Dallas, and yes whenever I get a chance this slowly growing level editor. Let’s see what I’ve been working on for the last little while:

Name: There has a been a name for the editor ever since I started working on it. I wasn’t sure if I was going to think up something really cool and change, or leave it. Turns out I just left it.

So from this point forward this project is christened: “Dodger”, or probably more correctly: “Dodger Level Editor”.

The name has its roots in the name of one of my cats and a history in the multitudes of level editors and game engines that I have tried to create in the past, but I wont’ go into that. So Dodger it is.

Saving and Loading: Saving and loading in the default YAML project type now really works. I still need to put in support for optional project types: XML, JSON, and other formats

Welcome Dialog: This was a real pain, but it’s made the last while so much easier. I added a working (at least I hope) welcome dialog with a way to create new projects, open old ones, and a recent file list. The recent file list really makes testing easier for me.

Rect tracking: Rect tracking is finally working properly. I’ve had the rect tracker in there for a while but it didn’t really do anything until now.

Multiple Selection/Multiple Properties: I’ve also finally got multiple selection going, which is what makes the rect tracker actually useful. You can select multiple sprties, move them around and add properties to all of them.

Remove Properties: Now you can remove custom properties that you have added. This was a must but I was lazy and left it for a while.

Steps towards being made public: A lot of the changes (and I do mean a lot) that I’ve been making have been behind the scenes. There has been a lot of refactoring and reorganizing of the code, often the result of quick and dirty implementations that I made earlier (sigh). I’ve also started working on getting the distribution of this editor going so that other people can use/develop it.

I’ve added support for zc.Buildout so that if anyone wants to develop they can quickly gather dependencies and won’t have to install the editor system wide, not that you have to anyways but zc.Buildout is really neat.

I’ve also worked on the license (GPLv3) and setup.py and README and all of that. None of it’s done but it’s working its way forward.

Faster: It’s also much faster now. None of you have used it so you’ll probably think it’s slow, but trust me it’s much faster then it was before.

Bugs: There have been loads of bugs that have been fixed and created. Plus one doozy related to
changes made to pyglet. Not pyglets fault but it took a long time to figure out what the issue was.

Google Code: The project has a temporary homepage over at google code: http://code.google.com/p/dodger-editor/ There’s nothing there yet but over time I will start to host the project there so that people can easily download it. I’ll still post updates here until the site has a full-time home.

I’m going to use Mercurial for the revision control system for the project so the CVS support at the google code site will just be for downloading. Eventually I will want to host the project on some space of my own and get a nice web interface for the mercurial repository going. I’ll have to find a new hosting company so it will take a while.

So that’s it, that’s what’s happened to the Dodger Level Editor over the last few months. I know I promised to make it public earlier but given the shape it was in at that time there really was no point. I want this to be at a point where people can actually almost use it before I make it public.

I know it’s been a while, and I know the few of you that actually care about this project have probably moved onto bigger and better things, but hopefully if you stick with me there will be something out soon.

Note: This article was first published the November 2007 issue of Python Magazine

While the equality operator works great on numbers and strings the fact the way it treats your custom objects really is not that useful. This article looks into overloading the equality operator so that you can easily compare your custom classes.

1. Introduction
2. Introducing the terms: operators and operator overloading
3. A Quick Example of the Default Equality Operator
4. Overloading the Equality Operator
5. Telling Python that the Comparison has Not Been Implemented
6. The Inequality Operator
7. Dangers
8. Conclusion

Introduction

In my experience as a professional programmer, testing for the equality between two instances of a class is a fairly common task. In other words, you are comparing the data that each class contains and checking whether the data in one class is identical to the data in the other class.

One of the nice features of Python is that it has a default equality operator defined for any custom objects that you create. The unfortunate thing about this default equality operator is that it doesn’t provide the functionality that you expect. This is because the equality operator (==) actually performs an identity comparison, rather than an equivalence test. If you were to run the following code:

if (object_one == object_two):

By default Python actually compares whether or not object_one is object_two (this is the same comparison that can be made using the is keyword) instead of determining whether or not object_one is equivalent to object_two. Fortunately for us, overloading the default equality operator in Python is a relatively easy task. There are, however, some “gotchas” and other interesting features of which one should be aware.

Introducing the terms: operators and operator overloading

An operator can be difficult to define, and like many programming definitions, sometimes the definition only serves to confuse the matter further. In general though, you can think of operators as being very similar to the operators that you encountered in Math class, such as: the + operator, the - operator, and so forth.

In Python the following are operators[1]:

+	-	*		/	//	%	<>>	&
|	^	~	<>	< =	>=	==	!=	<>

In programming languages we generally encounter binary operators. This means that each operator takes two operands. An operand serves as input to an operator. For example, in the statement:

2 + 6

+ is a binary operator that takes two operands, 2 and 6 as inputs. Similarly, in this statement:

my_value - 6

- is an operator that takes two operands, my_value and 6 as inputs.

Operator overloading is a programming term that means taking the default behaviour of an operator and overloading it. That is, changing the default implementation of an operator for a given object. An example of this (although something that you should never do) would be to overload the + operator to actually perform subtraction instead when it is applied to your class.

A Quick Example of the Default Equality Operator

Now that the definitions are out of the way, let’s look at an example where one might want to overload the equality operator. For this example I will bring back a favourite example from my Computer Science days: the Student class:

class Student(object):

	def __init__(self, name, student_number):
		self.name = name
		self.student_number = student_number

As you can see the Student class has two data members: 1) the student’s name, and, 2) her student number.

If we run the following code:

mark = Student("Mark Mruss", 067213)
guido = Student("Guido van Rossum", 000001)
if (mark == guido):
	print "Equal"
else:
	print "Not Equal"

“Not Equal” will be printed out as you would expect since the two students are clearly not equivalent. But what about this code:

mark = Student("Mark Mruss", 067213)
mark_two = Student("Mark Mruss", 067213)
if (mark == mark_two):
	print "Equal"
else:
	print "Not Equal"

Here, as in the previous example, “Not Equal” will be printed out. This is because, as mentioned earlier, the default implementation of the equality operator is to perform an identity comparison. In other words, the default equality operator asks, is mark the same object as mark_two? In Python the equality comparison depends on the type of objects being compared. For custom classes that you or I will create, the equality comparison will perform an identity comparison by comparing the object’s internal id. In other words, it will only result in True if the objects being compared actually are each other. For example:

student_one = Student("Mark Mruss", 067213)
student_two = student_one
if (student_one == student_two):
	print "Equal"
else:
	print "Not Equal"

Results in “Equal” being printed out, as would:

student_one = Student("Mark Mruss", 067213)
student_two = student_one
if (id(student_one) == id(student_two)):
	print "Equal"
else:
	print "Not Equal"

Note: The equality comparison for built-in objects and types like numbers, strings, lists, tuples, and mappings behave differently. Numbers are compared arithmetically. The numerical values of the characters within strings are compared arithmetically. The comparison of lists and tuples is simply a comparison of their inner values, while the comparison of mappings are comparisons of an ordered list of their values.[2]

Overloading the Equality Operator

Hopefully the above example illustrated a case where we might want to overload the equality operator to make it so that the following code:

student_one = Student("Mark Mruss", 067213)
student_two = Student("Mark Mruss", 067213)
if (student_one == student_two):
	print "Equal"
else:
	print "Not Equal"

Would result in “Equal” being printed out, i.e. a true equality comparison as opposed to an identity comparison. In order to do this we need to change to the default functionality of the equality operator. In other words we need to overload it.

In general, operator overloading in Python means adding a special function to your class that will perform the function of the operator it is meant to represent. There are two ways in which one can overload the equality operator in Python: 1) the first method is to use the __eq__ function, a so-called “rich comparison” function. “Rich comparison” functions are functions that overload specific comparison operators (i.e. __eq__ to overload ==). 2) The second is to use the __cmp__ function, which is used to overload all comparison operators if no “rich comparison” functions are present.

Since __cmp__ is used to override all comparison operators (==, !=, < , <=, >, >=), I would suggest using the “rich comparison” method unless you are using a version of Python that is earlier then version 2.1, or you are convinced that you know what < = means to our Student class. Let’s forget about the __cmp__ operator for now and focus on using the “rich comparison” functions to overload the equality operator.

“Rich comparison” functions can return any value, but you should try to return a value that is, or can be, interpreted as a boolean value. This is important because these functions will often be used in situations where the return value will be used in a boolean comparison.

When using the “rich comparison” functions it is important to know which functions are being called internally. For example, when we run:

student_one == student_two

If __eq__ exists in the Student class, the following is actually being called:

student_one.__eq__(student_two)

When we run:

student_two == student_one

The following is actually called:

student_two.__eq__(student_one)

As you can see it is the operand on the left-hand side whose __eq__ function will be called. It is important to note that if the operand on the left-hand side lacks the __eq__ function while the operand on the right-hand side has one, the right-hand operand’s __eq__ function will not be called.

Lets start off with a simple, but incorrect, example (the reasons for its incorrectness will be explained below):

def __eq__(self, other):
	return ((self.name == other.name)
		and (self.student_number == other.student_number))

This is very straightforward. In the equality comparison, we simply compare the Student class’ two data members. This performs as expected when we run:

student_one = Student("Mark Mruss", 067213)
student_two = Student("Guido van Rossum", 000001)
student_three = Student("Mark Mruss", 000001)
print (student_one == student_two)
print (student_one == student_three)

You get:

False
True

But what happens when we introduce the Professor class and try the overloaded equality operator:

class Professor(object):

	def __init__(self, instructor, course):
		self.instructor = instructor
		self.course = course

As you can see, the Professor class lacks the name and student_number data members. What happens when we compare an instance of the Professor class with an instance of the Student class?

guido = Student("Guido van Rossum", 000001)
rob = Professor("Rob Ward", "74-300")
print (guido == rob)

It results in something like this:

File "operators.py", line 10, in __eq__
    return ((self.name == other.name)
AttributeError: 'Professor' object has no attribute 'name'

The way we are overriding the equality operator is not correct because it automatically assumes that the other object has the name and student_number data members. There are a number of methods to get around this problem, including: 1) using the hasattr function, or 2) using the isinstance function. Using the hasattr function determines if other has the attributes we are looking for before actually querying them. hasattr simply tells us if an object has a specific attribute or not. Here is a quick example illustrating how to do this:

def __eq__(self, other):
	if (hasattr(other, "name") and hasattr(other, "student_number")):
		return ((self.name == other.name)
			and (self.student_number == other.student_number))
	else:
		return False

First, we check to see if other has the name and student_number attributes. If it does, we proceed as normal. If it does not, we simply return false. When we compare the professor and the student we get “False” as expected.

What’s nice about this method is that we don’t have to care what type other is. We only care whether or not it contains the attributes we need to compare. However, the drawback to this function is that you have to test for the existence of each attribute. Although this may not always be a big deal, if you are dealing with fifty data members in your classes this can quickly become a pain in the neck.

Another solution to the problem with our first overloading example is to use the isinstance function to make sure that other is an instance of our class type. This has the drawback of forcing other to be the same type as your class. In practice however, I believe this to be more of an advantage than a disadvantage.

def __eq__(self, other):
	if (isinstance(other, Student)):
		return ((self.name == other.name)
			and (self.student_number == other.student_number))
	else:
		return False

The first thing we do is check the variable other to make sure that it is an instance of the Student class. If it is, we then compare all of the data members in the Student class. If object is not an instance of the Student class, we return False.

In my opinion, this is the preferred method since knowing that the class is the correct type is often important. The hasattr method seems more appropriate for simple data containers like a “rect” or “vector” class where you are only interested in three or four data members.

Telling Python that the Comparison has Not Been Implemented

Up until this point in time we have been returning False when our __eq__ function does not support the type of object passed in as other. While this is acceptable and correct given the Python documentation, it seems to be “proper” to actually return NotImplemented. According to the Python documentation, “Numeric methods and rich comparison methods may return this value if they do not implement the operation for the operands provided. (The interpreter will then try the reflected operation, or some other fallback, depending on the operator.)” [4]Let’s forget abou In other words, if the left operand returns NotImplemented, Python will attempt to use the right hand operand’s equality operator. And if that does not exist, Python will fall back to the default equality operator.

We can return NotImplemted from our Student class if the operand passed in is not an instance of the Student:

def _eq__(self, other):
	if (isinstance(other, Student)):
		return ((self.name == other.name)
			and (self.student_number == other.student_number))
	else:
		return NotImplemented

Now if we perform the following comparison:

guido = Student("Guido van Rossum", 000001)
rob = Professor("Rob Ward", "74-300")
print guido == rob

The first step in the processing will be:

guido.__eq__(rob)

This returns NotImplemented. As a result, the reflected operation is attempted:

rob == guido

Because the Professor class does not have the equality operator overloaded, the default operation is executed and False is printed out just like we wanted.

NotImplemented is useful in because instead of returning False, which means that the two operand are not equivalent, you return a value that says that the comparison between the operands has not been implemented.

The Inequality Operator

Now that we know how to overload the equality operator, it stands to reason that we have the opposite operation, the inequality operator (!=) covered as well. But not so fast. In Python the inequality and equality operators are handled separately, meaning that inequality is not simply the opposite of equality. This means that whenever you overload the equality operator, you have to be sure to overload the inequality operator as well. If you don’t you might get some strange results. For example, when we use the current code (without the inequality operator overloaded), the following:

guido = Student("Guido van Rossum", 000001)
guido_too = Student("Guido van Rossum", 000001)
print guido == guido_too
print guido != guido_too

Results in:

True
True

In the first comparison the overloaded equality operator is used, and results in True being printed. Because the inequality operator is not overloaded in the second comparison, the default inequality operator is used (the identity comparison). True is printed because guido and guido_too are not the same instances.

Thankfully once you have overloaded the equality operator, overloading the inequality operator is very easy. As a general rule, you have to return the opposite of the equality operator, but because we are working with NotImplemented, we have to do a bit more processing to ensure that we don’t return False when we really want to return NotImplemented. Here is how we can overload the inequality operator in the Student class:

def __ne__(self, other):
	equal_result = self.__eq__(other)
	if (equal_result is not NotImplemented):
		return not equal_result
	return NotImplemented

First, we call self.__eq__ to test whether or not we are equal to other. We then check to make sure that equal_result is not NotImplemented. If it is not, we know that the equality test was implemented and we can safely return its’ opposite. If the result for the equality comparison was NotImplemented, we return NotImplemented for the inequality comparison.

Note: It is safe to use the is check on NotImplemented (rather than an isinstance check) because NotImplemented is a singleton, meaning that there is only ever one instance of NotImplemented at anytime.

Dangers

While it may seem like operator overloading should become part of every class that you write, a word of warning is necessary. There is a large school of thought that views operator overloading as a dangerous programming technique. They argue that overloading operators changes the default way that an operator works, and not always correctly. Moreover, instead of overriding the equality operator, one can simply add an is_equal_to function to perform the equality check.

The logic behind this criticism is that when someone is using a class or reading some code that you wrote, they will be unable to tell what the equality operator is doing. For example, if they see:

value = MyClass(10)
value_two = MyClass(10)
print value == value_two

What gets printed out? True or False? If “MyClass” overrode the equality operator then True will be printed. However, if the equality operator is not overloaded, the standard Python behaviour of equality will result with False being printed out.

Conclusion

While it’s true that overloading the equality operator does change the default way the Python functions, I feel that it’s generally a safe and beneficial addition to your classes. Especially since unless people know the ins and outs of the equality operator they will generally assume that should work the way it does when you overload it. Like all the decisions that you make when working with Python, context is key.

[1] http://docs.python.org/ref/operators.html
[2] http://docs.python.org/ref/comparisons.html
[3] http://docs.python.org/ref/customization.html
[4] http://docs.python.org/ref/types.html

Hey everyone,

Just to let you know there is a strong possibility that this blog was hacked recently. A helpful user (Eoin) informed me that when trying to visit the blog from Google he was re-directed to an advertising website.

In response I’ve updated the blog and tried my best to ensure that anything that was hacked is back to normal.

If anyone noticed anything strange happening on this blog please let me know.

Thanks.

Fraking hackers…

Note: This article was first published the October 2007 issue of Python Magazine

XML is everywhere. It seems you can’t do much these days unless you utilize XML in one way or another. Fortunately, Python developers have a new tool in our standard arsenal: the ElementTree module. This article aims to introduce you to reading, writing, saving, and loading XML using the ElementTree module.

1. Introduction
2. Reading XML data
3. Listing 1
4. Listing 2
5. Reading XML Attributes
6. Writing XML
7. Listing 3
8. Writing XML Attributes
9. Reading XML Files
10. Writing XML Data to a File
11. Reading from the Web
12. Conclusion

Introduction

It seems like everyone needs to parse XML these days. They’re either saving their own information in XML or loading in someone else’s data. This is why I was glad to learn that as of Python 2.5, the ElementTree XML package has been added to the standard library in the XML module.

What I like about the ElementTree module is that it just seems to make sense. This might seem like a strange thing to say about an XML module, but I’ve had to parse enough XML in my time to know that if an XML module makes sense the first time you use it, it’s probably a keeper. The ElementTree module allows me to work with XML data in a way that is similar to how I think about XML data.

A subset of the full ElementTree module is available in the Python 2.5 standard library as xml.etree, but you don’t have to use Python 2.5 in order to use the ElementTree module. If you are still using an older version of Python (1.5.2 or later) you can simply download the module from its website and manually install it on your system. The website also has very easy to follow installation instructions, which you should consult to avoid issues while installing ElementTree.

In general, the ElementTree module treats XML data as a list of lists. All XML has a root element that will have zero or more subelements (or child elements). Each of those subelements may in turn have subelements of their own. The best way to think about this is with a brief example.

First let’s take a look at some sample XML data:

<root>
<child>One</child>
<child>Two</child>
</root>

Here we have a root element with two child elements. Each child element has some text associated with it seen here as “one” and “two”. If we examine the XML as a hierarchical list of lists we see that we have one element “root” in our root list. Within the “root” element we have a list containing two subelements “child” and “child”. The two “child” elements would then contain empty lists representing their lack of subelements. Not too complicated so far, is it?

Reading XML data

Now let’s use the ElementTree package to parse this XML and print the text data associated with each child element. To start, we’ll create a Python file with the contents shown in Listing 1.

Listing 1

#!/usr/bin/env python

def main():
	pass

if __name__ == "__main__":
	main()

This is basically a template that I use for many of my simple “*.py” files. It doesn’t actually do anything except set up the script so that when the file is run, the main method will be executed. Some people like to use the Python interactive interpreter for simple hacking like this. Personally, I prefer having my code stored in a handy file so I can make simple changes and re-run the entire script when I am just playing around.

The first thing that we need to do in our Python code is import the ElementTree module:

from xml.etree import ElementTree as ET

Note: If you are not using Python 2.5 and have installed the ElementTree module on your own, you should import the ElementTree module as follows:

from elementtree import ElementTree as ET

This will import the ElementTree section of the module into your program aliased as ET. However, you don’t have to import ElementTree using an alias; you can simply import it and access it as ElementTree. Using ET is demonstrated in the Python 2.5 “What’s new” documentation[1] and I think it’s a great way to eliminate some key strokes.

Now we’ll begin writing code in the main method. The first step is to load the XML data described above. Normally you will be working with a file or URL; for now we want to keep this simple and load the XML data directly from the text:

element = ET.XML(
       "<root><child>One</child><child>Two</child></root>")

The XML function is described in the ElementTree documentation as follows: “Parses an XML document from a string constant. This function can be used to embed “XML literals” in Python code”[2].

Be careful here! The XML function returns an Element object, and not an ElementTree object as one might expect. Element objects are used to represent XML elements, whereas the ElementTree object is used to represent the entire XML document. Element objects may represent the entire XML document if they are the root element but will not if they are a subelement. ElementTree objects also add “some extra support for serialization to and from standard XML.”[3] The Element object that is returned represents the element in our XML data.

Thankfully, the Element object is an iterator object so we can use a for loop to loop through all of its child elements:

for subelement in element:

This will give us all the child elements in the root element. As mentioned earlier, each element in the XML tree is represented as an Element object, so as we iterate through the root element’s child elements we are getting Element objects with which to work. Meaning that each loop though the for loop will give us the next child element in the form of an Element object until there are no more children left. In order to print out the text associated with an Element object we simply have to access the Element object’s text attribute:

for subelement in element:
       print subelement.text

To recap, have a look at the code in Listing 2.

Listing 2

#!/usr/bin/env python

from xml.etree import ElementTree as ET

def main():
	element = ET.XML("<root><child>One</child><child>Two</child></root>")
	for subelement in element:
		print subelement.text

if __name__ == "__main__":
	# Someone is launching this directly
	main()

Once you run the code you should get the following output:

One
Two

If an XML element does not have any text associated with it, like our root element, the Element object’s text attribute will be set to None. If you want to check if an element had any text associated with it, you can do the following:

if element.text is not None:
       print element.text

Reading XML Attributes

Let’s alter the XML that we are working with to add attributes to the elements and look at how we would parse that information.

If the XML uses attributes in addition to, or instead of, inner text they can be accessed using the Element object’s attrib attribute. The attrib attribute is a Python dictionary and is relatively easy to use:

def main():
       element = ET.XML(
               '<root><child val="One"/><child val="Two"/></root>')
       for subelement in element:
               print subelement.attrib

When you run the code you get the following output:

{'val': 'One'}
{'val': 'Two'}

These are the attributes for each child element stored in a dictionary. Being able to work with an XML element’s attributes as a Python dictionary is a great feature and fits well with the dynamic nature of XML attributes.

Writing XML

Now that we’ve tried our hand at reading XML, let’s try creating some. If you understand the reading process, you should have no trouble understanding the creation process because it works in much the same manner. What we are going to do in this example is recreate the XML data that we were working with above.

The first step is to create our element:

#create the root <root>
root_element = ET.Element("root")

After this code is executed, the variable root_element is an Element object, just like the Element objects that we used earlier to parse the XML.

The next step is to create the two child elements. There are two ways to do this.

In the first method, if you know exactly what you are creating, it’s easiest to use the SubElement method, which creates an Element object that is a subelement (or child) of another Element object:

#create the first child <child>One</child>
child = ET.SubElement(root_element, "child")

This will create a Element that is a child of root_element. We then need to set the text associated with that element. To do this we use the same text attribute that we used in the first parsing example. However, instead of simply reading the text attribute we set its value:

child.text = "One"

The second approach to creating a child element is to create an Element object separately (rather than a sub element) and append it to a parent Element object. The results are exactly the same - this is simply a different approach that may come in handy when creating your XML,or working with two sets of XML data.

First we create an Element object in the same way that we created the root element:

#create the second child <child>Two</child>
child = ET.Element("child")
child.text = "Two"

This creates the child Element object and sets its text to “Two”. We then append it to the root element:

#now append
root_element.append(child)

Pretty simple! Now, if we want to look at the contents of our root_element (or any other Element object for that matter) we can use the handy tostring function. It does exactly what it says that it does: it converts an Element object into a human readable string.

#Let's see the results
print ET.tostring(root_element)

Listing 3

#!/usr/bin/env python

from xml.etree import ElementTree as ET

def main():
	#create the root </root><root>
	root_element = ET.Element("root")
	#create the first child <child>One</child>
	child = ET.SubElement(root_element, "child")
	child.text = "One"
	#create the second child <child>Two</child>
	child = ET.Element("child")
	child.text = "Two"
	#now append
	root_element.append(child)
	#Let's see the results
	print ET.tostring(root_element)

if __name__ == "__main__":
	# Someone is launching this directly
	main()

To recap, have a look at the code in Listing 3. When you run this code you will get the following output:

</root><root><child>One</child><child>Two</child></root>

Writing XML Attributes

If you want to create the XML with attributes (as illustrated in the second reading example), you can use the Element object’s set method. To add the val attribute to the first element, use the following:

child.set("val","One")

You may also set attributes when you create Element objects:

child = ET.Element("child", val="One")

Reading XML Files

Most of the time you won’t be working with XML data that you explicitly create in your code, instead you will usually read the XML data in from a data source, work with it, and then save it back out when you are done. Fortunately, configuring ElementTree to work with different data sources is very easy. For example, let’s take the XML data that we first used and save it into a file named our.xml in the same location as our Python file.

There are a few methods that we can use to load XML data from a file. We are going to use the parse function. This function is nice because it will accept, as a parameter, the path to a file OR a “file-like” object. The term “file-like” is used on purpose because the object does not have to be a file object per se - it simply has to be an object that behaves in a file-like manner. A “file-like” object is an object that implements a “file-like” interface, meaning that it shares many (if not all) methods with the file object. If an object is “file-like” this fact will usually be prominently mentioned in its documentation.

The first thing that we need in order to load the XML data is determine the full path to the our.xml file. In order to calculate this, we determine the full path of our Python source file, strip the filename from it, and then append our.xml to the path. This is rather simple given that the __file__ attribute (available in Python 2.2 and later) is the relative path and filename of our Python source file. Although the __file__ attribute will be a relative path, we can use it to calculate the absolute path using the standard os module:

import os

We then call the abspath function to get the absolute path:

xml_file = os.path.abspath(__file__)

However, since we only want the directory name (not the full path and filename of our Python source file) we have to strip off the filename:

xml_file = os.path.dirname(xml_file)

Now that we have the directory in which the our.xml file resides, all we have to do is append the our.xml filename to the xml_file variable. However, instead of just doing something like:

xml_file += "/our.xml"

we will use the os module to join the two paths so that the resulting path is always correct regardless of what operating system our code is executed on:

xml_file = os.path.join(xml_file, "our.xml")

Note: If you have any trouble understanding what any of the code used to determine the path of our.xml is doing, try printing out xml_file after each of the above lines and it should become clear.

We now have the full path to the our.xml file. In order to load its XML data we simply pass the path to the parse function:

tree = ET.parse(xml_file)

We now have an ElementTree object instance that represents our XML file.

Since we are working with files, we should watch out for incorrect paths, I/O errors, or the parse function failing for any other reason. If you wish to be extra careful, you can wrap the parse function in a try/except block in order to catch any exceptions that may be thrown:

try:
       tree = ET.parse("sar")
except Exception, inst:
       print "Unexpected error opening %s: %s" % (xml_file, inst)
       return

In the except block, I catch the Exception base class so that I catch any and all exceptions that may be thrown (in the case of a missing file it will most likely be an IOError exception).

Writing XML Data to a File

Now that we know how to read in XML data, we should look at how one writes XML data out to a file. Let’s assume that after reading in the out.xml fiie we want to add another item to the XML file that we just read in:

child = ET.SubElement(tree.getroot(), "child")
child.text = "Three"

Notice that in order to add a child to the root element we used the ElementTree object’s getroot function. The getroot function simply returns the root Element object of the XML data.

Now that we have a third child element, let’s write the XML data back out to our.xml. Thanks to ElementTree this is a painless experience:

tree.write(xml_file)

That’s it!

If we want to be really careful when writing the XML data out to a file, we’ll watch out for exceptions. However most of the time the write method will succeed without throwing an exception; it is more important to be sure that the path used is correct. Often times, instead of getting the exception that you want, you end up with an XML file stored in some far off and strange location on your hard drive because your path was incorrect or you did not specify the full path. But, as is often the case when programming, better safe than sorry:

try:
       tree.write(xml_file)
except Exception, inst:
       print "Unexpected error writing to file %s: %s" % (xml_file, inst)
       return

To recap you can find all of the code from this section in Listing 4.

Listing 4

#!/usr/bin/env python

from xml.etree import ElementTree as ET
import os

def main():

	xml_file = os.path.abspath(__file__)
	xml_file = os.path.dirname(xml_file)
	xml_file = os.path.join(xml_file, "our.xml")

	try:
		tree = ET.parse(xml_file)
	except Exception, inst:
		print "Unexpected error opening %s: %s" % (xml_file, inst)
		return

	child = ET.SubElement(tree.getroot(), "child")
	child.text = "Three"

	try:
		tree.write(xml_file)
	except Exception, inst:
		print "Unexpected error writing to file %s: %s" % (xml_file, inst)
		return

if __name__ == "__main__":
	# Someone is launching this directly
	main()

When you run the code and take a look at the our.xml file you should see that the the third child element has been added:

<root>
<child>One</child>
<child>Two</child>
<child>Three</child>
</root>

Reading from the Web

Working with a local file is very useful, but you might also be in a situation where you will have to work with an XML file that is located on the Internet, perhaps an RSS feed. Fortunately, since the parse function explained above works with file-like elements, loading a URL is very easy.

First off, you need to import the urllib module; a standard module that allows you to open URLs in a method similar to opening local files:

import urllib

In order to open a URL we use:

feed = urllib.urlopen("http://pythonmagazine.com/c/news/atom")
tree = ET.parse(feed)

Conclusion

And that’s that! This concludes our brief introduction to XML parsing using the ElementTree module. Hopefully throughout this article you have seen how easy it is to create and manipulate XML using ElementTree …and I’ve only scratched the surface. For more information take a look at the official Python documentation and some of the great examples on the effbot website. I’m sure you’ll be an XML wizard in no time.

[1] http://docs.python.org/whatsnew/modules.html#SECTION0001420000000000000000
[2] http://effbot.org/zone/pythondoc-elementtree-ElementTree.htm#elementtree.ElementTree.XML-function
[3] http://effbot.org/zone/pythondoc-elementtree-ElementTree.htm#elementtree.ElementTree.ElementTree-class