Note: This article was first published the March 2008 issue of Python Magazine

Mark Mruss

Over the past few years Google has expanded it’s services beyond those of a normal search engine. One of those new services is the Google Calendar. This article will provide an introduction to working with the Google Calendar using Python.

Introduction

As many of you know, Google has branched out and started offering more services besides their ubiquitous search engine. You have email, calendars, documents, spreadsheets, photos, maps, videos, source code hosting, and the list goes on. Fortunately for us Python programmers, Google released the Google data Python Client Library on March 26th, 2007, giving Python programmers easy access to some of these services.

What the Google data Python Client Library, or “gdata-python-client”, does is provide “a library and source code that makes it easy to access data through Google Data APIs.” [1] This leads to the question: “what are the Google Data APIs?” In the words of Google: “The Google data APIs provide a simple standard protocol for reading and writing data on the web. These APIs use either of two standard XML-based syndication formats: Atom or RSS.”[2]

The Google services that use the Google data APIs include many of the services that we have grown to know and love:

• Google Apps
• Google Base
• Blogger
• Google Calendar
• Google Code Search
• Google Documents
• Google Notebook
• Picasa Web Albums
• Spreadsheets
• YouTube

This tutorial will only deal with the Google Calendar service specifically, but it’s important to know that many of the techniques used here can easily be applied to other Google services.

The Google data Python Client Library requires Python 2.2 or greater and the ElementTree module to be installed. I recommend using Python 2.5 since the ElementTree module is included in that release of Python. This column assumes that you are using Python 2.5 and version 1.0.10 of the “gdata-python-client”.

Getting and Installing the “gdata-python-client”

We must first download and install the “gdata-python-client” files from the “gdata-python-client” website.[3] Once you have downloaded the compressed file, extract it’s contents to a folder. You will then need to browse to that folder and run the extracted setup.py file with the install command with root access. For me, the command looked like this:

# python2.5 setup.py install

Notice that I ran python2.5 instead of simply python. I did this to ensure that Python 2.5 was used instead of a previous version of Python.

Once you have done this, you can test to make sure that the gdata-python-client module has been installed properly by trying to import the gdata module. You can test this easily in the interactive Python shell:

$ python2.5
Python 2.5.2a0 (r251:54863, Jan  3 2008, 17:59:56)
[GCC 4.2.3 20071123 (prerelease) (Debian 4.2.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import gdata
>>>

If everything works properly you should see no errors when importing gdata. If you do run into problems, you can consult the install.txt file that was extracted with the rest of the files in the “gdata-python-client” archive that you downloaded.

Getting Started

There are some great Google Calendar examples that come with the “gdata-python-client” module. I found them to be very useful learning tools. The “Google Calendar Developers Guide”[4] is also very helpful. If you get stuck it is essential reading. The “Google Calendar API Reference Guide”[5] is also an indispensable help.

It is important to remember that when you are working with the Google Calendar service, you are actually working with XML data. You are sending and receiving XML data to and from the Google Calendar service – Atom or RSS feeds to be precise. The Python classes that wrap these feeds, or XML blocks, are dynamically “formed” around the XML. When you access something like the following:

e_link.href

You are actually accessing the “href” attribute of an XML link block that may look something like this:

<ns0 :link href="http://www.google.com/calendar/feeds/<username>/private/full" rel="alternate" type="application/atom+xml" xmlns:ns0="http:www.w3.org/2005/Atom" />

In the above example, if we had the link as an instance (perhaps an atom.Link object), e_link.rel would be equal to “alternate”.

Logging In

There are two types of Google calendars – public calendars and private calendars. Public calendars do not require any authentication while private calendars do. There are three forms of authentication for private calendars: “AuthSub proxy” authentication, “ClientLogin” authentication, and “Magic cookie” authentication.

“AuthSub proxy” authentication is meant for web applications so it will not be covered in this tutorial. “Magic cookie” authentication requires a string (the magic cookie) obtained from your Google Calendar settings page. This type of authentication gives you read only access to a private calendar. Its usage is quite specific and will not be covered in this column either.

For this column we will focus on good old-fashioned “ClientLogin” authentication. In order to perform “ClientLogin” authentication we need two things: a Google Calendar username and a password. If you do not have these, head over to the Google Calendar website [6] and get yourself signed up. If you have gmail or have signed up for other Google services, you can probably sign in using that username and password.

Note: I say username and so does the documentation, but this will actually be an email address.

We are going to read the username and password in from the command line using the raw_input function and the getpass module. I must thank the people who wrote the Google code examples for introducing me to the getpass module. Before looking through the examples, I had not heard of it.

The raw_input function simply reads a line of input from the command line and returns it with the newline stripped. The getpass.getpass function does the exact same thing as raw_input except it does not echo the input, which is handy when working with passwords.

We need to import the getpass module:

import getpass

Then, in our main function we will ask the user to input their username and password:

def main():
    username = raw_input("Enter your username: ")
    password = getpass.getpass("Enter your password: ")

if __name__ == "__main__":
    main()

Now that we have our username and password, we need to log the user in. To do this we are going to use the CalendarService class. The CalendarService “extends the GDataService to streamline Google Calendar operations.” [7] The GDataService class “provides CRUD ops. and programmatic login for GData services.” [8]. CRUD is an acronym that stands for Create, Retrieve, Update, and Delete. What the CalendarService class will allow us to do is create, retrieve, update, and delete things from a Google Calendar.

The first step in logging in is to create an instance of the CalendarService class. To do this we will pass in the username and password that we have collected:

calendar_service = gdata.calendar.service.CalendarService(username
    , password
    , "PythonMagazine-Calendar_Example-1.0")

The third parameter that we pass to the CalendarService constructor is the “source” string. This is a short “string identifying your application, for logging purposes. This string should take the form: “companyName-applicationName-versionID”"[9]

Now that we have a CalendarService instance, we need to login. To do this we will use the ProgrammaticLogin function. This will log into the Google Calendar using the CalendarService classes current username and password. The ProgrammaticLogin function can raise three possible exceptions that we want to be aware of:

1. CaptchaRequired – Raised if the login requires a “captcha” response in order to login.
2. BadAuthentication – Raised if the username and/or password were not accepted by the Google Calendar.
3. Error – Raised if a 403 error occurs that is not a “CaptchaRequired” or “BadAuthentication” error.

For this example, shown in Listing 1, we will only worry about the “BadAuthentication” exception.

Listing 1

def main():
    username = raw_input("Enter your username: ")
    password = getpass.getpass("Enter your password: ")

    calendar_service = gdata.calendar.service.CalendarService(username
        , password
        , "PythonMagazine-Calendar_Example-1.0")
    try:
        calendar_service.ProgrammaticLogin()
    except gdata.service.BadAuthentication, e:
        print "Authentication error logging in: %s" % e
        return
    except Exception, e:
        print "Error Logging in: %s" % e
        return

Working With Calendars

Now that we’ve logged into the calendar service, we can start working with the user’s available calendars. To get a list of all of the available calendars you can call the GetAllCalendarsFeed function. This will return a CalendarListFeed instance representing all of the user’s calendars. Since I want to show how to add and delete Calendars, I’m going to use the GetOwnCalendarsFeed function to get a list of all of the calendars that the “authenticated user has owner access to.”[10]

The GetOwnCalendarsFeed function returns a CalendarListFeed class instance. This contains some information (like a title) along with a list of CalendarListEntry classes. Each CalendarListEntry in this list represents a calendar. An example of a function that uses GetOwnCalendarsFeed and then prints out information about each calendar can be found in Listing 2. Sample output from this function can be found in Listing 3. The function was passed a CalendarService after logging in:

list_own_calendars(calendar_service)

Listing 2

def list_own_calendars(calendar_service):

    try:
        #Get the CalendarListFeed
        all_calendars_feed = calendar_service.GetOwnCalendarsFeed()
    except Exception, e:
        print "Error getting all calendar feed: %s" % (e)
        return
    #Print the feed's title
    print all_calendars_feed.title.text
    #Now loop through all of the CalendarListEntry items.
    for (index, cal) in enumerate(all_calendars_feed.entry):
        #Print out the title and the summary if ther is one
        if (cal.summary is not None):
            print "%d) %s - Summary: %s" % (
                index, cal.title.text, cal.summary.text)
        else:
            print "%d) %s" % (index, cal.title.text)
        #Print out the authors
        print "\tAuthors:"
        for author in cal.author:
            print "\t\t%s" % (author.name.text)
        #Print out other information
        print "\tPublished: %s \n\tUpdated: %s \n\ttimezone: %s" % (
            cal.published.text, cal.updated.text, cal.timezone.value)
        print "\tColour: %s \n\tHidden: %s \n\tSelected: %s" % (
            cal.color.value, cal.hidden.value, cal.selected.value)
        print "\tAccess Level: %s" % (cal.access_level.value)

Listing 3

Mark Mruss's Calendar List
0) Mark Mruss - Summary: Main Calendar
	Authors:
		Mark Mruss
	Published: 2008-02-07T04:15:15.701Z
	Updated: 2008-02-07T03:43:26.000Z
	timezone: America/Toronto
	Colour: #5229A3
	Hidden: false
	Selected: true
	Access Level: owner
1) PythonMagazine - Summary: Calendar for Articles
	Authors:
		PythonMagazine
	Published: 2008-02-07T04:15:15.702Z
	Updated: 2008-02-07T03:37:15.000Z
	timezone: America/Toronto
	Colour: #0D7813
	Hidden: false
	Selected: true
	Access Level: owner

Adding a Calendar

If we want to add a calendar, we need to create a CalendarListEntry class instance and then call the CalendarService classes InsertCalendar function. Let’s say I wanted to add a Banking calendar to my account, I could do the following to create the CalendarListEntry:

new_calendar = gdata.calendar.CalendarListEntry()
new_calendar.title = gdata.atom.Title(text="Banking")
new_calendar.summary = gdata.atom.Summary(text="Bills and payments")
new_calendar.timezone = gdata.calendar.Timezone(value="America/Toronto")
new_calendar.hidden = gdata.calendar.Hidden(value="false")
new_calendar.selected = gdata.calendar.Selected(value="true")

The calendar now needs to be added to the account:

try:
    created_calendar = calendar_service.InsertCalendar(new_calendar)
except gdata.service.RequestError, e:
    print "Error adding calendar: %s" % (e[0]["reason"])

This will add the new calendar to the authenticated users list of calendars, make it visible, and select it. Notice that InsertCalendar also returns a CalendarListEntry instance. This is the calendar that was actually added to the authenticated user’s list of calendars. It is wrapping the XML that represents the actual calendar, as opposed to the smaller version that we created for insertion.

Deleting a Calendar

Deleting a Calendar is very easy. You simply need to call the CalendarService classes DeleteCalendarEntry function. The DeleteCalendarEntry function takes three parameters which are documented in the source code:

1. edit_uri – The edit uri (Uniform Resource Identifier) of the Calendar that you want to delete.
2. url_params – Defaults to None. A dictionary containing URL parameters that will be included in the delete.
3. escape_params – Defaults to True. A boolean that controls whether or not the url_params will be escaped.

Note: There is another optional parameter: extra_headers, which is the second parameter, but at this point it does not seem to be used at all in the source code.

The simplest case is to ignore the optional parameters and simply pass in the edit uri of the calendar that you wish to delete. You can get the edit uri of a calendar by calling the GetEditLink function of the CalendarListEntry instance that represents the calendar that you are going to delete. An example of a function that will delete a calendar can be seen in Listing 4. This function takes a CalendarService instance and a CalendarListEntry instance as parameters.

Listing 4

def delete_calendar(calendar_service, calendar):
    e_link = calendar.GetEditLink()
    if (e_link is not None):
        try:
            calendar_service.DeleteCalendarEntry(e_link.href)
        except gdata.service.RequestError, e:
            print "Error deleting calendar: %s" % (e[0]["reason"])

Listing Events

Now let’s take a look at events. Events are items that are added to a specific calendar. If you want to remind yourself to pay your bills at the end of the month you might add that to your banking calendar. That “note” on your calendar is an event.

You can get a list of events for the primary calendar using the CalendarService classes GetCalendarEventFeed function. Since you may be working with more than one calendar, it’s probably more useful to be able to list the events for a specific calendar. You can do this in one of two ways:

1. You can pass in the optional uri parameter to the GetCalendarEventFeed function. From testing I found that a Calendar’s “alternate” link works.
2. You can use the CalendarService classes CalendarQuery function to query for a specific calendars event feed.

If you want to use the first option you can use the CalendarListEntry classes GetAlternateLink member function to get the uri, and then pass it to GetCalendarEventFeed:

a_link = calendar.GetAlternateLink()
#Make sure the link is valid
if (a_link is not None):
    event_feed = calendar_service.GetCalendarEventFeed(a_link.href)

If you use the CalendarQuery method, you need to get the calendar username (or id) of the calendar whose events you want to query. It seems strange to me that there appears to be no way to get a calendar’s username besides parsing one of the calendar’s links. The username of a calendar can be found in the alternate link after “feeds” and before the visibility and projection:

http://www.google.com/calendar/feeds/< <username>>/private/full

Note: You can use the username “default” to query the default calendar.

For the sake of simplicity I will use the alternate link method for my examples. A full example that prints out calendar data and a calendar’s events can be found in Listing 5. The method that prints out the event data is called print_event_feed. It is called near the end of the list_own_calendars method:

# Now Print out the events
print "Events:"
a_link = cal.GetAlternateLink()
if (a_link is not None):
    event_feed = calendar_service.GetCalendarEventFeed(a_link.href)
    print_event_feed(event_feed)

Listing 5

#! /usr/bin/env python

import gdata.calendar.service
import getpass

def print_event_feed(event_feed):

    for index, event in enumerate(event_feed.entry):
        print "\t%d) %s\r\n\tContent: %s" % (
                index, event.title.text, event.content.text)
        print "\t\tWho:"
        for person in event.who:
            print "\t\t\tName: %s\n\t\t\temail: %s" % (person.name
                , person.email)
        print "\t\tAuthors:"
        for author in event.author:
            print "\t\t\t%s" % (author.name.text)
        print "\t\tWhen:"
        for e_index, e_time in enumerate(event.when):
            print "\t\t\t%d) Start time: %s\n\t\t\tEnd time: %s" % (
                e_index
                , e_time.start_time
                , e_time.end_time)

def list_own_calendars(calendar_service):

    try:
        #Get the CalendarListFeed
        all_calendars_feed = calendar_service.GetOwnCalendarsFeed()
    except Exception, e:
        print "Error getting all calendar feed: %s" % (e)
        return
    #Print the feed's title
    print all_calendars_feed.title.text
    #Now loop through all of the CalendarListEntry items.
    for (index, cal) in enumerate(all_calendars_feed.entry):
        #Print out the title and the summary if there is one
        if (cal.summary is not None):
            print "%d) %s - Summary: %s" % (
                index, cal.title.text, cal.summary.text)
        else:
            print "%d) %s" % (index, cal.title.text)
        #Print out the authors
        print "\tAuthors:"
        for author in cal.author:
            print "\t\t%s" % (author.name.text)
        #Print out other information
        print "\tPublished: %s \n\tUpdated: %s \n\ttimezone: %s" % (
            cal.published.text, cal.updated.text, cal.timezone.value)
        print "\tColour: %s \n\tHidden: %s \n\tSelected: %s" % (
            cal.color.value, cal.hidden.value, cal.selected.value)
        print "\tAccess Level: %s" % (cal.access_level.value)
        # Now Print out the events
        print "\tEvents:"
        a_link = cal.GetAlternateLink()
        if (a_link is not None):
            event_feed = calendar_service.GetCalendarEventFeed(a_link.href)
            print_event_feed(event_feed)

def main():
    username = raw_input("Enter your username: ")
    password = getpass.getpass("Enter your password: ")

    calendar_service = gdata.calendar.service.CalendarService(username
        , password
        , "PythonMagazine-Calendar_Example-1.0")
    try:
        calendar_service.ProgrammaticLogin()
    except gdata.service.BadAuthentication, e:
        print "Authentication error logging in: %s" % e
        return
    except Exception, e:
        print "Error Logging in: %s" % e
        return

    list_own_calendars(calendar_service)

if __name__ == "__main__":
    main()

An example of the output produced by Listing 5 can be seen in Listing 6. Don’t mind the formatting – it’s not pretty. It’s merely meant to give you an example of some of the data that you can mine from an event feed.

Listing 6

Mark Mruss's Calendar List
0) Mark Mruss - Summary: Main Calendar
	Authors:
		Mark Mruss
	Published: 2008-02-12T02:30:18.731Z
	Updated: 2008-02-08T04:03:10.000Z
	timezone: America/Toronto
	Colour: #5229A3
	Hidden: false
	Selected: true
	Access Level: owner
	Events:
	0) Dinner at the Drake
	Content: Dinner
		Who:
			Name: Mark Mruss
			email: mark.mruss@gmail.com
		Authors:
			Mark Mruss
		When:
			0) Start time: 2008-02-15T21:00:00.000-05:00
			End time: 2008-02-15T22:30:00.000-05:00
	1) Cezanne's Closet
	Content: Cezanne's Closet
		Who:
			Name: Mark Mruss
			email: mark.mruss@gmail.com
		Authors:
			Mark Mruss
		When:
			0) Start time: 2008-02-09
			End time: 2008-02-11

Adding an Event

Adding a new event to a calendar is very similar to creating a new calendar. You need to create a new CalendarEventEntry instance. You then populate it with options, and pass it to the CalendarService classes InsertEvent function, along with the alternate link of the calendar to which you would like to add the event.

If you take a look at Listing 7 you can see a simple example of how to add an all day event. If you want to specify an event that lasts for less then a day, or include a specific start and end time, you need to use the “RFC 3339 timestamp” format for your start_time and end_time values. Also notice that just like adding a Calendar, the newly created event is returned by the InsertEvent function.

Listing 7

a_link = calendar.GetAlternateLink()
if (a_link is not None):

    new_event = gdata.calendar.CalendarEventEntry()
    new_event.title = gdata.atom.Title(text="New Article")
    new_event.content = gdata.atom.Content(text="Write Article")
    new_event.when.append(gdata.calendar.When(
        start_time="2008-02-20"
        , end_time="2008-02-21"))
    try:
        created_event = calendar_service.InsertEvent(new_event
            , a_link.href)
    except gdata.service.RequestError, e:
        print "Error adding event: %s" % (e[0]["reason"])

Deleting an Event

Deleting an event is (you guessed it) almost identical to deleting an entire calendar. This is because both helper functions wrap the same GDataService base class function. However the 'DeleteEvent event function does a little bit more work than the DeleteCalendarEntry function by making sure that your edit_uri is in the correct format.

To delete an event you simply need to call the DeleteEvent function of the CalendarService class. An example of a function that will delete an event can be seen in Listing 8. This function takes a CalendarService instance and a CalendarEventEntry instance as parameters. As with the majority of these actions, the CalendarService instance should already be authenticated.

Listing 8

def delete_event(calendar_service, event):
    e_link = event.GetEditLink()
    if (e_link is not None):
        try:
            calendar_service.DeleteEvent(e_link.href)
        except gdata.service.RequestError, e:
            print "Error deleting event: %s" % (e[0]["reason"])

Conclusion

Hopefully this article has given you a small taste of what is possible with the gdata-python-client module and Google Calendars. Maybe you can already imagine a use for these features in your next Python application? Please keep in mind that there is much more that can be done with Google Calendars, including updating existing Calendars and Events (CalendarService member functions: UpdateCalendar and UpdateEvent), and almost any other task you can perform using the “online” version of the Google Calendar service.

Also, it is important to note that the Python documentation for the Google Calendar service is sparse but growing. A lot of what is contained in this article was found through browsing the examples, the gdata-python-client source code, other people’s examples, as well as a whole bunch of trial and error on my part. Since this is the case, there may be other ways, or more preferred methods of accomplishing what I have shown here.

[1] http://code.google.com/p/gdata-python-client/
[2] http://code.google.com/apis/gdata/index.html
[3] http://code.google.com/p/gdata-python-client/
[4] http://code.google.com/apis/calendar/developers_guide_python.html
[5] http://code.google.com/apis/calendar/reference.html
[6] http://www.google.com/calendar
[7] gdata.calendar.service.py (From the source comments)
[8] gdata.service.py (From the source comments)
[9] http://code.google.com/apis/accounts/AuthForInstalledApps.html
[10] http://code.google.com/apis/calendar/developers_guide_python.html

By: Mark Mruss

Note: This article was first published the February 2008 issue of Python Magazine

Of all the tasks assigned to programmers, commenting code and writing documentation are among the most disliked. This article introduces you to Python’s documentation strings. While they won’t make commenting your code any more enjoyable, they will provide a systematic approach to doing it, as well as access to additional tools for documentation generation and testing.

You’ve just finished your new Python module. You can’t wait to upload it to the Web and let all the other Python hackers start using it. The only step left is the most dreaded for many programmers: documentation. Unless you’ve been commenting and documenting your code as you wrote it, you’re going to have to go back through each source file, class, and function, trying to remember exactly what your code was supposed to do. Not an enjoyable task.

Sound familiar to you? If it does, you’re probably not using documentation strings, commonly known as doc strings or docstrings (I prefer docstrings, without the space), or any of the handy tools that work with docstrings. This article will introduce you to Python’s docstrings, and a few of the tools that make them a great addition to your code.

Docstrings

If you are not already using docstrings in your Python code, you really should. They provide a standard way to comment your code, giving you and other developers (who might want to use your code at some point) easy access to descriptions of the modules, classes, and functions found within.

At the heart of it, docstrings are simply comments placed in special locations in your Python source code. These comments can then be looked at by tools designed to work with docstrings or other Python programmers using your code. Note that I said using// your code, not reading your code. This is because docstrings are accessible via a Python object’s __doc__ attribute. This is very helpful is you are testing out a new module in Python’s interactive shell and really need to know what sort of parameters a certain function needs.

Pep 257 has a good definition of docstrings: “A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. Such a docstring becomes the __doc__ special attribute of that object.”[1] Definitions are nice but it might be easier to look at a quick example of some docstrings:

def add(x, y):
    """This is the add function's docstring."""
	return x + y

def subtract(x, y):
    """This is the subtract function's docstring.
    It is longer then the add functions, it goes
    on and on and on and on."""
    return x - y

Since docstrings are comments they can be in any format that you want, however since this is Python there are a few style guidelines that you should probably be aware of. You don’t have to follow these guidlines but if you do it will be easier for other Python programmers to understand and work with your code.

In general there are two types of docstrings: one-line docstrings and multi-line docstrings. The difference between the two should be fairly obvious: one-line docstrings are only one line in length and multi-line docstrings are more then one line in length. One-line docstrings and multi-line docstrings each have different style guidelines that will be explained in more detail in the following two sections.

One-Line Docstrings

Let’s look at a quick example of a one-line docstring for a simple function:

def add(x, y):
    """Return the sum of two numbers."""
    return x + y

In this example """Return the sum of two numbers.""" is the docstring of the add function. If you were to run the following:

print add.__doc__

The output would look like this:

Return the sum of two numbers.

In his “Python Style Guide” Guido van Rossum has a few notes on the preferred style of one line docstrings:

• Triple quotes are used even though the string fits on one line. This makes it easy to later expand it.
• The closing quotes are on the same line as the opening quotes. This looks better for one-liners.
• There’s no blank line either before or after the doc string.
• The doc string is a phrase ending in a period. It prescribes the function’s effect as a command (”Do this”, “Return that”), not as a description: e.g. don’t write “Returns the pathname …” [2]

One-line docstrings should only be used to document the simplest of cases. If what you are documenting does anything complicated, accepts input, or returns a value, it’s probably a good idea to use a multi-line docstring.

Multi-Line Docstrings

Multi-line docstrings should be used to document the majority of your modules, classes, functions, and methods. This is because most of what you are programming performs tasks more complicated then that which can fit into a single sentence summary. Multi-line docstrings should be used to documents the input, output, and complex behaviour of your objects. Like one-line docstrings, multi-line docstrings should start with a single sentence summary. After that there should be a blank line and then a more detailed description. The blank line separating the one line summary and the additional information is important as certain tools will use the blank line to separate the summary from the rest of the docstring. An example of a multi-line comment can be found in Listing 1.

Listing 1

def subtract(x, y):
    """Return the difference between two numbers.

    Arguments:
    x -- The minuend.
    y -- The subtrahend.

    Returns:
    A number, the difference between x and y (ie. x - y)

    """
    return x - y

What to Document

There are also style guidelines that dictate what you should include in your multi-line docstrings when documenting different sections of your code. These are general guidelines but following them will help ensure that your code is well documented and easily understood by other Python programmers. For more information on this please see Guido’s “Python Style Guide”.[2]

Modules – Document the module and provide a one line summary of everything that is exported by the module. (eg. classes, exceptions, and functions).

Classes – Summarize the functionality of the class. List all public methods and data members of the class. If there is any addition information needed to subclass the class, or if there is an additional interface for subclasses provide a description.

Functions and Methods – Summarize the functionality of the function and “document its arguments, return value(s), side effects, exceptions raised, and restrictions on when it can be called (all if applicable).”[3]

An example of all of these can be found in Listing 2.

Listing 2

#!/usr/bin/env python
"""A simple math module.

Exported Classes:

Math -- A simple math class with mathematical functions.

"""

class Math(object):
    """A simple math class with mathematical functions.

    Public functions:
    add -- Adds two numbers together and
    returns the result.

    subtract -- Returns the difference between
    two numbers.

    """

    def subtract(self, x, y):
        """Return the difference between two numbers.

        Arguments:
        x -- The minuend.
        y -- The subtrahend.

        Returns:
        A number, the difference between x and y (ie. x - y)

        """
        return x - y

    def add(self, x, y):
        """Return the sum of two numbers.

        Arguments:
        x -- Number to be summed.
        y -- Number to be summed.

        Returns:
        A number, the sum of x and y (ie. x + y)

        """
        return x + y

Documentation Generation

Generally if you have a complicated module or API people would rather read a help file, or online documentation, as opposed to constantly having find and browse the source code. Thankfully for us there are many different documentation generation tools out there that make use of docstrings. So when you are writing your docstrings, you aren’t just commenting your source code you’re also writing your help file!

The easiest tool to use is PyDoc, it’s a module, and a stand-alone application, that has been included in the Python standard library since version 2.1. There is much that can be done with PyDoc but for this column we are going to focus on it’s documentation generation. PyDoc can take the docstrings found within a module and output them as either simple text documentation (much like UNIX or Linux man pages) or HTML documentation.

Creating HTML documentation of the SimpleMath module used in Listing 2 is very easy. On a UNIX like computer (Linux, OS X, etc.) it can be accomplished using the following command:

$ pydoc -w /home/selsine/python/SimpleMath.py

On Windows the command will look something like this:

C:>c:Python25Libpydoc.py -w c:pythonSimpleMath.py

This will create an HTML file named SimpleMath.html in the current folder. A sample of what the generated HTML looks like can be seen in Figure 1.

Figure 1 - pydoc

While PyDoc is a great tool and easy to use because it is in the standard library, there are a few other tools out there that you might consider using. If you look at the HTML file that PyDoc generates you will notice that it does not mark up your docstrings. It simply reads them from your source code and spits them back out. If you are looking for something a little bit fancier with a few more options you might consider Epydoc[4] or docutils[5].

Both Epydoc and docutils use simple markup languages to give the documentation generated a bit more punch. Docutils uses the reStructuredText markup language. While Epydoc uses the Epytext markup language, as well as being able to work with Javadoc and reStructuredText. An example of the HTML that Epydoc produces can be seen in Figure 2. The code that was used to generate the HTML can be found in Listing 3.

Figure 2 – Epydoc

Listing 3

#!/usr/bin/env python
"""A simple math module.

Exported Classes:

Math -- A simple math class with mathematical functions.

"""

class Math(object):
    """A simple math class with mathematical functions.

    Public functions:
    add -- Adds two numbers together and
    returns the result.

    subtract -- Returns the difference between
    two numbers.

    """

    def subtract(self, x, y):
        """Return the difference between two numbers.

        @type   x: number
        @param  x: The minuend.
        @type   y: number
        @param  y: The subtrahend.

        @rtype: number
        @returns: A number, the difference between x and y (ie. x - y)

        """
        return x - y

    def add(self, x, y):
        """Return the sum of two numbers.

        @type   x: number
        @param  x: Number to be summed.
        @type   y: number
        @param  y: Number to be summed.

        @rtype: number
        @returns: A number, the sum of x and y (ie. x + y)

        """
        return x + y

def _test():
    import doctest
    doctest.testmod()

if __name__ == "__main__":
    _test()

Doctest

One of the most interesting uses of docstrings is unit testing using the doctest module. As we all know testing our code is important, and (as many of us have begun to learn) writing unit tests for large projects is a great way to ensure that changes in one area of a project don’t cause problems elsewhere in the code. If you don’t know what a unit test is it’s basically a simple test case to prove whether a specific area of your code is functioning properly. Generally at least one unit test is created for each object in order to test the correctness of the project as a whole.

In a nutshell the the doctest module searches your dostrings "for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown."[6] This means that it will search your docstrings for lines that start with >>> or with ... if they are the continuation of a statement (i.e. the inside of a function). If there is output generated by the statements it “must immediately follow the final ‘>>> ‘ or ‘… ‘ line containing the code, and … extends to the next ‘>>> ‘ or all-whitespace line.”[7] Adding doctests doesn’t just add unit tests to your code it also provides working examples in your docstrings and documentation.

Since doctests are formatted to look like interactive Python sessions a simple way to write them is to use Python’s interactive shell. You do this by simply executing your code in the interactive shell, and then copying and pasting the resulting test into your docstrings. For example, if we were to use this method to write a doctest for our subtract method we could do something like the following in the interactive shell:

>>> from SimpleMath import Math
>>> simple_math = Math()
>>> simple_math.subtract(10, 7)
3
>>>

We don’t need to import our module for the doctest so all we will need to copy from the interactive shell are the middle three lines:

>>> simple_math = Math()
>>> simple_math.subtract(10, 7)
3

As you can see, what we have here is a test compromising of two lines of Python code and then the expected result. The SimpleMath module containing a doctest for each function can be found in Listing 4.

Listing 4

#!/usr/bin/env python
"""A simple math module.

Exported Classes:

Math -- A simple math class with mathematical functions.

"""

class Math(object):
    """A simple math class with mathematical functions.

    Public functions:
    add -- Adds two numbers together and
    returns the result.

    subtract -- Returns the difference between
    two numbers.

    """

    def subtract(self, x, y):
        """Return the difference between two numbers.

        >>> simple_math = Math()
        >>> simple_math.subtract(10, 7)
        3

        @type   x: number
        @param  x: The minuend.
        @type   y: number
        @param  y: The subtrahend.

        @rtype: number
        @returns: A number, the difference between x and y (ie. x - y)

        """
        return x - y

    def add(self, x, y):
        """Return the sum of two numbers.

        >>> simple_math = Math()
        >>> simple_math.add(10, 7)
        17

        @type   x: number
        @param  x: Number to be summed.
        @type   y: number
        @param  y: Number to be summed.

        @rtype: number
        @returns: A number, the sum of x and y (ie. x + y)

        """
        return x + y

def _test():
    import doctest
    doctest.testmod()

if __name__ == "__main__":
    _test()

If you look at Listing 4 you will also notice the following code at the end of the source:

def _test():
    import doctest
    doctest.testmod()

if __name__ == "__main__":
    _test()

This is the code that will be executed if the module is launched directly from the command line. The code will import the doctest module and then use the testmod method to test the current module. Now when we run our SimpleMath.py file directly we will get the following:

$ python SimpleMath.py
$

Since nothing was written out to the command line we know that all of the doctests were successful. If you want to get more information you can use the -v flag to produce verbose output:

$ python SimpleMath.py -v

If you were to encounter an error it would look something like Listing 5.

Listing 5

**********************************************************************
File "SimpleMath.py", line 26, in __main__.Math.subtract
Failed example:
    simple_math.subtract(10, 7)
Expected:
    4
Got:
    3
**********************************************************************
1 items had failures:
   1 of   2 in __main__.Math.subtract
***Test Failed*** 1 failures.

As an added bonus if you are using Epydoc or docutils to generate your documentation, both tools will recognize doctest sections and highlight them. An example of what it looks like when Epydoc does this can be seen in Figure 3.

Figure 3 - doctest

Conclusion

Hopefully by this point you can see how useful docstrings can be to any Python code that you write. They give you a structured way to document your source code, the ability to easily generate great looking documentation, and a simple way to add unit tests to your code. But that's not all, as more and more people start using doctrings you can be sure that the list of docstring tools will continue to grow.

We all know that commenting source code and writing documentation are among the least enjoyable tasks a programmer can face. In fact the only thing worse then commenting and documenting is trying to use code with no comments and poor documentation! So do me, and yourself, a favour: start writing those docstrings.

[1] http://www.python.org/dev/peps/pep-0257/#what-is-a-docstring
[2] http://www.python.org/doc/essays/styleguide.html
[3] http://www.python.org/doc/essays/styleguide.html
[4] http://epydoc.sourceforge.net/
[5] http://docutils.sourceforge.net/
[6] http://docs.python.org/lib/module-doctest.html
[7] http://docs.python.org/lib/doctest-finding-examples.html

By: Mark Mruss

Note: This article was first published the January 2008 issue of Python Magazine

Iterators, iterables, and generators are features handled so wall by Python that people programming in other languages cannot help but drool over. Fortunately for us, creating iterators, iterables and generators is a relatively simple task. This article introduces the concepts of iterators, iterables, and generators and illustrates how easy it is to add them to your code.

1. Introduction
2. Iteration in Python
3. An Initial Example
4. Creating An Iterator
5. Looking More Closely At The Iterator
6. The Upside And Downside Of Iterators
7. Generators
8. Looking Closely At The Generator
9. But What About Iterables?
10. Creating An Iterable Object
11. Conclusion

Introduction

In this article I’m going to introduce three related Python features: iterators, iterables, and generators. Generators are easy to define, they are functions that create and return an iterator. Iterators and iterables on the other hand, are easier to use than they are to define. An iterable object is a “container object capable of returning its members one at a time.”[1] An iterator object is “An object representing a stream of data. Repeated calls to the iterator’s next() method return successive items in the stream. When no more data is available a StopIteration exception is raised instead. At this point, the iterator object is exhausted and any further calls to its next() method just raise StopIteration again.”[1] You can think of the difference between the two in this way: an iterable object can be iterated over multiple times, whereas an iterator object can only be iterated over once. In general an iterable produces an iterator every time something wants to iterate over its data.

Note: Classes that define the __getitem__ function are also considered iterables, but since that falls outside the scope of this article, it will not be covered here.

In this tutorial, I will begin by discussing iterators, the most basic concept. Then I will move onto generators, and finish by discussing iterables, the most wide open topic of the three.

Iteration in Python

Iterators objects are used in Python in order to iterate over an objects data. For example, we all know how to do this in Python when we work with lists:

my_list = [1,2,3]
for num in my_list:
	print num

This code will iterate over the list object my_list and print out all of the list items , i.e., the numbers 1, 2, and 3. Iterating over sequences in this simple and transparent manner happens to be one of my favourite features of Python.

According to our definition above, lists are iterables since you can iterate over them multiple times. In fact, each time you iterate over a list you are actually using a listiterator iterator object produced by the list.

It may not be immediately clear to you when you should add an iteration support to a class. However, the more you work with Python the more you’ll find instances when doing just this is very useful, sometimes the only advantage is cleaner looking code. One nice thing about iterators (and generators too) is that the processing for each item happens as you need it. Instead of collecting all of the data into a list and then running through the list, you will collect each item as you need it. This might not seem like a large difference but imagine if there were tens of thousands of items to process? What if you were collection your data from an online source? Performing all of your processing up front may take a very long time, especially if you only wanted the first few items.

An initial example

In order to explain iteration further, let’s look at a simple example task where we might use iterators. For this example we will create a class that takes a string of characters as input and then converts each character into its byte value. If we were NOT going to use iterators we might do something like what is found in Listing 1.

Listing 1

class ByteValue(object):

	def __init__(self, data):
		self.data = data

	def to_bytes(self):
		bytes = []
		for char in self.data:
			bytes.append(ord(char))
		return bytes

This code is pretty simple. We have a data member, named data, that we use to store the string that was used to initialize the class. In the to_bytes function we loop through the string, converting each character to its byte value using the built in ord function. We store each byte value in a list and once we have collected all of the values we return that list.

When we run the following:

bv = ByteValue("abcdef")
for byte in bv.to_bytes():
	print byte

we would get this as our output:

97
98
99
100
101
102

Creating an Iterator

Let’s convert this into an iterator. Making your class into an iterator requires adding “two methods, which together form the iterator protocol.”[2] The two functions needed are: 1) the __iter__ function; and, 2) the next function. The __iter__ function will return the object itself, while the next function will return the next item. The next function is where the actual iteration work occurs. The next function iterates by returning the next item in the “sequence” each time it is called for as long as there is a “next” item. When there are no more items to iterate over, the next function must raise the StopIteration exception to halt the iteration.

To be clear, in order to make your class an iterator you need to do two things:

1) Add an __iter__ function that returns the object itself (self)

2) Add a next function that returns the next item in the sequence each time it is called. When there are no more items in the sequence, the next function raises a StopIteration exception signal the end of the iteration.

For those of you still confused, the following example will help illustrate how iterators work. If we were to convert our ByteValue class into an iterator object, it might look something like Listing 2.

Listing 2

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0

	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value

Let’s compare the code in Listings 1 and Listing 2 in detail, focusing on the iterator in Listing 2. The first difference is the addition of the data member current_item to the class, initialized in the __init__ function. current_item serves as a counter and keeps track of the current character in the string while we iterate over it. The counter must have class scope since the iterator works through successive calls to the next function. If current_item were local to the next function, its value would be reset with each subsequent call, and would not be of much use.

The second difference between the listings is the addition in Listing 2, of the __iter__ function where we return self.

The final addition to Listing 2 is the next function, where we first check to see if current_item is equal to the length of our string. If current_item is equal to the strings length we raise the StopIteration exception to signal the end of the iteration because we have no more characters left to iterate over. If there are more characters to iterate over we calculate the byte value of the current character, increase our current_item counter, and then return the byte value.

Note: Notice that current_item is only initialized when the ByteValue object is created. This happens because according to the iteration protocol, “once an iterator’s next() method raises StopIteration, it will continue to do so on subsequent calls.”[2] If we were to re-initialize current_item we would then be able to iterate over the iterator more than once breaking the iteration protocol.

Now that we have converted our class into an iterator we can use it as follows:

for byte in ByteValue("abcdef"):
	print byte

Doing so would result in:

97
98
99
100
101
102

Notice that we do not store the instance of our ByteValue class in a variable. Doing so would be useless because since ByteValue is an iterator it is only good for one pass of the data. If ByteValue were an iterable (returning an iterator object when __iter__ was called) it would make sense to keep an instance around because we could iterate over the instance more than once. We will look at creating iterables later on in this article.

Looking more closely at the Iterator

Let’s look at what is happening in more detail by examining what is happening behind the scenes during the iteration process. In order to illustrate what is happening in the for loop I will demonstrate the order in which things are being called behind the scenes.

The first step in the iteration process is to call to the __iter__ function in order to get the iterator object that will perform the iteration. Notice that this works on iterator and iterable objects, since iterators returns themselves and iterables return an iterator.

bv = ByteValue("abcdef")
iterator = bv.__iter__()

Now that we have the iterator object, we start iterating by calling the next function in order to get the next value:

print iterator.next()

This executes the next function which will return the byte value of character in the string with which we are currently working. Since this is the first call to the next function current_item will be zero and we will calculate the byte value of the first character in our string (’a') resulting in:

97

If we continue the iteration process by calling the next function six more times we would get the results shown in Listing 3. Notice that we have now made a total of seven calls to the next function, one more then the number of characters in our string. I’m running the python code from a file (iter.py found in Listing 4). Depending on how you are running it, you might get slightly different results. However, the most important thing to observe in this example is the exception that is raised.

Listing 3

97
98
99
100
101
102
Traceback (most recent call last):
  File "Listing4.py", line 34, in ?
    main()
  File "Listing4.py", line 30, in main
    print iterator.next()
  File "Listing4.py", line 13, in next
    raise StopIteration
StopIteration

Listing 4

#!/usr/bin/env python

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0
	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value
		return self.data

def main():
    for v in ByteValue("abc"):
        if v in ByteValue("abc"):
            print "We have a %d" % v

    bv = ByteValue("abcdef")
    iterator = bv.__iter__()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()

if __name__ == "__main__":
	# Someone is launching this directly
	main()

For the most part you won’t be calling an iterator object’s __iter__ or next functions manually, instead you’ll probably just let the for loop do it all for you.

The upside and downside of Iterators

Now that our class is an iterator object we can use any of the built-in functions and methods that work on iterators and iterables, such as: the sum, tuple, sorted, and list functions, to name a few.

In the first example the bytes function returned a list object that we could work with. If we want a list instead of an iterator for any reason, it will be as simple as using the list function, which takes an iterator as a parameter and returns a list:

list(ByteValue("abcdef"))

If we want a sorted list:

sorted(ByteValue("abcdef"))

If we want the sum of all of the bytes:

sum(ByteValue("abcdef"))

While very useful, iterators do have some downsides to them. The most obvious is that they are only good for one pass over the data. They also generally require you to add extra data members to your class in order to keep track of your current iterator position. Depending on what you are iterating over, this process can become quite complex. Iterators also only allow you to perform one “type” of iteration, i.e., in one direction or over one piece of internal data. You might address this by adding flags to your class but this will further clutter the class and decrease readability. So how can you simply add multiple types of iteration to a class? Enter the generator!

Generators

A generator is a function that creates, or generates, an iterator. In order for a function to become a generator it must return a value using the yield keyword.

Generators are interesting because they are functions, yet execution does not run through them as it does in a normal function. The first time execution enters a generator function it will start at the beginning of the function and continue until the yield keyword is encountered. When the iteration continues, execution will continue in the generator function on the statement immediately following the yield keyword. All local variables in the function will remain intact. If the yield statement occurs within a loop, execution will continue within the loop as though execution had not been interrupted.

Continuing with the ByteValue example, let’s add a generator function named reverse that can be used to iterate through the byte vales of the string in reverse order:

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		yield ord(self.data[current_item])

So what’s going on here? The first thing to notice is that we did not add anymore data members to our class, this generator is a self-contained unit. Secondly, since this is a generator, our counter current_item can be a local variable.

In the reverse function the first step is to initialize current_item, which represents the current character in the string, to be equal to the length our string. We initialize it to the length of the string instead of zero since we are iterating through the string in reverse. Next we have a while loop that loops while current_item is greater than zero. We then subtract one from our counter, to give us the current character to process. Finally, we yield the byte value of the current character.

Note: We subtract one from our counter the first time through the loop because Python is zero-based and the length of a list minus one gives us the position of the last item in the list. In our examples we have used the following string:

abcdef
012345

When we calculate the length of the string we get 6. We then subtract 1 from that number, leaving 5, which is the index of the last number in the string.

Making use of our new generator function, we run the following:

bv = ByteValue("abcdef")
for byte in bv.reverse():
	print byte

We get our favourite byte values in reverse:

102
101
100
99
98
97

Looking closely at the Generator

Let’s take a detailed look at what is happening in the generator in the same way that we did earlier with the iterator object. The first thing that happens when you call a generator function is NOT the execution of the actual function, rather, it is the creation of a generator object. Running the following:

bv = ByteValue("abcdef")
gen = bv.reverse()
print gen

will result in:

<generator object at 0xb7d8e04c>

This demonstrates that, as stated above, the first call to our generator function does not return the byte value of that last character in the string, instead it creates a generator object. In fact the first time a generator function is called, the actual function is not executed at all. A generator object is “what Python uses to implement generator iterators. They are normally created by iterating over a function that yields values”.[3]

Once we have a generator object we can start calling its next function (sound familiar?) to perform the action iteration. An example of this can be found in Listing 5. The results of this execution can be found in Listing 6. The results will seem very familiar to you, especially the StopIteration exception.

Listing 5

#!/usr/bin/env python

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0
	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value
		return self.data

	def reverse(self):
		current_item = len(self.data)
		while (current_item > 0):
			current_item -= 1
			yield ord(self.data[current_item])

def main():

	bv = ByteValue("abcdef")
	gen = bv.reverse()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()

if __name__ == "__main__":
	# Someone is launching this directly
	main()

Listing 6

102
101
100
99
98
97
Traceback (most recent call last):
  File "Listing5.py", line 40, in ?
    main()
  File "Listing5.py", line 36, in main
    print gen.next()
StopIteration

If we want to look at the contents of the generator object we could use the following code:

print dir(gen)

And getting the following results:

['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'gi_frame', 'gi_running', 'next']

Notice that the generator object contains the __iter__ and next functions making the generator object itself an iterator. Because the generator itself is an iterator object, the same built-in iterator functions I mentioned earlier can also be used on our generators:

print sum(bv.reverse())

It is also important to remember that your generator or iterator does not have to perform only “dumb” iteratation, simply moving through some sort of an internal list. Rather, it can make decisions just like any other block of code.

For example, let’s say that in our reverse generator we want to use the byte value 99 as an end condition. We can do something similar to the example found in Listing 7.

Listing 7

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		value = ord(self.data[current_item])
		if (value == 99):
			return
		yield value

In Listing 7 if the byte value equals 99 we return from our generator function using the return keyword. Since we didn’t yield anything this will cause the StopIteration exception to be fired halting the iteration.

Be careful about getting too smart with your iteration because you cannot return any information from a generator you can only yield it. So if you tried to execute the code in Listing 8, in an attempt to return one last value before quitting (or if you wanted to return a success or failure code) you will get the following error when the return value line of code is executed:

SyntaxError: 'return' with argument inside generator

Listing 8

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		value = ord(self.data[current_item])
		if (value == 99):
			return value
		yield value

But what about iterables?

I’m sure that by now you are wondering about the iterable objects that I mentioned at the start of this tutorial. As you have probably guessed, simply making your class an iterator isn’t that useful unless it is only performing one task like our ByteValue example. Since your classes will generally be performing more than one task, you will likely want to make your class an iterable object rather than an iterator object. To recap, iterable objects return an iterator object when their __iter__ function is called, which allows for multiple passes over their data.

Creating an Iterable object

Since the definition of an iterable object is an object that returns an iterator object when its __iter__ function is called, creating an iterable object can be done in a variety of ways. Two options that come to mind are: 1)creating an iterator helper class to perform the iteration and, 2) using a generator function. I prefer using a generator function since it keeps the functionality within the main class.

See Listing 9 for an example of creating an iterarable object using a generator. You will see that we have replaced the next function with the forward function. The forward function is a generator that iterates through the data in the “forward” direction. In the __iter__ function we return the results of a call to the forward function, a generator object. Since generator objects contain the iterator protocol and are, in fact iterators, by returning one from our __init__ function we have successfully created an iterable.

Listing 9

class ByteValue(object):

	def __init__(self, data):
		self.data = data

	def __iter__(self):
		#We are an iterable, so return our iterator
		return self.forward()

	def forward(self):
		#The forward generator
		current_item = 0
		while (current_item < len(self.data)):
			byte_value = ord(self.data[current_item])
			current_item += 1
			yield byte_value

	def reverse(self):
		#The reverse generator
		current_item = len(self.data)
		while (current_item > 0):
			current_item -= 1
			yield ord(self.data[current_item])

def main():
    bv = ByteValue("abc")
    for v in bv:
        if v in bv:
            print "We have a %d" % v

if __name__ == "__main__":
    main()

Now that we have an iterable object, we can iterate over it as many times as we want. We can even have fun with nested iteration:

bv = ByteValue("abcdef")
for value in bv:
	print value
	for second_value in bv:
		print second_value

Conclusion

This concludes our introduction to iterators, iterables, and generators. I hope I have demonstrated the immense power and flexibility that they provide. In general, making your object an iterable object and/or using generators allows for more flexibility than simple iterator objects provide. For most complex classes or complex sets of data, multiple iterations are a given.

With all of the praise that I have heaped upon iterators, iterables, and generators, it’s important to remember that they are not a panacea and should not be used in every case where a sequence of items is needed. There are many instances where a returning a list is the desired result. This being said, iterators, iterables, and generators are extremely useful and provide a great way to loop through data.

[1] http://docs.python.org/tut/node18.html
[2] http://docs.python.org/lib/typeiter.html
[3] http://docs.python.org/api/gen-objects.html

By: Mark Mruss

Note: This article was first published the November 2007 issue of Python Magazine

While the equality operator works great on numbers and strings the fact the way it treats your custom objects really is not that useful. This article looks into overloading the equality operator so that you can easily compare your custom classes.

1. Introduction
2. Introducing the terms: operators and operator overloading
3. A Quick Example of the Default Equality Operator
4. Overloading the Equality Operator
5. Telling Python that the Comparison has Not Been Implemented
6. The Inequality Operator
7. Dangers
8. Conclusion

Introduction

In my experience as a professional programmer, testing for the equality between two instances of a class is a fairly common task. In other words, you are comparing the data that each class contains and checking whether the data in one class is identical to the data in the other class.

One of the nice features of Python is that it has a default equality operator defined for any custom objects that you create. The unfortunate thing about this default equality operator is that it doesn’t provide the functionality that you expect. This is because the equality operator (==) actually performs an identity comparison, rather than an equivalence test. If you were to run the following code:

if (object_one == object_two):

By default Python actually compares whether or not object_one is object_two (this is the same comparison that can be made using the is keyword) instead of determining whether or not object_one is equivalent to object_two. Fortunately for us, overloading the default equality operator in Python is a relatively easy task. There are, however, some “gotchas” and other interesting features of which one should be aware.

Introducing the terms: operators and operator overloading

An operator can be difficult to define, and like many programming definitions, sometimes the definition only serves to confuse the matter further. In general though, you can think of operators as being very similar to the operators that you encountered in Math class, such as: the + operator, the – operator, and so forth.

In Python the following are operators[1]:

+	-	*		/	//	%	<>>	&
|	^	~	<>	< =	>=	==	!=	<>

In programming languages we generally encounter binary operators. This means that each operator takes two operands. An operand serves as input to an operator. For example, in the statement:

2 + 6

+ is a binary operator that takes two operands, 2 and 6 as inputs. Similarly, in this statement:

my_value - 6

- is an operator that takes two operands, my_value and 6 as inputs.

Operator overloading is a programming term that means taking the default behaviour of an operator and overloading it. That is, changing the default implementation of an operator for a given object. An example of this (although something that you should never do) would be to overload the + operator to actually perform subtraction instead when it is applied to your class.

A Quick Example of the Default Equality Operator

Now that the definitions are out of the way, let’s look at an example where one might want to overload the equality operator. For this example I will bring back a favourite example from my Computer Science days: the Student class:

class Student(object):

	def __init__(self, name, student_number):
		self.name = name
		self.student_number = student_number

As you can see the Student class has two data members: 1) the student’s name, and, 2) her student number.

If we run the following code:

mark = Student("Mark Mruss", 067213)
guido = Student("Guido van Rossum", 000001)
if (mark == guido):
	print "Equal"
else:
	print "Not Equal"

“Not Equal” will be printed out as you would expect since the two students are clearly not equivalent. But what about this code:

mark = Student("Mark Mruss", 067213)
mark_two = Student("Mark Mruss", 067213)
if (mark == mark_two):
	print "Equal"
else:
	print "Not Equal"

Here, as in the previous example, “Not Equal” will be printed out. This is because, as mentioned earlier, the default implementation of the equality operator is to perform an identity comparison. In other words, the default equality operator asks, is mark the same object as mark_two? In Python the equality comparison depends on the type of objects being compared. For custom classes that you or I will create, the equality comparison will perform an identity comparison by comparing the object’s internal id. In other words, it will only result in True if the objects being compared actually are each other. For example:

student_one = Student("Mark Mruss", 067213)
student_two = student_one
if (student_one == student_two):
	print "Equal"
else:
	print "Not Equal"

Results in “Equal” being printed out, as would:

student_one = Student("Mark Mruss", 067213)
student_two = student_one
if (id(student_one) == id(student_two)):
	print "Equal"
else:
	print "Not Equal"

Note: The equality comparison for built-in objects and types like numbers, strings, lists, tuples, and mappings behave differently. Numbers are compared arithmetically. The numerical values of the characters within strings are compared arithmetically. The comparison of lists and tuples is simply a comparison of their inner values, while the comparison of mappings are comparisons of an ordered list of their values.[2]

Overloading the Equality Operator

Hopefully the above example illustrated a case where we might want to overload the equality operator to make it so that the following code:

student_one = Student("Mark Mruss", 067213)
student_two = Student("Mark Mruss", 067213)
if (student_one == student_two):
	print "Equal"
else:
	print "Not Equal"

Would result in “Equal” being printed out, i.e. a true equality comparison as opposed to an identity comparison. In order to do this we need to change to the default functionality of the equality operator. In other words we need to overload it.

In general, operator overloading in Python means adding a special function to your class that will perform the function of the operator it is meant to represent. There are two ways in which one can overload the equality operator in Python: 1) the first method is to use the __eq__ function, a so-called “rich comparison” function. “Rich comparison” functions are functions that overload specific comparison operators (i.e. __eq__ to overload ==). 2) The second is to use the __cmp__ function, which is used to overload all comparison operators if no “rich comparison” functions are present.

Since __cmp__ is used to override all comparison operators (==, !=, < , <=, >, >=), I would suggest using the “rich comparison” method unless you are using a version of Python that is earlier then version 2.1, or you are convinced that you know what < = means to our Student class. Let’s forget about the __cmp__ operator for now and focus on using the “rich comparison” functions to overload the equality operator.

“Rich comparison” functions can return any value, but you should try to return a value that is, or can be, interpreted as a boolean value. This is important because these functions will often be used in situations where the return value will be used in a boolean comparison.

When using the “rich comparison” functions it is important to know which functions are being called internally. For example, when we run:

student_one == student_two

If __eq__ exists in the Student class, the following is actually being called:

student_one.__eq__(student_two)

When we run:

student_two == student_one

The following is actually called:

student_two.__eq__(student_one)

As you can see it is the operand on the left-hand side whose __eq__ function will be called. It is important to note that if the operand on the left-hand side lacks the __eq__ function while the operand on the right-hand side has one, the right-hand operand’s __eq__ function will not be called.

Lets start off with a simple, but incorrect, example (the reasons for its incorrectness will be explained below):

def __eq__(self, other):
	return ((self.name == other.name)
		and (self.student_number == other.student_number))

This is very straightforward. In the equality comparison, we simply compare the Student class’ two data members. This performs as expected when we run:

student_one = Student("Mark Mruss", 067213)
student_two = Student("Guido van Rossum", 000001)
student_three = Student("Mark Mruss", 000001)
print (student_one == student_two)
print (student_one == student_three)

You get:

False
True

But what happens when we introduce the Professor class and try the overloaded equality operator:

class Professor(object):

	def __init__(self, instructor, course):
		self.instructor = instructor
		self.course = course

As you can see, the Professor class lacks the name and student_number data members. What happens when we compare an instance of the Professor class with an instance of the Student class?

guido = Student("Guido van Rossum", 000001)
rob = Professor("Rob Ward", "74-300")
print (guido == rob)

It results in something like this:

File "operators.py", line 10, in __eq__
    return ((self.name == other.name)
AttributeError: 'Professor' object has no attribute 'name'

The way we are overriding the equality operator is not correct because it automatically assumes that the other object has the name and student_number data members. There are a number of methods to get around this problem, including: 1) using the hasattr function, or 2) using the isinstance function. Using the hasattr function determines if other has the attributes we are looking for before actually querying them. hasattr simply tells us if an object has a specific attribute or not. Here is a quick example illustrating how to do this:

def __eq__(self, other):
	if (hasattr(other, "name") and hasattr(other, "student_number")):
		return ((self.name == other.name)
			and (self.student_number == other.student_number))
	else:
		return False

First, we check to see if other has the name and student_number attributes. If it does, we proceed as normal. If it does not, we simply return false. When we compare the professor and the student we get “False” as expected.

What’s nice about this method is that we don’t have to care what type other is. We only care whether or not it contains the attributes we need to compare. However, the drawback to this function is that you have to test for the existence of each attribute. Although this may not always be a big deal, if you are dealing with fifty data members in your classes this can quickly become a pain in the neck.

Another solution to the problem with our first overloading example is to use the isinstance function to make sure that other is an instance of our class type. This has the drawback of forcing other to be the same type as your class. In practice however, I believe this to be more of an advantage than a disadvantage.

def __eq__(self, other):
	if (isinstance(other, Student)):
		return ((self.name == other.name)
			and (self.student_number == other.student_number))
	else:
		return False

The first thing we do is check the variable other to make sure that it is an instance of the Student class. If it is, we then compare all of the data members in the Student class. If object is not an instance of the Student class, we return False.

In my opinion, this is the preferred method since knowing that the class is the correct type is often important. The hasattr method seems more appropriate for simple data containers like a “rect” or “vector” class where you are only interested in three or four data members.

Telling Python that the Comparison has Not Been Implemented

Up until this point in time we have been returning False when our __eq__ function does not support the type of object passed in as other. While this is acceptable and correct given the Python documentation, it seems to be “proper” to actually return NotImplemented. According to the Python documentation, “Numeric methods and rich comparison methods may return this value if they do not implement the operation for the operands provided. (The interpreter will then try the reflected operation, or some other fallback, depending on the operator.)” [4]Let’s forget abou In other words, if the left operand returns NotImplemented, Python will attempt to use the right hand operand’s equality operator. And if that does not exist, Python will fall back to the default equality operator.

We can return NotImplemted from our Student class if the operand passed in is not an instance of the Student:

def _eq__(self, other):
	if (isinstance(other, Student)):
		return ((self.name == other.name)
			and (self.student_number == other.student_number))
	else:
		return NotImplemented

Now if we perform the following comparison:

guido = Student("Guido van Rossum", 000001)
rob = Professor("Rob Ward", "74-300")
print guido == rob

The first step in the processing will be:

guido.__eq__(rob)

This returns NotImplemented. As a result, the reflected operation is attempted:

rob == guido

Because the Professor class does not have the equality operator overloaded, the default operation is executed and False is printed out just like we wanted.

NotImplemented is useful in because instead of returning False, which means that the two operand are not equivalent, you return a value that says that the comparison between the operands has not been implemented.

The Inequality Operator

Now that we know how to overload the equality operator, it stands to reason that we have the opposite operation, the inequality operator (!=) covered as well. But not so fast. In Python the inequality and equality operators are handled separately, meaning that inequality is not simply the opposite of equality. This means that whenever you overload the equality operator, you have to be sure to overload the inequality operator as well. If you don’t you might get some strange results. For example, when we use the current code (without the inequality operator overloaded), the following:

guido = Student("Guido van Rossum", 000001)
guido_too = Student("Guido van Rossum", 000001)
print guido == guido_too
print guido != guido_too

Results in:

True
True

In the first comparison the overloaded equality operator is used, and results in True being printed. Because the inequality operator is not overloaded in the second comparison, the default inequality operator is used (the identity comparison). True is printed because guido and guido_too are not the same instances.

Thankfully once you have overloaded the equality operator, overloading the inequality operator is very easy. As a general rule, you have to return the opposite of the equality operator, but because we are working with NotImplemented, we have to do a bit more processing to ensure that we don’t return False when we really want to return NotImplemented. Here is how we can overload the inequality operator in the Student class:

def __ne__(self, other):
	equal_result = self.__eq__(other)
	if (equal_result is not NotImplemented):
		return not equal_result
	return NotImplemented

First, we call self.__eq__ to test whether or not we are equal to other. We then check to make sure that equal_result is not NotImplemented. If it is not, we know that the equality test was implemented and we can safely return its’ opposite. If the result for the equality comparison was NotImplemented, we return NotImplemented for the inequality comparison.

Note: It is safe to use the is check on NotImplemented (rather than an isinstance check) because NotImplemented is a singleton, meaning that there is only ever one instance of NotImplemented at anytime.

Dangers

While it may seem like operator overloading should become part of every class that you write, a word of warning is necessary. There is a large school of thought that views operator overloading as a dangerous programming technique. They argue that overloading operators changes the default way that an operator works, and not always correctly. Moreover, instead of overriding the equality operator, one can simply add an is_equal_to function to perform the equality check.

The logic behind this criticism is that when someone is using a class or reading some code that you wrote, they will be unable to tell what the equality operator is doing. For example, if they see:

value = MyClass(10)
value_two = MyClass(10)
print value == value_two

What gets printed out? True or False? If “MyClass” overrode the equality operator then True will be printed. However, if the equality operator is not overloaded, the standard Python behaviour of equality will result with False being printed out.

Conclusion

While it’s true that overloading the equality operator does change the default way the Python functions, I feel that it’s generally a safe and beneficial addition to your classes. Especially since unless people know the ins and outs of the equality operator they will generally assume that should work the way it does when you overload it. Like all the decisions that you make when working with Python, context is key.

[1] http://docs.python.org/ref/operators.html
[2] http://docs.python.org/ref/comparisons.html
[3] http://docs.python.org/ref/customization.html
[4] http://docs.python.org/ref/types.html