Note: This article was first published the May 2008 issue of Python Magazine

Introducing Descriptors and Properties

Mark Mruss

New-style classes were introduced to Python with the release of Python 2.2. And with these new-style classes came descriptors and properties. This article will introduce the descriptor protocol, descriptors, and properties.

Introduction

New-style classes were introduced to Python with the release of Python 2.2. A new-style class is any class that is derived from the object base class. New-style classes give Python programmers many new (and initially confusing) features. One such feature is the descriptor protocol, and more specifically descriptors themselves.

Descriptors give Python programmers the ability to easily and efficiently create “managed attributes”. Managed attributes can be thought of as attributes that are not accessed directly. Instead their access is “managed” by something else, generally a class or a function.

If you haven’t come across this before you are probably wondering why one would want to manage attribute access? One reason might be that you don’t want people to be able to delete the attribute. Another reason may be that you need to ensure that your attribute data is always valid. Or perhaps attribute x is based on attribute y, so every time the value of y changes you want to update the value of x. From these few examples you can see the many possible cases where you might want to control access to certain attributes.

For those of you familiar with other programming languages, this type of access is often referred to as “getters and setters”. In many language, implementing “getters and setters” means using private variables and public functions that get and set the variable’s value. Since Python doesn’t (really) have private variables, the descriptor protocol is basically a built-in and “Python-ic” way to way to achieve something similar.

This article will introduce you to the descriptor protocol, descriptors, and properties. It will focus on demonstrating how to use them to create managed attributes. Since the descriptor protocol requires new-style classes, all of the examples in this article require Python 2.2 or newer.

A few definitions

Before moving forward, it is important to understand a few related terms. These terms will introduce some basic concepts and help you follow along with the remainder of the article.

descriptor protocol – The following three methods make up the descriptor protocol: __get__, __set__, and __delete__.

descriptor – An “object attribute with binding behavior, one whose attribute access has been overridden by methods in the descriptor protocol.” [1] In other words an “attributes whose usage resembles attribute access, but whose implementation uses method calls.”[2]

data descriptor – A descriptor with the __get__ and __set__ methods of the descriptor protocol defined.

non-data descriptor – A descriptor with only the __get__ method of the descriptor protocol defined. “Python methods (including staticmethod() and classmethod()) are implemented as non-data descriptors.” [3]

property – A built-in type that implements the descriptor protocol and allows you to easily create data descriptors.

Don’t worry if you don’t fully understand these definitions, the remainder of this article will hopefully clarify any confusion you have.

The Descriptor Protocol

Let’s take a closer look at the descriptor protocol and see how we can use it to create a descriptor. As previously mentioned, the descriptor protocol is made up of three methods: __get__, __set__, and __delete__. These methods have specific signatures and they are as follows, where self is the class that owns the methods:

__get__(self, instance, owner)
__set__(self, instance, value)
__delete__(self, instance)

These three methods represent the three basic operations that you perform on attributes in general: querying the value of that attribute; assigning a value to it; and, (very rarely) deleting it. They work as follows:

• The __get__ method is called when the attributes value is being queried. The __get__ method should return the (computed) attribute value or raise an AttributeError exception.” [4] This is where access to the attribute’s value is managed.
• The __set__ method is used in the assignment operation. It is called when we want to set the attribute value. This is where you can control what values, or types of values are being assigned to your attribute.
• Finally, the __delete__ method is called when we want to delete the attribute. Here you can (rarely) decide whether or not to delete the attribute.

There are also three different parameters passed to the three methods (excluding the standard self parameter for methods that belong to a class):

• owner – This “is always the owner class.” [5] This means that it is the actual class, and not an instance of the class. So if the descriptor is in a class called MyClass, owner will be that class.
• instance – An instance of class type owner. It is “the instance that the attribute was accessed through” [6], or None if the attribute is being accessed through the class (owner) instead of an instance.
• value- The value that the attribute is being set to.

This difference between owner and instance might be a bit confusing, so let’s look at a quick example. Let’s say we have a descriptor my_descriptor in the class MyClass, if we were to run the following code:

my_class_instance = MyClass()
print my_class_instance.my_descriptor

The second line queries the descriptors value and results in the __get__ method being called with the instance parameter being my_class_instance. The owner parameter will be the MyClass class.

Note: Notice that we treat the descriptor my_descriptor as though it is a normal attribute. We don’t call print my_class_instance.my_descriptor.__get__(my_class, MyClass). This is what was meant by: “attributes whose usage resembles attribute access, but whose implementation uses method calls.”[7]

If the following code were run:

print MyClass.my_descriptor

The __get__ method will again be called. This time the instance parameter will be None and the owner parameter will be the MyClass class.

Note: Only the __get__ method has the owner parameter. This means that it is the only function in the descriptor protocol that can be accessed through the class. Setting and deleting the descriptor through the class actually changes what the variable is. For example, if we tried to assign the numeric value 2 to a descriptor variable using the class, we would not access the descriptors __set__ method. Instead we would change the type of the variable from a descriptor to an integer with the value of 2.

A Simple Descriptor Example

A simple “transparent” descriptor example can be found in Listing 1. The first thing to notice in the code is that both the SimpleDescriptor and MyClass classes are “new-style” classes because they are derived from object. This is important because, as mentioned above, descriptors only work with “new-style” classes. The second point to notice is that the descriptor has class scope as opposed to instance scope. There are also two extra print statements included in the code. They are there to let us follow the execution a little more easily.

Listing 1

class SimpleDescriptor(object):

    def __get__(self, instance, owner):
        # Check if the value has been set
        if (not hasattr(self, "_value")):
            raise AttributeError
        print "Getting value: %s" % self._value
        return self._value

    def __set__(self, instance, value):
        print "Setting to %s" % value
        self._value = value

    def __delete__(self, instance):
        del(self._value)

class MyClass(object):
    my_value = SimpleDescriptor()

Using Listing 1 to execute the following code:

my_instance = MyClass()
my_instance.my_value = 416
print my_instance.my_value

The output would be the following:

Setting to 416
Getting value: 416
416

As you can see the second line (my_instance.data_descriptor = 416) calls the __set__ method and sets the _value attribute. When we call print my_instance.data_descriptor the __get__, method is called and the _value attribute is returned.

The Problem with the Simple Example

The previous example may look like a perfectly good descriptor, but there is something wrong with it. Take a look at what happens when we runs this new code:

my_instance = MyClass() #Create the first instance
my_instance.my_value = 416 #Set its' value
my_second_instance = MyClass() #Create the second instance
my_second_instance.my_value = 204 #Set its' value
print my_instance.my_value #What was the fist instance's value?

We get the following output:

Setting to 416
Setting to 204
Getting value: 204
204

Notice that when we set the second instance’s my_value descriptor to be 204, we are also setting the first instances. This is because my_value has class scope, so both instances (and the class itself) share the same instance of the SimpleDescriptor class. Since SimpleDescriptor only stores one value, they all actually share the same value. We will get the same results if we check what the classes value of my_value is:

my_instance = MyClass() #Create the first instance
my_instance.my_value = 416 #Set its' value
my_second_instance = MyClass() #Create the second instance
my_second_instance.my_value = 204 #Set its' value
print MyClass.my_value #What's the classes value?

We get the following results:

Setting to 416
Setting to 204
Getting value: 204
204

In the last line of the code (print MyClass.my_value #What's the classes value?) we simply use the class (ignoring both instances) in order to get the value of my_data. This will be an instance where the __get__ function will be called with the instance parameter set to None.

Note: This is not a problem if you want to share the value across all instances.

The Solution to the Problem

In order to get around this issue, you have to remember that if you want values unique to each instance your descriptors must store values that are unique to each instance. This can be in the instance itself, in dictionary in the descriptor, or perhaps in a text file. Just make sure that the value is unique to each instance. Though it seems like a simple solution, in practice a suitable solution for all cases is difficult to implement. As a result you should probably pick a specific implementation for your specific situations.

Using each instance as a key, one can store the value in a dictionary in the descriptor itself. The problem with this solution should be obvious to programmers familiar with Python dictionaries: only immutable types can be used as keys. This is fine if you know what object you are working with, but what about in the future when you want to add a descriptor to your sub-classed list?

Another solution is to store the value in the instance itself. You can do this by easily adding the value to the instance’s __dict__. The limitation is that the descriptor need to be given a suitable key so that there is no collision with anything already in the instance’s __dict__.

Listing 2 shows a solution to the problem from the previous section where the value is stored in the instance’s __dict__. Values are indexed using a key name provided during the creation of the descriptor. Aside from where we store the value, there is little difference between Listing 1 and Listing 2. Notice that the value name is a converted into a string. This is done to ensure that it can be used as a key.

Listing 2

class FixedDescriptor(object):

def __init__(self, value_name):
    self.value_name = str(value_name)

def __get__(self, instance, owner):
    if (instance is None):
        raise AttributeError
    elif (not instance.__dict__.has_key(self.value_name)):
        raise AttributeError
    return instance.__dict__[self.value_name]

    def __set__(self, instance, value):
        print "Setting to %s" % value
        instance.__dict__[self.value_name] = value

    def __delete__(self, instance):
        if (instance.__dict__.has_key(self.value_name)):
            del(instance.__dict__[self.value_name])


class MyClass(object):
    my_value = FixedDescriptor("__value")

The major usage difference between Listing 1 and Listing 2 is the fact that the __get__ method no longer works at class level; it only works at instance level. This should be obvious since this solution stores the value in the instance. If there is no instance, we have nowhere to store the value! If you attempt to access the descriptor at class level (the first if statement) an attribute error will be raised:

def __get__(self, instance, owner):
    if (instance is None):
        raise AttributeError
    elif (not instance.__dict__.has_key(self.value_name)):
        raise AttributeError
    return instance.__dict__[self.value_name]

Easy Data Descriptors with Properties

The descriptor provided in Listing 2 will work for many situations but for my money the easiest way to implement data descriptors is to use the property type. I would recommend using it unless you need the same descriptor across many different classes or attributes (i.e. for type validation). Properties can be thought of as a simple and easy way to create data descriptors. The property type implements the descriptor protocol and gets around the problem of where to store the descriptor value in an easy way: it lets the class that created the descriptor deal with it.

The full signature of the property function is as follows:

property(fget=None, fset=None, fdel=None, doc=None)

Where fget, fset, fdel are methods that will be called when the __get__, __set__, and __del__ members of the descriptor protocol are called. The doc parameter is a string that will be used as the docstring for the descriptor. If the doc parameter is not specified, the docstring of the fget method is used.

The signature of the fget, fset, fdel functions are as follows:

fget(instance)
fset(instance, value)
fdel(instance)

In the three signatures, instance is a reference to the object that owns the property attribute. This is the same instance that gets passed to the descriptor protocol. Since instance is the first parameter of each method, these are, for all intents and purposes, member methods of a class. The value parameter is the same as the value parameter that is passed to the __set__ method. It is what we are setting the attribute to.

The basic way to implement properties can be seen in Listing 3. As you can see, what we do for a property is very similar to what we did for our initial descriptor in Listing 1. We set up the three functions necessary for the descriptor protocol, and then use them to create our property attribute. Creating the property is very simple:

Listing 3

class MyClass(object):
    #Create the fget, fset, and fdel methods
    def __get_value(self):
        print "Getting value: %s" % self.__value
        return self.__value

    def __set_value(self, value):
        print "Setting to %s" % value
        self.__value = value

    def __del_value(self):
        del(self.__value)

    #Create the property
    my_value = property(__get_value
            , __set_value
            , __del_value
            , "This is my property")

    def __init__(self, value=0):
        self.my_value = value

my_value = property(__get_value
        , __set_value
        , __del_value
        , "This is my property")

Make Data Attributes Data-Descriptors

A common reason to use descriptors is to create read-only attributes. This means allowing callers to get (or access) an attribute’s value, but not allowing them to set its’ value. At first glance it seems as though one can accomplish this by using descriptors with only the __get__ method defined, i.e. a “non-data descriptor”.

While this seems to create a read-only descriptor attribute, it actually does nothing of the sort. Instead of using the descriptor’s __set__ function for the assignment operation (since no __set__ function is defined), the default Python assignment operation is used. This means that instead of stopping the assignment operation, a new attribute will be created in the instance (remember that descriptors have class scope) having the same name as the descriptor attribute and taking precedence in future operations.

This may be a bit confusing so let’s look at the example in Listing 4. It’s very much like our previous descriptor examples except it does not specify the __set__ and __del__ functions. Let’s look at what happens when we try to use this as though it were a read-only attribute:

Listing 4

class WonkyDescriptor(object):
    """This Descriptor isn't read only"""
    def __init__(self, value_name):
        self.value_name = str(value_name)

    def __get__(self, instance, owner):
        if (instance is None):
            raise AttributeError
        elif (not instance.__dict__.has_key(self.value_name)):
            raise AttributeError
        print "Getting a value"
        return instance.__dict__[self.value_name]

class MyClass(object):

    my_value = WonkyDescriptor("_value")

    def __init__(self, value):
        #initial value
        self._value = value

my_instance = MyClass(23) #Create the first instance
print my_instance.my_value
my_instance.my_value = "Don't set me!"
print my_instance.my_value

When we run this, we get the following output:

Getting a value
23
Don't set me!

As you can see we did not succeed in creating a read-only attribute, instead something else happened. The “Getting a value” string is printed out when we print out my_value for the first time. This means that we are accessing the value through the descriptor. We then set the value of the descriptor attribute and print out that new value. Since “Getting a value” isn’t printed out a second time, we know that the descriptor’s __get__ method is not accessed when the second print statement is called.

Let’s take a closer look at what is happening using the following code, which prints out the instance’s __dict__:

my_instance = MyClass(23) #Create the first instance
print my_instance.__dict__
my_instance.my_value = "Don't set me!"
print my_instance.__dict__

This results in the following, which explains what is happening:

{'_value': 23}
{'_value': 23, 'my_value': "Don't set me!"}

When we first access the __dict__ we see our value indexed by the _value key that we told the descriptor to use. Now look at the second __dict__. The old value is still there, but so is a new value my_value. When we assigned “Don’t set me!” to my_value, we didn’t overwrite the descriptor attribute at class scope, we created a new instance attribute! And it is not exactly a read-only attribute.

In order to create a read-only attribute, you need to create a data descriptor where the __set__ method raises an AttributeError exception. An easy way to do this is to create a property and only set the fget function. Instead of using the descriptor in Listing 4, we can construct a read-only attribute as follows:

def __get_value(self):
    return self._value
my_value = property(__get_value)

Conclusion

I hope that after reading this article you can see how powerful descriptors (especially data-descriptors) can be. Once you get the hang of them, they are a great way to implement “getters and setters” and read-only attributes. Data-Descriptors are also very useful for performing validation during the assignment operation, i.e. ensuring that a value remains a specific type, or in a specific form.

The more you play with descriptors and use them in your code, the more you’ll see how sophisticated and useful they can be. That being said, it’s important to remember that unless you have a good reason to use descriptors you probably don’t need them. There’s no need to sacrifice the readability of your code or the dynamism of Python just for the sake of using descriptors. But when you have a real reason for managed attributes, you’ll find that descriptors are more than up to the task.

[1] http://docs.python.org/ref/descriptor-invocation.html
[2] http://www.python.org/download/releases/2.2/descrintro/#property
[3] http://docs.python.org/ref/descriptor-invocation.html
[4] http://docs.python.org/ref/descriptors.html
[5] http://docs.python.org/ref/descriptors.html
[6] http://docs.python.org/ref/descriptors.html
[7] http://www.python.org/download/releases/2.2/descrintro/#property

Note: This article was first published the April 2008 issue of Python Magazine

By: Mark Mruss

GUI programming, like many other types of programming, can sometimes prove exhausting because you must repeat yourself over and over again. AVC is one tool available to Python GUI programmers that attempts to simplify things by synchronizing application data and GUI widgets.

Introduction

Every once in a while I find myself browsing the Internet trying to find out what’s new and exciting in the Python world. Sometimes I browse to find topics for this article; other times mere curiosity draws me across the web. While I was browsing the other day, I stumbled across AVC: the Application View Controller [1]. I was immediately intrigued by it because its’ name is so similar to the Model View Controller (MVC) pattern. Being familiar with the Model View Controller pattern, and admittedly having struggles with it in the past, I decided to check out AVC to determine if it might be a viable alternative.

After reading about AVC I was intrigued for several reasons. The main reason was the promise of “a multiplatform, fully automatic, live connection among graphical interface widgets and application variables.” [2] This means that graphical widgets can be connected to variables and automatically synchronized. One of the (many?) problems with Graphical User Interface (GUI) programming is that you often find yourself doing the same thing over and over again. One of the things that you end of doing over and over again is setting the contents of a widget based on the value of a variable, and then subsequently, setting that variable’s value based on the current state of the widget. Whenever someone promises me an automatic connection between GUI widgets and my variables, I’m interested.

The other reason for investigating AVC is my past struggles with the Model View Controller pattern. I like the decoupling that the MVC pattern seems to provide, but I often feel as though I am needlessly duplicating code in order to stick with the pattern. I looked into AVC hoping to find a simple framework that will avoid this and guide the separation of my application data from my GUI logic. However in the end, I realized that AVC (in its current state), does not appear to achieve this across all widget toolkits. It does allow you to separate your application from much of the GUI logic that one would normally need, but it does not appear to allow you to fully separate the data from the GUI.

AVC is being developed by Fabrizio Pollastri, and is released under version 3 of the GPL. I was unable to find much information about the project’s future or design objectives. But judging by the release notes, it is still being actively developed. The project is only at version 0.5 so don’t be surprised if the API (Application Programming Interface) or some of the goals change. Although relatively new, AVC bears further investigation because it already simplifies your GUI programming and contains the potential to do much more. In this article, I will introduce some of the main concepts behind AVC, and provide a small, working example using AVC and PyGTK.

Installing AVC

AVC is a “Python module written in pure python” [3]. This means that AVC is easy to install and does not have any dependencies aside from Python 2.2 or greater. You can download the current version (0.5.0) from the website or, if you are a lucky Linux user like me, you can install via your handy package manger. If you download the source, you can install it after extraction, using the following command:

python setup.py install

Depending on which widget toolkit you plan on using, the following requirements must also be met:

• GTK+: PyGTK 2.8 – 2.10
• Qt: PyQt or PyQt4v3 – v4
• Tk: Tkinter 2.4
• wxWidgets: wxPython 2.6

You can easily install any of the above toolkits using your favourite package manager, a source archive, or a binary installer found on each toolkit’s main website. Once AVC is installed, you can test the installation by importing the avc module from the interactive interpreter. If all goes well you will see something similar to the following:

$ python
Python 2.4.4 (#2, Jan  3 2008, 13:36:28)
[GCC 4.2.3 20071123 (prerelease) (Debian 4.2.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import avc
>>>

Widget Support

In its current state, AVC does not support all widgets available across all toolkits. It does currently support the following widgets:

• Button
• Check Box
• Combo Box (Not supported in Tk)
• Entry
• Label
• Radio Button
• Slider
• Spin Button
• Status Bar (Only supported by GTK+ and wxWidgets)
• Text View
• Toggle Button.

The support of each of these widget types is then handled by specific widgets in each toolkit.

Name-Matching

In order to automatically synchronize variables and GUI widgets, AVC uses a name-matching technique. AVC supports two methods of name matching. The first method requires that the variable and the widget have the same name. For example, if you have a variable named my_data it will match with a widget also named my_data. In other words, if a variable and a widget have the same name they will be connected and their values synchronized.

The second name-matching technique allows you to connect variables to one or more widgets. Widgets can only be connected to one variable, but variables can be connected to any number of widgets. Using the second name-matching technique, you have a match if the widget name contains a double-underscore (__), and the portion before the double-underscore matches the variable name. For example, if you have variable named my_numeric_value and you want to match it with a Label widget and an Entry widget, you do this by naming the widgets as follows:

my_numeric_value__label
my_numeric_value__entry

Again, the text before the “__” must match the variable name, and the text after the “__” can be whatever you want to differentiate your widgets or ensure uniqueness (if the toolkit requires it).

A GTK+, PyGTK, and AVC Example

For this AVC example, I will use the GTK+ widget toolkit and PyGTK. PyGTK a “is a set of bindings to the GTK+ user interface toolkit for the Python language” [4]. This means that it lets us write GUI applications in Python using the GTK+ widget toolkit. To avoid any flame wars: I did not choose GTK+ because it is the best toolkit but because it is the toolkit with which I am the most familiar. In order to run this example you will need to have PyGTK 2.8 or later installed. You can get the latest version of PyGTK from the PyGTK website. [5]

As I mentioned earlier, AVC works with many different widgets types. It also has many different features. I won’t be covering all of the features or widgets types in this column; if you are interesting in learning about some of these features or how to work with different widgets types, please see the AVC documentation. [6]

This example is relatively simple, but it should illustrate some of the basic features of AVC. The example application asks for a person’s name and then displays that name in pop-up dialog. It will have the following features:

1. A gtk.TextView widget to display the person’s name and allow the name to be edited. The gtk.TextView widget will be synchronised with a variable.
2. A gtkButton that will pop-up a gtk.MessageDialog to show the person’s name.
3. A gtk.Button that will reset the name to the name with which the application was initialized.

To begin we need to import all of the modules necessary:

import gtk
import avc.avcgtk

Note: I am using a gtk.TextView widget instead of a gtk.Entry widget. In its current state, AVC does not propagate changes in gtk.Entry widgets back to the connected variable. I’m not sure if this is a current limitation or part of the design.

The above code imports the PyGTK module necessary to work with GTK+. It also imports the avcgtk module, which is used to link our “application variables” with our GTK+ GUI. Depending on what widget toolkit you use, you will have to use different modules. For example, if you are working with Qt4, you will import:

import avc.avcqt4

The next step is to create the “application” class. This will be the class that contains the data. In this example it will also create the GUI:

class gtkAVC(avc.avcgtk.AVC):

Notice that we derive our application from the avcgtk module’s AVC class. This is the GTK+ specific AVC class with which we will work. We will also create an __init__ method in the gtkAVC class. This is where we will initialize the data that will be connected to the GUI and create the GUI via the init_gui method:

def __init__(self, name=""):
    #set the variable and save the initial name
    self.initial_name = name
    self.name = name
    #setup GUI
    self.init_gui()

What we have here is pretty simple. We have the optional parameter name that we store as the current name and as the initial name. We then call the init_gui method that will create the GUI. The contents of the init_gui method can be seen in Listing 1. I won’t explain most of this code since the majority of it is PyGTK specific. However I will make note of several relevant segments below.

Listing 1

def init_gui(self):

    #Setup the window and the layout manager
    self.window = gtk.Window()
    self.window.connect("destroy", gtk.main_quit)

    self.h_layout = gtk.HBox()
    self.window.add(self.h_layout)

    #create the widgets
    self.text_label = gtk.Label("Say hello:")
    self.h_layout.add(self.text_label)

    self.frame = gtk.Frame()
    self.h_layout.add(self.frame)
    self.scrolled_window = gtk.ScrolledWindow()
    self.frame.add(self.scrolled_window)

    self.text_view = gtk.TextView()
    self.text_view.set_name("name__text_view")
    self.scrolled_window.add(self.text_view)

    self.hello_button = gtk.Button("Hello!")
    self.hello_button.connect("clicked", self.on_hello_clicked)
    self.h_layout.add(self.hello_button)

    self.reset_button = gtk.Button("Reset")
    self.reset_button.connect("clicked", self.on_reset_clicked)
    self.h_layout.add(self.reset_button)

    self.window.show_all()

The only AVC specific code segment in the init_gui function is where we set the name of the gtk.TextView widget:

self.text_view.set_name("name__text_view")

This is all the code that we need to connect the gtk.TextView widget with the name variable that we created at the beginning of the __init__ function. (Now, how’s that for simple?) Notice that the name section of the “name__text_view” string will match with the self.name variable. It is also important to note that the variable and the widget are not yet connected. They will become connected when the avc_init method is called.

The other non-AVC related code of note is the connection of both gtk.Button widgets’ clicked signals with signal handlers. We connect the “hello” button to the on_hello_clicked method and the “reset” button to the on_reset_clicked method.

That’s it for the init_gui function, now let’s look at the two signal handlers. The on_hello_clicked method, shown in Listing 2, simply displays the name in a gtk.MessageDialog. What should be interesting to GUI programmers about this function is that instead of reading the data from the gtk.TextView widget, we simply reference self.name for the dialog’s text. We can do this because self.name and the gtk.TextView widget are connected and their values are always the same.

Listing 2

def on_hello_clicked(self, widget):
    def close(dialog, response):
        #Close the dialog on ok
        dialog.destroy()
    message_dlg = gtk.MessageDialog(self.window
        , 0
        , gtk.MESSAGE_INFO
        , gtk.BUTTONS_OK
        , ("Hello: %s" % self.name))
    message_dlg.connect("response", close)
    message_dlg.run()

The on_reset_clicked function is also very simple and straightforward. We simply reset the name to the initial name:

def on_reset_clicked(self, widget):
    #Reset the name to the initial name
    self.name = self.initial_name

Again, notice that we don’t have to worry about doing anything to the gtk.TextView widget that displays the current name. By simply modifying the data, what is displayed in the GUI will automatically be updated by AVC.

The next and final step in our test application is to create our class instances and actually show the GUI:

if (__name__ == "__main__"):
    gtk_avc = gtkAVC("Mark Mruss")
    gtk_avc.avc_init()
    gtk.main()

Since I am working in a Python file, I perform a simple test to make sure that the file is being launched directly instead of being imported as a module. I then create an instance of the gtkAVC class setting the name to be my name. The avc_init method is then called (which is a member of the avc.avcgtk.AVC class). The avc_init method is where all of the magic happens. It is where AVC goes through an application’s variables and connects them to the GUI widgets based upon the name matching technique explained earlier. avc_init must be called after all of the data in the “application” and the widgets have been created.

The final step in this example, is to run the gtk.main function. This is the “main loop” of our PyGTK based application and is standard when working with PyGTK.

The entire code can be seen in Listing 3. When you run the code you will be greeted with something similar to Figure 1. It may not seem like much, but what we have here is a fully functional AVC “application”. This simple “application” illustrates how easy it is to create an automatic connection between your data and your GUI.

Figure1

Listing 3

#! /usr/bin/env python
import gtk
import avc.avcgtk

class gtkAVC(avc.avcgtk.AVC):

    def __init__(self, name=""):
        #set the variable and save the initial name
        self.initial_name = name
        self.name = name
        #setup GUI
        self.init_gui()

    def init_gui(self):

        #Setup the window and the layout manager
        self.window = gtk.Window()
        self.window.connect("destroy", gtk.main_quit)

        self.h_layout = gtk.HBox()
        self.window.add(self.h_layout)

        #create the widgets
        self.text_label = gtk.Label("Say hello:")
        self.h_layout.add(self.text_label)

        self.frame = gtk.Frame()
        self.h_layout.add(self.frame)
        self.scrolled_window = gtk.ScrolledWindow()
        self.frame.add(self.scrolled_window)

        self.text_view = gtk.TextView()
        self.text_view.set_name("name__text_view")
        self.scrolled_window.add(self.text_view)

        self.hello_button = gtk.Button("Hello!")
        self.hello_button.connect("clicked", self.on_hello_clicked)
        self.h_layout.add(self.hello_button)

        self.reset_button = gtk.Button("Reset")
        self.reset_button.connect("clicked", self.on_reset_clicked)
        self.h_layout.add(self.reset_button)

        self.window.show_all()

    def on_hello_clicked(self, widget):
        def close(dialog, response):
            #Close the dialog on ok
            dialog.destroy()
        message_dlg = gtk.MessageDialog(self.window
            , 0
            , gtk.MESSAGE_INFO
            , gtk.BUTTONS_OK
            , ("Hello: %s" % self.name))
        message_dlg.connect("response", close)
        message_dlg.run()

    def on_reset_clicked(self, widget):
        #Reset the name to the initial name
        self.name = self.initial_name

if (__name__ == "__main__"):
    gtk_avc = gtkAVC("Mark Mruss")
    gtk_avc.avc_init()
    gtk.main()

Benefits and Limitations

For a project that is little more than one year old (the first release was in January 2007) AVC appears quite stable and ready to be used. This is not to say that AVC is perfect. I did encounter a few issues.

The most notable issue that I encountered is the tightly coupled structure that AVC seems to impose on your application. For a module that tries to simplify the relationship between application data and GUI’s, this feels like a bit of an issue because. For example, when testing using PyGTK I was able to easily separate the GUI and the “application” into separate classes. But when I tried to do the same thing using PyQt4, I encountered an error because the avc_timer function in the avc.avcqt4.AVC module expects that the “application” class is itself a QObject (namely the actual QApplication). I find this counterintuitive since being able to separate the data from the GUI-logic seems to be a goal of AVC and similar tools. If you are able to separate the “application” fully from the GUI, you can easily connect your “application” to different GUI’s, and (by changing the AVC base class) even different widget toolkits.

I would also be interested in the ability to connect more types of data to more types of widgets. For example, I am interested in synchronizing Python lists with the contents of List boxes and Combo boxes. This is in addition to the current ability to synchronize the selection in the two widgets with integers.

Of course AVC also has a lot going for it. For one, I really like that AVC works with different widget toolkits. I have not seen this very often. It is a real benefit, especially when it comes to people adopting AVC. Hopefully in the future AVC will allow for further separation of the data and the GUI, truly enabling us to use different GUI’s and different widget toolkits with our data.

What I like most about AVC is its simplicity. I have played around with similar solutions and AVC is one of the leaders in terms of ease of use. I was able to get create a simple AVC based application in almost no time at all. (The time that it did take was spent trying to remember PyGTK.) Much of this has to do with the streamlined nature of AVC. There is very little to configure regardless of what widget toolkit you use. While this simplicity limits what AVC can currently achieve compared to other solutions, it also makes the barrier to entry low and the usage trivial. User friendliness is certainly a good thing in a package that aims to make GUI application development easier.

Conclusion

While the data that we worked with in the earlier example wasn’t that complicated (a simple string), I hope that it illustrates how helpful AVC is if you are working with a large amount of data or data that could be updated by an external source. Instead of having to worry about what widgets or toolkits the data was being displayed in, AVC lets us work with the data and GUI transparently, leaving the synchronization to AVC.

If what AVC offers sounds interesting to you, download it from the AVC website and try it out. Although it it’s not there just yet with enough attention, usage, and contributions from the Python community, AVC should reach full-feature status in no time.

[1] http://avc.inrim.it/html/
[2] http://avc.inrim.it/html/
[3] http://avc.inrim.it/html/
[4] http://faq.pygtk.org/index.py?req=all#1.1
[5] http://www.pygtk.org/downloads.html
[6] http://avc.inrim.it/doc/user_manual.pdf

Note: If you haven’t already, please check out my Python version poll.

Note: This article was first published the March 2008 issue of Python Magazine

Mark Mruss

Over the past few years Google has expanded it’s services beyond those of a normal search engine. One of those new services is the Google Calendar. This article will provide an introduction to working with the Google Calendar using Python.

Introduction

As many of you know, Google has branched out and started offering more services besides their ubiquitous search engine. You have email, calendars, documents, spreadsheets, photos, maps, videos, source code hosting, and the list goes on. Fortunately for us Python programmers, Google released the Google data Python Client Library on March 26th, 2007, giving Python programmers easy access to some of these services.

What the Google data Python Client Library, or “gdata-python-client”, does is provide “a library and source code that makes it easy to access data through Google Data APIs.” [1] This leads to the question: “what are the Google Data APIs?” In the words of Google: “The Google data APIs provide a simple standard protocol for reading and writing data on the web. These APIs use either of two standard XML-based syndication formats: Atom or RSS.”[2]

The Google services that use the Google data APIs include many of the services that we have grown to know and love:

• Google Apps
• Google Base
• Blogger
• Google Calendar
• Google Code Search
• Google Documents
• Google Notebook
• Picasa Web Albums
• Spreadsheets
• YouTube

This tutorial will only deal with the Google Calendar service specifically, but it’s important to know that many of the techniques used here can easily be applied to other Google services.

The Google data Python Client Library requires Python 2.2 or greater and the ElementTree module to be installed. I recommend using Python 2.5 since the ElementTree module is included in that release of Python. This column assumes that you are using Python 2.5 and version 1.0.10 of the “gdata-python-client”.

Getting and Installing the “gdata-python-client”

We must first download and install the “gdata-python-client” files from the “gdata-python-client” website.[3] Once you have downloaded the compressed file, extract it’s contents to a folder. You will then need to browse to that folder and run the extracted setup.py file with the install command with root access. For me, the command looked like this:

# python2.5 setup.py install

Notice that I ran python2.5 instead of simply python. I did this to ensure that Python 2.5 was used instead of a previous version of Python.

Once you have done this, you can test to make sure that the gdata-python-client module has been installed properly by trying to import the gdata module. You can test this easily in the interactive Python shell:

$ python2.5
Python 2.5.2a0 (r251:54863, Jan  3 2008, 17:59:56)
[GCC 4.2.3 20071123 (prerelease) (Debian 4.2.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import gdata
>>>

If everything works properly you should see no errors when importing gdata. If you do run into problems, you can consult the install.txt file that was extracted with the rest of the files in the “gdata-python-client” archive that you downloaded.

Getting Started

There are some great Google Calendar examples that come with the “gdata-python-client” module. I found them to be very useful learning tools. The “Google Calendar Developers Guide”[4] is also very helpful. If you get stuck it is essential reading. The “Google Calendar API Reference Guide”[5] is also an indispensable help.

It is important to remember that when you are working with the Google Calendar service, you are actually working with XML data. You are sending and receiving XML data to and from the Google Calendar service – Atom or RSS feeds to be precise. The Python classes that wrap these feeds, or XML blocks, are dynamically “formed” around the XML. When you access something like the following:

e_link.href

You are actually accessing the “href” attribute of an XML link block that may look something like this:

<ns0 :link href="http://www.google.com/calendar/feeds/<username>/private/full" rel="alternate" type="application/atom+xml" xmlns:ns0="http:www.w3.org/2005/Atom" />

In the above example, if we had the link as an instance (perhaps an atom.Link object), e_link.rel would be equal to “alternate”.

Logging In

There are two types of Google calendars – public calendars and private calendars. Public calendars do not require any authentication while private calendars do. There are three forms of authentication for private calendars: “AuthSub proxy” authentication, “ClientLogin” authentication, and “Magic cookie” authentication.

“AuthSub proxy” authentication is meant for web applications so it will not be covered in this tutorial. “Magic cookie” authentication requires a string (the magic cookie) obtained from your Google Calendar settings page. This type of authentication gives you read only access to a private calendar. Its usage is quite specific and will not be covered in this column either.

For this column we will focus on good old-fashioned “ClientLogin” authentication. In order to perform “ClientLogin” authentication we need two things: a Google Calendar username and a password. If you do not have these, head over to the Google Calendar website [6] and get yourself signed up. If you have gmail or have signed up for other Google services, you can probably sign in using that username and password.

Note: I say username and so does the documentation, but this will actually be an email address.

We are going to read the username and password in from the command line using the raw_input function and the getpass module. I must thank the people who wrote the Google code examples for introducing me to the getpass module. Before looking through the examples, I had not heard of it.

The raw_input function simply reads a line of input from the command line and returns it with the newline stripped. The getpass.getpass function does the exact same thing as raw_input except it does not echo the input, which is handy when working with passwords.

We need to import the getpass module:

import getpass

Then, in our main function we will ask the user to input their username and password:

def main():
    username = raw_input("Enter your username: ")
    password = getpass.getpass("Enter your password: ")

if __name__ == "__main__":
    main()

Now that we have our username and password, we need to log the user in. To do this we are going to use the CalendarService class. The CalendarService “extends the GDataService to streamline Google Calendar operations.” [7] The GDataService class “provides CRUD ops. and programmatic login for GData services.” [8]. CRUD is an acronym that stands for Create, Retrieve, Update, and Delete. What the CalendarService class will allow us to do is create, retrieve, update, and delete things from a Google Calendar.

The first step in logging in is to create an instance of the CalendarService class. To do this we will pass in the username and password that we have collected:

calendar_service = gdata.calendar.service.CalendarService(username
    , password
    , "PythonMagazine-Calendar_Example-1.0")

The third parameter that we pass to the CalendarService constructor is the “source” string. This is a short “string identifying your application, for logging purposes. This string should take the form: “companyName-applicationName-versionID”"[9]

Now that we have a CalendarService instance, we need to login. To do this we will use the ProgrammaticLogin function. This will log into the Google Calendar using the CalendarService classes current username and password. The ProgrammaticLogin function can raise three possible exceptions that we want to be aware of:

1. CaptchaRequired – Raised if the login requires a “captcha” response in order to login.
2. BadAuthentication – Raised if the username and/or password were not accepted by the Google Calendar.
3. Error – Raised if a 403 error occurs that is not a “CaptchaRequired” or “BadAuthentication” error.

For this example, shown in Listing 1, we will only worry about the “BadAuthentication” exception.

Listing 1

def main():
    username = raw_input("Enter your username: ")
    password = getpass.getpass("Enter your password: ")

    calendar_service = gdata.calendar.service.CalendarService(username
        , password
        , "PythonMagazine-Calendar_Example-1.0")
    try:
        calendar_service.ProgrammaticLogin()
    except gdata.service.BadAuthentication, e:
        print "Authentication error logging in: %s" % e
        return
    except Exception, e:
        print "Error Logging in: %s" % e
        return

Working With Calendars

Now that we’ve logged into the calendar service, we can start working with the user’s available calendars. To get a list of all of the available calendars you can call the GetAllCalendarsFeed function. This will return a CalendarListFeed instance representing all of the user’s calendars. Since I want to show how to add and delete Calendars, I’m going to use the GetOwnCalendarsFeed function to get a list of all of the calendars that the “authenticated user has owner access to.”[10]

The GetOwnCalendarsFeed function returns a CalendarListFeed class instance. This contains some information (like a title) along with a list of CalendarListEntry classes. Each CalendarListEntry in this list represents a calendar. An example of a function that uses GetOwnCalendarsFeed and then prints out information about each calendar can be found in Listing 2. Sample output from this function can be found in Listing 3. The function was passed a CalendarService after logging in:

list_own_calendars(calendar_service)

Listing 2

def list_own_calendars(calendar_service):

    try:
        #Get the CalendarListFeed
        all_calendars_feed = calendar_service.GetOwnCalendarsFeed()
    except Exception, e:
        print "Error getting all calendar feed: %s" % (e)
        return
    #Print the feed's title
    print all_calendars_feed.title.text
    #Now loop through all of the CalendarListEntry items.
    for (index, cal) in enumerate(all_calendars_feed.entry):
        #Print out the title and the summary if ther is one
        if (cal.summary is not None):
            print "%d) %s - Summary: %s" % (
                index, cal.title.text, cal.summary.text)
        else:
            print "%d) %s" % (index, cal.title.text)
        #Print out the authors
        print "\tAuthors:"
        for author in cal.author:
            print "\t\t%s" % (author.name.text)
        #Print out other information
        print "\tPublished: %s \n\tUpdated: %s \n\ttimezone: %s" % (
            cal.published.text, cal.updated.text, cal.timezone.value)
        print "\tColour: %s \n\tHidden: %s \n\tSelected: %s" % (
            cal.color.value, cal.hidden.value, cal.selected.value)
        print "\tAccess Level: %s" % (cal.access_level.value)

Listing 3

Mark Mruss's Calendar List
0) Mark Mruss - Summary: Main Calendar
	Authors:
		Mark Mruss
	Published: 2008-02-07T04:15:15.701Z
	Updated: 2008-02-07T03:43:26.000Z
	timezone: America/Toronto
	Colour: #5229A3
	Hidden: false
	Selected: true
	Access Level: owner
1) PythonMagazine - Summary: Calendar for Articles
	Authors:
		PythonMagazine
	Published: 2008-02-07T04:15:15.702Z
	Updated: 2008-02-07T03:37:15.000Z
	timezone: America/Toronto
	Colour: #0D7813
	Hidden: false
	Selected: true
	Access Level: owner

Adding a Calendar

If we want to add a calendar, we need to create a CalendarListEntry class instance and then call the CalendarService classes InsertCalendar function. Let’s say I wanted to add a Banking calendar to my account, I could do the following to create the CalendarListEntry:

new_calendar = gdata.calendar.CalendarListEntry()
new_calendar.title = gdata.atom.Title(text="Banking")
new_calendar.summary = gdata.atom.Summary(text="Bills and payments")
new_calendar.timezone = gdata.calendar.Timezone(value="America/Toronto")
new_calendar.hidden = gdata.calendar.Hidden(value="false")
new_calendar.selected = gdata.calendar.Selected(value="true")

The calendar now needs to be added to the account:

try:
    created_calendar = calendar_service.InsertCalendar(new_calendar)
except gdata.service.RequestError, e:
    print "Error adding calendar: %s" % (e[0]["reason"])

This will add the new calendar to the authenticated users list of calendars, make it visible, and select it. Notice that InsertCalendar also returns a CalendarListEntry instance. This is the calendar that was actually added to the authenticated user’s list of calendars. It is wrapping the XML that represents the actual calendar, as opposed to the smaller version that we created for insertion.

Deleting a Calendar

Deleting a Calendar is very easy. You simply need to call the CalendarService classes DeleteCalendarEntry function. The DeleteCalendarEntry function takes three parameters which are documented in the source code:

1. edit_uri – The edit uri (Uniform Resource Identifier) of the Calendar that you want to delete.
2. url_params – Defaults to None. A dictionary containing URL parameters that will be included in the delete.
3. escape_params – Defaults to True. A boolean that controls whether or not the url_params will be escaped.

Note: There is another optional parameter: extra_headers, which is the second parameter, but at this point it does not seem to be used at all in the source code.

The simplest case is to ignore the optional parameters and simply pass in the edit uri of the calendar that you wish to delete. You can get the edit uri of a calendar by calling the GetEditLink function of the CalendarListEntry instance that represents the calendar that you are going to delete. An example of a function that will delete a calendar can be seen in Listing 4. This function takes a CalendarService instance and a CalendarListEntry instance as parameters.

Listing 4

def delete_calendar(calendar_service, calendar):
    e_link = calendar.GetEditLink()
    if (e_link is not None):
        try:
            calendar_service.DeleteCalendarEntry(e_link.href)
        except gdata.service.RequestError, e:
            print "Error deleting calendar: %s" % (e[0]["reason"])

Listing Events

Now let’s take a look at events. Events are items that are added to a specific calendar. If you want to remind yourself to pay your bills at the end of the month you might add that to your banking calendar. That “note” on your calendar is an event.

You can get a list of events for the primary calendar using the CalendarService classes GetCalendarEventFeed function. Since you may be working with more than one calendar, it’s probably more useful to be able to list the events for a specific calendar. You can do this in one of two ways:

1. You can pass in the optional uri parameter to the GetCalendarEventFeed function. From testing I found that a Calendar’s “alternate” link works.
2. You can use the CalendarService classes CalendarQuery function to query for a specific calendars event feed.

If you want to use the first option you can use the CalendarListEntry classes GetAlternateLink member function to get the uri, and then pass it to GetCalendarEventFeed:

a_link = calendar.GetAlternateLink()
#Make sure the link is valid
if (a_link is not None):
    event_feed = calendar_service.GetCalendarEventFeed(a_link.href)

If you use the CalendarQuery method, you need to get the calendar username (or id) of the calendar whose events you want to query. It seems strange to me that there appears to be no way to get a calendar’s username besides parsing one of the calendar’s links. The username of a calendar can be found in the alternate link after “feeds” and before the visibility and projection:

http://www.google.com/calendar/feeds/< <username>>/private/full

Note: You can use the username “default” to query the default calendar.

For the sake of simplicity I will use the alternate link method for my examples. A full example that prints out calendar data and a calendar’s events can be found in Listing 5. The method that prints out the event data is called print_event_feed. It is called near the end of the list_own_calendars method:

# Now Print out the events
print "Events:"
a_link = cal.GetAlternateLink()
if (a_link is not None):
    event_feed = calendar_service.GetCalendarEventFeed(a_link.href)
    print_event_feed(event_feed)

Listing 5

#! /usr/bin/env python

import gdata.calendar.service
import getpass

def print_event_feed(event_feed):

    for index, event in enumerate(event_feed.entry):
        print "\t%d) %s\r\n\tContent: %s" % (
                index, event.title.text, event.content.text)
        print "\t\tWho:"
        for person in event.who:
            print "\t\t\tName: %s\n\t\t\temail: %s" % (person.name
                , person.email)
        print "\t\tAuthors:"
        for author in event.author:
            print "\t\t\t%s" % (author.name.text)
        print "\t\tWhen:"
        for e_index, e_time in enumerate(event.when):
            print "\t\t\t%d) Start time: %s\n\t\t\tEnd time: %s" % (
                e_index
                , e_time.start_time
                , e_time.end_time)

def list_own_calendars(calendar_service):

    try:
        #Get the CalendarListFeed
        all_calendars_feed = calendar_service.GetOwnCalendarsFeed()
    except Exception, e:
        print "Error getting all calendar feed: %s" % (e)
        return
    #Print the feed's title
    print all_calendars_feed.title.text
    #Now loop through all of the CalendarListEntry items.
    for (index, cal) in enumerate(all_calendars_feed.entry):
        #Print out the title and the summary if there is one
        if (cal.summary is not None):
            print "%d) %s - Summary: %s" % (
                index, cal.title.text, cal.summary.text)
        else:
            print "%d) %s" % (index, cal.title.text)
        #Print out the authors
        print "\tAuthors:"
        for author in cal.author:
            print "\t\t%s" % (author.name.text)
        #Print out other information
        print "\tPublished: %s \n\tUpdated: %s \n\ttimezone: %s" % (
            cal.published.text, cal.updated.text, cal.timezone.value)
        print "\tColour: %s \n\tHidden: %s \n\tSelected: %s" % (
            cal.color.value, cal.hidden.value, cal.selected.value)
        print "\tAccess Level: %s" % (cal.access_level.value)
        # Now Print out the events
        print "\tEvents:"
        a_link = cal.GetAlternateLink()
        if (a_link is not None):
            event_feed = calendar_service.GetCalendarEventFeed(a_link.href)
            print_event_feed(event_feed)

def main():
    username = raw_input("Enter your username: ")
    password = getpass.getpass("Enter your password: ")

    calendar_service = gdata.calendar.service.CalendarService(username
        , password
        , "PythonMagazine-Calendar_Example-1.0")
    try:
        calendar_service.ProgrammaticLogin()
    except gdata.service.BadAuthentication, e:
        print "Authentication error logging in: %s" % e
        return
    except Exception, e:
        print "Error Logging in: %s" % e
        return

    list_own_calendars(calendar_service)

if __name__ == "__main__":
    main()

An example of the output produced by Listing 5 can be seen in Listing 6. Don’t mind the formatting – it’s not pretty. It’s merely meant to give you an example of some of the data that you can mine from an event feed.

Listing 6

Mark Mruss's Calendar List
0) Mark Mruss - Summary: Main Calendar
	Authors:
		Mark Mruss
	Published: 2008-02-12T02:30:18.731Z
	Updated: 2008-02-08T04:03:10.000Z
	timezone: America/Toronto
	Colour: #5229A3
	Hidden: false
	Selected: true
	Access Level: owner
	Events:
	0) Dinner at the Drake
	Content: Dinner
		Who:
			Name: Mark Mruss
			email: mark.mruss@gmail.com
		Authors:
			Mark Mruss
		When:
			0) Start time: 2008-02-15T21:00:00.000-05:00
			End time: 2008-02-15T22:30:00.000-05:00
	1) Cezanne's Closet
	Content: Cezanne's Closet
		Who:
			Name: Mark Mruss
			email: mark.mruss@gmail.com
		Authors:
			Mark Mruss
		When:
			0) Start time: 2008-02-09
			End time: 2008-02-11

Adding an Event

Adding a new event to a calendar is very similar to creating a new calendar. You need to create a new CalendarEventEntry instance. You then populate it with options, and pass it to the CalendarService classes InsertEvent function, along with the alternate link of the calendar to which you would like to add the event.

If you take a look at Listing 7 you can see a simple example of how to add an all day event. If you want to specify an event that lasts for less then a day, or include a specific start and end time, you need to use the “RFC 3339 timestamp” format for your start_time and end_time values. Also notice that just like adding a Calendar, the newly created event is returned by the InsertEvent function.

Listing 7

a_link = calendar.GetAlternateLink()
if (a_link is not None):

    new_event = gdata.calendar.CalendarEventEntry()
    new_event.title = gdata.atom.Title(text="New Article")
    new_event.content = gdata.atom.Content(text="Write Article")
    new_event.when.append(gdata.calendar.When(
        start_time="2008-02-20"
        , end_time="2008-02-21"))
    try:
        created_event = calendar_service.InsertEvent(new_event
            , a_link.href)
    except gdata.service.RequestError, e:
        print "Error adding event: %s" % (e[0]["reason"])

Deleting an Event

Deleting an event is (you guessed it) almost identical to deleting an entire calendar. This is because both helper functions wrap the same GDataService base class function. However the 'DeleteEvent event function does a little bit more work than the DeleteCalendarEntry function by making sure that your edit_uri is in the correct format.

To delete an event you simply need to call the DeleteEvent function of the CalendarService class. An example of a function that will delete an event can be seen in Listing 8. This function takes a CalendarService instance and a CalendarEventEntry instance as parameters. As with the majority of these actions, the CalendarService instance should already be authenticated.

Listing 8

def delete_event(calendar_service, event):
    e_link = event.GetEditLink()
    if (e_link is not None):
        try:
            calendar_service.DeleteEvent(e_link.href)
        except gdata.service.RequestError, e:
            print "Error deleting event: %s" % (e[0]["reason"])

Conclusion

Hopefully this article has given you a small taste of what is possible with the gdata-python-client module and Google Calendars. Maybe you can already imagine a use for these features in your next Python application? Please keep in mind that there is much more that can be done with Google Calendars, including updating existing Calendars and Events (CalendarService member functions: UpdateCalendar and UpdateEvent), and almost any other task you can perform using the “online” version of the Google Calendar service.

Also, it is important to note that the Python documentation for the Google Calendar service is sparse but growing. A lot of what is contained in this article was found through browsing the examples, the gdata-python-client source code, other people’s examples, as well as a whole bunch of trial and error on my part. Since this is the case, there may be other ways, or more preferred methods of accomplishing what I have shown here.

[1] http://code.google.com/p/gdata-python-client/
[2] http://code.google.com/apis/gdata/index.html
[3] http://code.google.com/p/gdata-python-client/
[4] http://code.google.com/apis/calendar/developers_guide_python.html
[5] http://code.google.com/apis/calendar/reference.html
[6] http://www.google.com/calendar
[7] gdata.calendar.service.py (From the source comments)
[8] gdata.service.py (From the source comments)
[9] http://code.google.com/apis/accounts/AuthForInstalledApps.html
[10] http://code.google.com/apis/calendar/developers_guide_python.html

By: Mark Mruss

Note: This article was first published the February 2008 issue of Python Magazine

Of all the tasks assigned to programmers, commenting code and writing documentation are among the most disliked. This article introduces you to Python’s documentation strings. While they won’t make commenting your code any more enjoyable, they will provide a systematic approach to doing it, as well as access to additional tools for documentation generation and testing.

You’ve just finished your new Python module. You can’t wait to upload it to the Web and let all the other Python hackers start using it. The only step left is the most dreaded for many programmers: documentation. Unless you’ve been commenting and documenting your code as you wrote it, you’re going to have to go back through each source file, class, and function, trying to remember exactly what your code was supposed to do. Not an enjoyable task.

Sound familiar to you? If it does, you’re probably not using documentation strings, commonly known as doc strings or docstrings (I prefer docstrings, without the space), or any of the handy tools that work with docstrings. This article will introduce you to Python’s docstrings, and a few of the tools that make them a great addition to your code.

Docstrings

If you are not already using docstrings in your Python code, you really should. They provide a standard way to comment your code, giving you and other developers (who might want to use your code at some point) easy access to descriptions of the modules, classes, and functions found within.

At the heart of it, docstrings are simply comments placed in special locations in your Python source code. These comments can then be looked at by tools designed to work with docstrings or other Python programmers using your code. Note that I said using// your code, not reading your code. This is because docstrings are accessible via a Python object’s __doc__ attribute. This is very helpful is you are testing out a new module in Python’s interactive shell and really need to know what sort of parameters a certain function needs.

Pep 257 has a good definition of docstrings: “A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. Such a docstring becomes the __doc__ special attribute of that object.”[1] Definitions are nice but it might be easier to look at a quick example of some docstrings:

def add(x, y):
    """This is the add function's docstring."""
	return x + y

def subtract(x, y):
    """This is the subtract function's docstring.
    It is longer then the add functions, it goes
    on and on and on and on."""
    return x - y

Since docstrings are comments they can be in any format that you want, however since this is Python there are a few style guidelines that you should probably be aware of. You don’t have to follow these guidlines but if you do it will be easier for other Python programmers to understand and work with your code.

In general there are two types of docstrings: one-line docstrings and multi-line docstrings. The difference between the two should be fairly obvious: one-line docstrings are only one line in length and multi-line docstrings are more then one line in length. One-line docstrings and multi-line docstrings each have different style guidelines that will be explained in more detail in the following two sections.

One-Line Docstrings

Let’s look at a quick example of a one-line docstring for a simple function:

def add(x, y):
    """Return the sum of two numbers."""
    return x + y

In this example """Return the sum of two numbers.""" is the docstring of the add function. If you were to run the following:

print add.__doc__

The output would look like this:

Return the sum of two numbers.

In his “Python Style Guide” Guido van Rossum has a few notes on the preferred style of one line docstrings:

• Triple quotes are used even though the string fits on one line. This makes it easy to later expand it.
• The closing quotes are on the same line as the opening quotes. This looks better for one-liners.
• There’s no blank line either before or after the doc string.
• The doc string is a phrase ending in a period. It prescribes the function’s effect as a command (”Do this”, “Return that”), not as a description: e.g. don’t write “Returns the pathname …” [2]

One-line docstrings should only be used to document the simplest of cases. If what you are documenting does anything complicated, accepts input, or returns a value, it’s probably a good idea to use a multi-line docstring.

Multi-Line Docstrings

Multi-line docstrings should be used to document the majority of your modules, classes, functions, and methods. This is because most of what you are programming performs tasks more complicated then that which can fit into a single sentence summary. Multi-line docstrings should be used to documents the input, output, and complex behaviour of your objects. Like one-line docstrings, multi-line docstrings should start with a single sentence summary. After that there should be a blank line and then a more detailed description. The blank line separating the one line summary and the additional information is important as certain tools will use the blank line to separate the summary from the rest of the docstring. An example of a multi-line comment can be found in Listing 1.

Listing 1

def subtract(x, y):
    """Return the difference between two numbers.

    Arguments:
    x -- The minuend.
    y -- The subtrahend.

    Returns:
    A number, the difference between x and y (ie. x - y)

    """
    return x - y

What to Document

There are also style guidelines that dictate what you should include in your multi-line docstrings when documenting different sections of your code. These are general guidelines but following them will help ensure that your code is well documented and easily understood by other Python programmers. For more information on this please see Guido’s “Python Style Guide”.[2]

Modules – Document the module and provide a one line summary of everything that is exported by the module. (eg. classes, exceptions, and functions).

Classes – Summarize the functionality of the class. List all public methods and data members of the class. If there is any addition information needed to subclass the class, or if there is an additional interface for subclasses provide a description.

Functions and Methods – Summarize the functionality of the function and “document its arguments, return value(s), side effects, exceptions raised, and restrictions on when it can be called (all if applicable).”[3]

An example of all of these can be found in Listing 2.

Listing 2

#!/usr/bin/env python
"""A simple math module.

Exported Classes:

Math -- A simple math class with mathematical functions.

"""

class Math(object):
    """A simple math class with mathematical functions.

    Public functions:
    add -- Adds two numbers together and
    returns the result.

    subtract -- Returns the difference between
    two numbers.

    """

    def subtract(self, x, y):
        """Return the difference between two numbers.

        Arguments:
        x -- The minuend.
        y -- The subtrahend.

        Returns:
        A number, the difference between x and y (ie. x - y)

        """
        return x - y

    def add(self, x, y):
        """Return the sum of two numbers.

        Arguments:
        x -- Number to be summed.
        y -- Number to be summed.

        Returns:
        A number, the sum of x and y (ie. x + y)

        """
        return x + y

Documentation Generation

Generally if you have a complicated module or API people would rather read a help file, or online documentation, as opposed to constantly having find and browse the source code. Thankfully for us there are many different documentation generation tools out there that make use of docstrings. So when you are writing your docstrings, you aren’t just commenting your source code you’re also writing your help file!

The easiest tool to use is PyDoc, it’s a module, and a stand-alone application, that has been included in the Python standard library since version 2.1. There is much that can be done with PyDoc but for this column we are going to focus on it’s documentation generation. PyDoc can take the docstrings found within a module and output them as either simple text documentation (much like UNIX or Linux man pages) or HTML documentation.

Creating HTML documentation of the SimpleMath module used in Listing 2 is very easy. On a UNIX like computer (Linux, OS X, etc.) it can be accomplished using the following command:

$ pydoc -w /home/selsine/python/SimpleMath.py

On Windows the command will look something like this:

C:>c:Python25Libpydoc.py -w c:pythonSimpleMath.py

This will create an HTML file named SimpleMath.html in the current folder. A sample of what the generated HTML looks like can be seen in Figure 1.

Figure 1 - pydoc

While PyDoc is a great tool and easy to use because it is in the standard library, there are a few other tools out there that you might consider using. If you look at the HTML file that PyDoc generates you will notice that it does not mark up your docstrings. It simply reads them from your source code and spits them back out. If you are looking for something a little bit fancier with a few more options you might consider Epydoc[4] or docutils[5].

Both Epydoc and docutils use simple markup languages to give the documentation generated a bit more punch. Docutils uses the reStructuredText markup language. While Epydoc uses the Epytext markup language, as well as being able to work with Javadoc and reStructuredText. An example of the HTML that Epydoc produces can be seen in Figure 2. The code that was used to generate the HTML can be found in Listing 3.

Figure 2 – Epydoc

Listing 3

#!/usr/bin/env python
"""A simple math module.

Exported Classes:

Math -- A simple math class with mathematical functions.

"""

class Math(object):
    """A simple math class with mathematical functions.

    Public functions:
    add -- Adds two numbers together and
    returns the result.

    subtract -- Returns the difference between
    two numbers.

    """

    def subtract(self, x, y):
        """Return the difference between two numbers.

        @type   x: number
        @param  x: The minuend.
        @type   y: number
        @param  y: The subtrahend.

        @rtype: number
        @returns: A number, the difference between x and y (ie. x - y)

        """
        return x - y

    def add(self, x, y):
        """Return the sum of two numbers.

        @type   x: number
        @param  x: Number to be summed.
        @type   y: number
        @param  y: Number to be summed.

        @rtype: number
        @returns: A number, the sum of x and y (ie. x + y)

        """
        return x + y

def _test():
    import doctest
    doctest.testmod()

if __name__ == "__main__":
    _test()

Doctest

One of the most interesting uses of docstrings is unit testing using the doctest module. As we all know testing our code is important, and (as many of us have begun to learn) writing unit tests for large projects is a great way to ensure that changes in one area of a project don’t cause problems elsewhere in the code. If you don’t know what a unit test is it’s basically a simple test case to prove whether a specific area of your code is functioning properly. Generally at least one unit test is created for each object in order to test the correctness of the project as a whole.

In a nutshell the the doctest module searches your dostrings "for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown."[6] This means that it will search your docstrings for lines that start with >>> or with ... if they are the continuation of a statement (i.e. the inside of a function). If there is output generated by the statements it “must immediately follow the final ‘>>> ‘ or ‘… ‘ line containing the code, and … extends to the next ‘>>> ‘ or all-whitespace line.”[7] Adding doctests doesn’t just add unit tests to your code it also provides working examples in your docstrings and documentation.

Since doctests are formatted to look like interactive Python sessions a simple way to write them is to use Python’s interactive shell. You do this by simply executing your code in the interactive shell, and then copying and pasting the resulting test into your docstrings. For example, if we were to use this method to write a doctest for our subtract method we could do something like the following in the interactive shell:

>>> from SimpleMath import Math
>>> simple_math = Math()
>>> simple_math.subtract(10, 7)
3
>>>

We don’t need to import our module for the doctest so all we will need to copy from the interactive shell are the middle three lines:

>>> simple_math = Math()
>>> simple_math.subtract(10, 7)
3

As you can see, what we have here is a test compromising of two lines of Python code and then the expected result. The SimpleMath module containing a doctest for each function can be found in Listing 4.

Listing 4

#!/usr/bin/env python
"""A simple math module.

Exported Classes:

Math -- A simple math class with mathematical functions.

"""

class Math(object):
    """A simple math class with mathematical functions.

    Public functions:
    add -- Adds two numbers together and
    returns the result.

    subtract -- Returns the difference between
    two numbers.

    """

    def subtract(self, x, y):
        """Return the difference between two numbers.

        >>> simple_math = Math()
        >>> simple_math.subtract(10, 7)
        3

        @type   x: number
        @param  x: The minuend.
        @type   y: number
        @param  y: The subtrahend.

        @rtype: number
        @returns: A number, the difference between x and y (ie. x - y)

        """
        return x - y

    def add(self, x, y):
        """Return the sum of two numbers.

        >>> simple_math = Math()
        >>> simple_math.add(10, 7)
        17

        @type   x: number
        @param  x: Number to be summed.
        @type   y: number
        @param  y: Number to be summed.

        @rtype: number
        @returns: A number, the sum of x and y (ie. x + y)

        """
        return x + y

def _test():
    import doctest
    doctest.testmod()

if __name__ == "__main__":
    _test()

If you look at Listing 4 you will also notice the following code at the end of the source:

def _test():
    import doctest
    doctest.testmod()

if __name__ == "__main__":
    _test()

This is the code that will be executed if the module is launched directly from the command line. The code will import the doctest module and then use the testmod method to test the current module. Now when we run our SimpleMath.py file directly we will get the following:

$ python SimpleMath.py
$

Since nothing was written out to the command line we know that all of the doctests were successful. If you want to get more information you can use the -v flag to produce verbose output:

$ python SimpleMath.py -v

If you were to encounter an error it would look something like Listing 5.

Listing 5

**********************************************************************
File "SimpleMath.py", line 26, in __main__.Math.subtract
Failed example:
    simple_math.subtract(10, 7)
Expected:
    4
Got:
    3
**********************************************************************
1 items had failures:
   1 of   2 in __main__.Math.subtract
***Test Failed*** 1 failures.

As an added bonus if you are using Epydoc or docutils to generate your documentation, both tools will recognize doctest sections and highlight them. An example of what it looks like when Epydoc does this can be seen in Figure 3.

Figure 3 - doctest

Conclusion

Hopefully by this point you can see how useful docstrings can be to any Python code that you write. They give you a structured way to document your source code, the ability to easily generate great looking documentation, and a simple way to add unit tests to your code. But that's not all, as more and more people start using doctrings you can be sure that the list of docstring tools will continue to grow.

We all know that commenting source code and writing documentation are among the least enjoyable tasks a programmer can face. In fact the only thing worse then commenting and documenting is trying to use code with no comments and poor documentation! So do me, and yourself, a favour: start writing those docstrings.

[1] http://www.python.org/dev/peps/pep-0257/#what-is-a-docstring
[2] http://www.python.org/doc/essays/styleguide.html
[3] http://www.python.org/doc/essays/styleguide.html
[4] http://epydoc.sourceforge.net/
[5] http://docutils.sourceforge.net/
[6] http://docs.python.org/lib/module-doctest.html
[7] http://docs.python.org/lib/doctest-finding-examples.html

By: Mark Mruss

Note: This article was first published the January 2008 issue of Python Magazine

Iterators, iterables, and generators are features handled so wall by Python that people programming in other languages cannot help but drool over. Fortunately for us, creating iterators, iterables and generators is a relatively simple task. This article introduces the concepts of iterators, iterables, and generators and illustrates how easy it is to add them to your code.

1. Introduction
2. Iteration in Python
3. An Initial Example
4. Creating An Iterator
5. Looking More Closely At The Iterator
6. The Upside And Downside Of Iterators
7. Generators
8. Looking Closely At The Generator
9. But What About Iterables?
10. Creating An Iterable Object
11. Conclusion

Introduction

In this article I’m going to introduce three related Python features: iterators, iterables, and generators. Generators are easy to define, they are functions that create and return an iterator. Iterators and iterables on the other hand, are easier to use than they are to define. An iterable object is a “container object capable of returning its members one at a time.”[1] An iterator object is “An object representing a stream of data. Repeated calls to the iterator’s next() method return successive items in the stream. When no more data is available a StopIteration exception is raised instead. At this point, the iterator object is exhausted and any further calls to its next() method just raise StopIteration again.”[1] You can think of the difference between the two in this way: an iterable object can be iterated over multiple times, whereas an iterator object can only be iterated over once. In general an iterable produces an iterator every time something wants to iterate over its data.

Note: Classes that define the __getitem__ function are also considered iterables, but since that falls outside the scope of this article, it will not be covered here.

In this tutorial, I will begin by discussing iterators, the most basic concept. Then I will move onto generators, and finish by discussing iterables, the most wide open topic of the three.

Iteration in Python

Iterators objects are used in Python in order to iterate over an objects data. For example, we all know how to do this in Python when we work with lists:

my_list = [1,2,3]
for num in my_list:
	print num

This code will iterate over the list object my_list and print out all of the list items , i.e., the numbers 1, 2, and 3. Iterating over sequences in this simple and transparent manner happens to be one of my favourite features of Python.

According to our definition above, lists are iterables since you can iterate over them multiple times. In fact, each time you iterate over a list you are actually using a listiterator iterator object produced by the list.

It may not be immediately clear to you when you should add an iteration support to a class. However, the more you work with Python the more you’ll find instances when doing just this is very useful, sometimes the only advantage is cleaner looking code. One nice thing about iterators (and generators too) is that the processing for each item happens as you need it. Instead of collecting all of the data into a list and then running through the list, you will collect each item as you need it. This might not seem like a large difference but imagine if there were tens of thousands of items to process? What if you were collection your data from an online source? Performing all of your processing up front may take a very long time, especially if you only wanted the first few items.

An initial example

In order to explain iteration further, let’s look at a simple example task where we might use iterators. For this example we will create a class that takes a string of characters as input and then converts each character into its byte value. If we were NOT going to use iterators we might do something like what is found in Listing 1.

Listing 1

class ByteValue(object):

	def __init__(self, data):
		self.data = data

	def to_bytes(self):
		bytes = []
		for char in self.data:
			bytes.append(ord(char))
		return bytes

This code is pretty simple. We have a data member, named data, that we use to store the string that was used to initialize the class. In the to_bytes function we loop through the string, converting each character to its byte value using the built in ord function. We store each byte value in a list and once we have collected all of the values we return that list.

When we run the following:

bv = ByteValue("abcdef")
for byte in bv.to_bytes():
	print byte

we would get this as our output:

97
98
99
100
101
102

Creating an Iterator

Let’s convert this into an iterator. Making your class into an iterator requires adding “two methods, which together form the iterator protocol.”[2] The two functions needed are: 1) the __iter__ function; and, 2) the next function. The __iter__ function will return the object itself, while the next function will return the next item. The next function is where the actual iteration work occurs. The next function iterates by returning the next item in the “sequence” each time it is called for as long as there is a “next” item. When there are no more items to iterate over, the next function must raise the StopIteration exception to halt the iteration.

To be clear, in order to make your class an iterator you need to do two things:

1) Add an __iter__ function that returns the object itself (self)

2) Add a next function that returns the next item in the sequence each time it is called. When there are no more items in the sequence, the next function raises a StopIteration exception signal the end of the iteration.

For those of you still confused, the following example will help illustrate how iterators work. If we were to convert our ByteValue class into an iterator object, it might look something like Listing 2.

Listing 2

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0

	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value

Let’s compare the code in Listings 1 and Listing 2 in detail, focusing on the iterator in Listing 2. The first difference is the addition of the data member current_item to the class, initialized in the __init__ function. current_item serves as a counter and keeps track of the current character in the string while we iterate over it. The counter must have class scope since the iterator works through successive calls to the next function. If current_item were local to the next function, its value would be reset with each subsequent call, and would not be of much use.

The second difference between the listings is the addition in Listing 2, of the __iter__ function where we return self.

The final addition to Listing 2 is the next function, where we first check to see if current_item is equal to the length of our string. If current_item is equal to the strings length we raise the StopIteration exception to signal the end of the iteration because we have no more characters left to iterate over. If there are more characters to iterate over we calculate the byte value of the current character, increase our current_item counter, and then return the byte value.

Note: Notice that current_item is only initialized when the ByteValue object is created. This happens because according to the iteration protocol, “once an iterator’s next() method raises StopIteration, it will continue to do so on subsequent calls.”[2] If we were to re-initialize current_item we would then be able to iterate over the iterator more than once breaking the iteration protocol.

Now that we have converted our class into an iterator we can use it as follows:

for byte in ByteValue("abcdef"):
	print byte

Doing so would result in:

97
98
99
100
101
102

Notice that we do not store the instance of our ByteValue class in a variable. Doing so would be useless because since ByteValue is an iterator it is only good for one pass of the data. If ByteValue were an iterable (returning an iterator object when __iter__ was called) it would make sense to keep an instance around because we could iterate over the instance more than once. We will look at creating iterables later on in this article.

Looking more closely at the Iterator

Let’s look at what is happening in more detail by examining what is happening behind the scenes during the iteration process. In order to illustrate what is happening in the for loop I will demonstrate the order in which things are being called behind the scenes.

The first step in the iteration process is to call to the __iter__ function in order to get the iterator object that will perform the iteration. Notice that this works on iterator and iterable objects, since iterators returns themselves and iterables return an iterator.

bv = ByteValue("abcdef")
iterator = bv.__iter__()

Now that we have the iterator object, we start iterating by calling the next function in order to get the next value:

print iterator.next()

This executes the next function which will return the byte value of character in the string with which we are currently working. Since this is the first call to the next function current_item will be zero and we will calculate the byte value of the first character in our string (’a') resulting in:

97

If we continue the iteration process by calling the next function six more times we would get the results shown in Listing 3. Notice that we have now made a total of seven calls to the next function, one more then the number of characters in our string. I’m running the python code from a file (iter.py found in Listing 4). Depending on how you are running it, you might get slightly different results. However, the most important thing to observe in this example is the exception that is raised.

Listing 3

97
98
99
100
101
102
Traceback (most recent call last):
  File "Listing4.py", line 34, in ?
    main()
  File "Listing4.py", line 30, in main
    print iterator.next()
  File "Listing4.py", line 13, in next
    raise StopIteration
StopIteration

Listing 4

#!/usr/bin/env python

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0
	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value
		return self.data

def main():
    for v in ByteValue("abc"):
        if v in ByteValue("abc"):
            print "We have a %d" % v

    bv = ByteValue("abcdef")
    iterator = bv.__iter__()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()

if __name__ == "__main__":
	# Someone is launching this directly
	main()

For the most part you won’t be calling an iterator object’s __iter__ or next functions manually, instead you’ll probably just let the for loop do it all for you.

The upside and downside of Iterators

Now that our class is an iterator object we can use any of the built-in functions and methods that work on iterators and iterables, such as: the sum, tuple, sorted, and list functions, to name a few.

In the first example the bytes function returned a list object that we could work with. If we want a list instead of an iterator for any reason, it will be as simple as using the list function, which takes an iterator as a parameter and returns a list:

list(ByteValue("abcdef"))

If we want a sorted list:

sorted(ByteValue("abcdef"))

If we want the sum of all of the bytes:

sum(ByteValue("abcdef"))

While very useful, iterators do have some downsides to them. The most obvious is that they are only good for one pass over the data. They also generally require you to add extra data members to your class in order to keep track of your current iterator position. Depending on what you are iterating over, this process can become quite complex. Iterators also only allow you to perform one “type” of iteration, i.e., in one direction or over one piece of internal data. You might address this by adding flags to your class but this will further clutter the class and decrease readability. So how can you simply add multiple types of iteration to a class? Enter the generator!

Generators

A generator is a function that creates, or generates, an iterator. In order for a function to become a generator it must return a value using the yield keyword.

Generators are interesting because they are functions, yet execution does not run through them as it does in a normal function. The first time execution enters a generator function it will start at the beginning of the function and continue until the yield keyword is encountered. When the iteration continues, execution will continue in the generator function on the statement immediately following the yield keyword. All local variables in the function will remain intact. If the yield statement occurs within a loop, execution will continue within the loop as though execution had not been interrupted.

Continuing with the ByteValue example, let’s add a generator function named reverse that can be used to iterate through the byte vales of the string in reverse order:

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		yield ord(self.data[current_item])

So what’s going on here? The first thing to notice is that we did not add anymore data members to our class, this generator is a self-contained unit. Secondly, since this is a generator, our counter current_item can be a local variable.

In the reverse function the first step is to initialize current_item, which represents the current character in the string, to be equal to the length our string. We initialize it to the length of the string instead of zero since we are iterating through the string in reverse. Next we have a while loop that loops while current_item is greater than zero. We then subtract one from our counter, to give us the current character to process. Finally, we yield the byte value of the current character.

Note: We subtract one from our counter the first time through the loop because Python is zero-based and the length of a list minus one gives us the position of the last item in the list. In our examples we have used the following string:

abcdef
012345

When we calculate the length of the string we get 6. We then subtract 1 from that number, leaving 5, which is the index of the last number in the string.

Making use of our new generator function, we run the following:

bv = ByteValue("abcdef")
for byte in bv.reverse():
	print byte

We get our favourite byte values in reverse:

102
101
100
99
98
97

Looking closely at the Generator

Let’s take a detailed look at what is happening in the generator in the same way that we did earlier with the iterator object. The first thing that happens when you call a generator function is NOT the execution of the actual function, rather, it is the creation of a generator object. Running the following:

bv = ByteValue("abcdef")
gen = bv.reverse()
print gen

will result in:

<generator object at 0xb7d8e04c>

This demonstrates that, as stated above, the first call to our generator function does not return the byte value of that last character in the string, instead it creates a generator object. In fact the first time a generator function is called, the actual function is not executed at all. A generator object is “what Python uses to implement generator iterators. They are normally created by iterating over a function that yields values”.[3]

Once we have a generator object we can start calling its next function (sound familiar?) to perform the action iteration. An example of this can be found in Listing 5. The results of this execution can be found in Listing 6. The results will seem very familiar to you, especially the StopIteration exception.

Listing 5

#!/usr/bin/env python

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0
	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value
		return self.data

	def reverse(self):
		current_item = len(self.data)
		while (current_item > 0):
			current_item -= 1
			yield ord(self.data[current_item])

def main():

	bv = ByteValue("abcdef")
	gen = bv.reverse()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()

if __name__ == "__main__":
	# Someone is launching this directly
	main()

Listing 6

102
101
100
99
98
97
Traceback (most recent call last):
  File "Listing5.py", line 40, in ?
    main()
  File "Listing5.py", line 36, in main
    print gen.next()
StopIteration

If we want to look at the contents of the generator object we could use the following code:

print dir(gen)

And getting the following results:

['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'gi_frame', 'gi_running', 'next']

Notice that the generator object contains the __iter__ and next functions making the generator object itself an iterator. Because the generator itself is an iterator object, the same built-in iterator functions I mentioned earlier can also be used on our generators:

print sum(bv.reverse())

It is also important to remember that your generator or iterator does not have to perform only “dumb” iteratation, simply moving through some sort of an internal list. Rather, it can make decisions just like any other block of code.

For example, let’s say that in our reverse generator we want to use the byte value 99 as an end condition. We can do something similar to the example found in Listing 7.

Listing 7

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		value = ord(self.data[current_item])
		if (value == 99):
			return
		yield value

In Listing 7 if the byte value equals 99 we return from our generator function using the return keyword. Since we didn’t yield anything this will cause the StopIteration exception to be fired halting the iteration.

Be careful about getting too smart with your iteration because you cannot return any information from a generator you can only yield it. So if you tried to execute the code in Listing 8, in an attempt to return one last value before quitting (or if you wanted to return a success or failure code) you will get the following error when the return value line of code is executed:

SyntaxError: 'return' with argument inside generator

Listing 8

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		value = ord(self.data[current_item])
		if (value == 99):
			return value
		yield value

But what about iterables?

I’m sure that by now you are wondering about the iterable objects that I mentioned at the start of this tutorial. As you have probably guessed, simply making your class an iterator isn’t that useful unless it is only performing one task like our ByteValue example. Since your classes will generally be performing more than one task, you will likely want to make your class an iterable object rather than an iterator object. To recap, iterable objects return an iterator object when their __iter__ function is called, which allows for multiple passes over their data.

Creating an Iterable object

Since the definition of an iterable object is an object that returns an iterator object when its __iter__ function is called, creating an iterable object can be done in a variety of ways. Two options that come to mind are: 1)creating an iterator helper class to perform the iteration and, 2) using a generator function. I prefer using a generator function since it keeps the functionality within the main class.

See Listing 9 for an example of creating an iterarable object using a generator. You will see that we have replaced the next function with the forward function. The forward function is a generator that iterates through the data in the “forward” direction. In the __iter__ function we return the results of a call to the forward function, a generator object. Since generator objects contain the iterator protocol and are, in fact iterators, by returning one from our __init__ function we have successfully created an iterable.

Listing 9

class ByteValue(object):

	def __init__(self, data):
		self.data = data

	def __iter__(self):
		#We are an iterable, so return our iterator
		return self.forward()

	def forward(self):
		#The forward generator
		current_item = 0
		while (current_item < len(self.data)):
			byte_value = ord(self.data[current_item])
			current_item += 1
			yield byte_value

	def reverse(self):
		#The reverse generator
		current_item = len(self.data)
		while (current_item > 0):
			current_item -= 1
			yield ord(self.data[current_item])

def main():
    bv = ByteValue("abc")
    for v in bv:
        if v in bv:
            print "We have a %d" % v

if __name__ == "__main__":
    main()

Now that we have an iterable object, we can iterate over it as many times as we want. We can even have fun with nested iteration:

bv = ByteValue("abcdef")
for value in bv:
	print value
	for second_value in bv:
		print second_value

Conclusion

This concludes our introduction to iterators, iterables, and generators. I hope I have demonstrated the immense power and flexibility that they provide. In general, making your object an iterable object and/or using generators allows for more flexibility than simple iterator objects provide. For most complex classes or complex sets of data, multiple iterations are a given.

With all of the praise that I have heaped upon iterators, iterables, and generators, it’s important to remember that they are not a panacea and should not be used in every case where a sequence of items is needed. There are many instances where a returning a list is the desired result. This being said, iterators, iterables, and generators are extremely useful and provide a great way to loop through data.

[1] http://docs.python.org/tut/node18.html
[2] http://docs.python.org/lib/typeiter.html
[3] http://docs.python.org/api/gen-objects.html