Hey everyone, just a quick note to let you know that Bohdan has translated the Creating a GUI using PyGTK and Glade tutorial in to Belorussian/Belarusian. You can view the translation here: Belorussian translation.

Many thanks to Bohdan and all the other who have translated some of these tutorials over the years.

Note: This article was first published the June 2008 issue of Python Magazine

Mark Mruss

With the release of Python 3.0 only a few months away many Python programmers have visions of compatibility problems dancing in their heads. This article will introduce the concept of future statements including two future statements that you can use to help prepare your code for version 3.0.

Introduction

Python 3.0 (or Python 3000 as many people know it) is something that many Python programmers have started thinking about. Besides being the next major version of our beloved programming language, version 3.0 of Python will break backwards compatibility with the current 2.0 branch of Python. This means that a some of the code that you are writing right now in Python 2.X won’t immediately work in Python 3.0. Since Python 3.0 has a scheduled release date of September 2008 [1] now may be a good time to start thinking about future migration.

Of course there’s no reason to be alarmed yet. Guido, himself, has said that there is no rush to switch over to Python 3.0. [2] According to his PyCon 2008 essay “Python 3000 and You” you should switch when the following are true: “1. You’re ready 2. All your dependencies have been ported.” [3]

In the same essay Guido also says that Python programmers should be prepared and that they should “start writing future-proof” code for 2.5″ [4] In this sprit, this article will introduce “future statements” and two ways that you can use them now in order to make the migration process from the Python 2.0 branch to the Python 3.0 branch as smooth as possible.

The rest of this article, and the examples within, will assume that you are working with Python 2.5.

Import the Future

A very important step in getting your code ready for Python 3.0 is to make use of the __future__ module. The __future__ module allows you to both import changes that are going to be made in future version of Python and use them in your current version of Python.

For example, if feature X is slated to change to feature Y in a future version of Python, there is a chance that it may be supported by the __future__ module. If feature Y is supported by the __future__ module, you can import feature Y from the __future__ module in the version of Python that is still using feature X. This means that instead of having to alter your code when feature X is deprecated and feature Y is implemented, you can write your code now using feature Y and not worry about when the switch will occur.

Future Statements

When you use the __future__ module you will be creating future statements. Future statements, and the purpose of the __future__ module are defined quite well in the Python documentation:

“A future statement is a directive to the compiler that a particular module should be compiled using syntax or semantics that will be available in a specified future release of Python. The future statement is intended to ease migration to future versions of Python that introduce incompatible changes to the language. It allows use of the new features on a per-module basis before the release in which the feature becomes standard.” [5]

Future statements are what will import the new features from the __future__ module. Future statements must be at the top of your module; only comments, blank lines, or other future statements can be located before them.

Let’s take a look at a future statement that will let us work with the changes that will happen to the division operation (explained later):

from __future__ import division

This will change the implementation of the division operation, in whatever module the future statement occurs, from its current integer form to its future float form.

Floating Division

Now that we know what a future statement is and how to use the __future__ module, let’s look at the changes that will be made to the division operation. In the current branch of Python the division operation can be somewhat confusing. It can be integer division (returning the floor of the division, i.e. only the integer quotient) or floating point division (returning the result as a float including the remainder).

Confusion within the current state of Python’s division operation can be seen in the following example:

print 3/2
print 3/2.0

This results in:

1
1.5


Here Python is looking at the type of the operands in the operation. If they are both of type int, integer division will be performed as shown in the first result. If one (or both) of the values is a float, the operation will return a float that will include the remainder (if there is one).

Note that division of this sort (where the result type is dependent on the operands) is common in other programming languages.

While the correct type of division to be performed may be obvious when literal numbers are being used, unexpected results can occur when you are working with variables whose type may be unknown:

def divide_by_three(value):
    return 3/value

This will result in both integer and float return values depending on the type of ‘value’.

As a result of this confusion, in Python 3.0 changes are being made to what the division operation returns. All division using a single slash (/) will return a floating point number including the remainder. In order to perform integer division, you will need to use two slashes (//). The type of division performed now depends on the number of slashes used and not the type of the operands. The only gotcha to be aware of is that if one of the operands in the division operation is a float, the return value will also be a float with 0 as the remainder.

Listing 1 uses a future statement to import the changes to the division operation and replicates our previous division test. If you were to run the code found in Listing 1 the output would be as follows:

1.5
1.0


If you are writing code where you perform division of any sort, I’d suggest importing division from __future__ so that when the move to Python 3.0 occurs, you won’t find yourself with any strange results.

Note that the // division operation has been in Python since version 2.2 so you can already use it without importing division from __future__. Importing division from __future__ changes the functionality of the / division operation.

The future of Imports

In Python 3.0 all imports will be absolute. This means that when you perform an import, you will import “a module or package reachable from sys.path.”[6] While at first glance this may seem to make importing local modules impossible, it is important to remember that when you launch a Python script file directly, the directory where that script file is located will be added to the start of sys.path in position 0.

To illustrate the reason for the change to absolute imports, let’s take a look at a silly example. Let’s say we had a project that used a tokenizer package using the directory tree shown in Figure 1 where string.py is a helper module (with an unfortunate name) that the parse.py module needs to use.

In this example my_proj.py won’t do anything besides import the tokenizer package and create an instance of a Parse object:

import tokenizer

if __name__ == '__main__':
    parser = tokenizer.Parse()

The __init__.py modules in the tokenizer package imports the Parse class from the parse module. This is done so that the Parse class can be accessed as tokenizer.Parse as opposed to tokenizer.parse.Parse:

from parse import Parse

Finally, parse.py is where all of the magic happens:

import string

class Parse(object):
    def __init__(self):
        print(string)

Well it’s not really magic! All that the parse.py file does is import the string module. When a Parse instance is created, a string representation of the string module that was imported is printed out to the command line.

In other words, what happens here is:

1. my_proj.py imports the tokenizer package, which then imports Parse from parse.py.
2. my_project.py then creates an instance of the Parse object.
3. The Parse instance will then print out a string representation of the string module it imported.

When we run my_proj.py, we get the following:

<module 'tokenizer.string' from '/home/project/tokenizer/string.py'>
</module>

As we can see, our local string module was imported by the parse.py module. This is a relative import which means that the “import statement first looks in the containing package before looking in the standard module search path.” [7]

So What’s the Problem?

I’m sure that many of you realize that there is a string module built into Python, and that our string module and the built-in string module share the same name. As long as everything in the tokenizer package only needs the tokenizer.string module, and not the built-in string module, everything will be fine.

But what happens a few months down the road when some new module in the tokenizer package needs to import the built-in string module? Whenever it tries to import string it gets tokenizer.string instead. The obvious solution is to make sure that your module does not share the same name as a built-in module. But this does not fully solve the problem, as we can’t anticipate the names of future modules that will be added to the standard distribution.

Absolute Imports

PEP 328 describes the issue explained in our tokenizer example and the solution that the Python team has put in place:

“In Python 2.4 and earlier, if you’re reading a module located inside a package, it is not clear whether import foo refers to a top-level module or to another module inside the package. As Python’s library expands, more and more existing package internal modules suddenly shadow standard library modules by accident. It’s a particularly difficult problem inside packages because there’s no way to specify which module is meant. To resolve the ambiguity, it is proposed that foo will always be a module or package reachable from sys.path. This is called an absolute
import.” [8]

As stated earlier, once Python gets to version 3.0 all import statements will be absolute imports. We can already use this functionality in Python 2.5 by using a future statement to import absolute_import from the __future__ module:

from __future__ import absolute_import

When we add a future statement to parse.py that imports the absolute_import feature, it will look like the following:

from __future__ import absolute_import
import string

class Parse(object):
    def __init__(self):
        print(string)

When we run my_proj.py, the output will be something similar to the following:

<module 'string' from '/usr/lib/python2.5/string.pyc'></module>

As you can see, enabling absolute imports changes import string from a relative import into an absolute import. This solves our problem when we want import string to import the built-in module, but what about when we want to import the local string module in the tokenizer package? The solution is to use an absolute import to import the local module. The following version of parse.py will import the local string module:

from __future__ import absolute_import
from tokenizer import string

class Parse(object):
    def __init__(self):
        print(string)

We can also use the following absolute import to import the local string module:

from __future__ import absolute_import
import tokenizer.string

class Parse(object):
    def __init__(self):
        print(tokenizer.string)

Relative Imports in an Absolute World

Once you start importing the absolute_import feature, you may still want to use relative imports. Relative imports generally are not needed in smaller packages. But if you have a very large package that contains many sub-packages, you might not want to rewrite your import statements every time the internal structure of the main package changes. Fortunately, it is still possible to perform relative imports in Python 3.0 or with absolute_imports enabled. If you want to perform a relative import, you will need to use the following syntax:

from . import foo

Notice that a dot is used to indicate that the import is a relative import. “A single leading dot indicates a relative import, starting with the current package. Two or more leading dots give a relative import to the parent(s) of the current package, one level per dot after the first.”[9] The “dot syntax” is only available to imports that use the “from” syntax. “import . foo” is not allowed.

To import our local string module in the parse.py module with absolute imports turned on, we do the following:

from __future__ import absolute_import
from . import string

class Parse(object):
    def __init__(self):
        print(string)

Conclusion

I hope I have convinced you that importing division and absolute_imports from the __future__ module is an easy way to add a little “3.0 protection” to your current Python projects. Deciding to make these changes as you create new code, will make migrating your code to Python 3.0 an easier process when the big day comes.

Of course these are not the only changes that you should start making to your code to prepare for Python 3.0. (For example, some of you may have noticed that I was using the print function in this article and not the print statement. Since the print statement is no longer going to be available in Python 3.0, I’m trying to get used to using the print function.) If you want a more detailed description of all the upcoming changes to Python 3.0, and steps you can take to prepare your code now, please see the Python wiki. [10]

You don’t have to worry about Python 3.0 too much right now since the true migration path will be through Python 2.6 and the tools that will be made available. In fact for some people full migration to Python 3.0 may be many years away. But like Guido said it’s best to be prepared and start writing future-proof code now. [11]

[1] http://www.python.org/dev/peps/pep-0361/
[2] http://www.python.org/doc/essays/ppt/pycon2008/Py3kAndYou.pdf
[3] http://www.python.org/doc/essays/ppt/pycon2008/Py3kAndYou.pdf
[4] http://www.python.org/doc/essays/ppt/pycon2008/Py3kAndYou.pdf
[5] http://docs.python.org/ref/future.html
[6] http://www.python.org/dev/peps/pep-0328/#rationale-for-absolute-imports
[7] http://docs.python.org/tut/node8.html#SECTION008420000000000000000
[8] http://www.python.org/dev/peps/pep-0328/#rationale-for-absolute-imports
[9] http://www.python.org/dev/peps/pep-0328/#guido-s-decision
[10] http://wiki.python.org/moin/FutureProofPython
[11] http://www.python.org/doc/essays/ppt/pycon2008/Py3kAndYou.pdf

The results are in for the first annual LearningPython.com Python version quiz:

By a landslide version 2.6 is the winner, with 3.1 and 2.5 following far behind. A grand total of 1084 people voted and 700 of those still use Python 2.6, 187 use 3.1 and 182 use 2.5. While not the largest sampling of users the wide margin of victory probably means that most Python programmers are still targeting 2.6, or at the very least the newest version of the 2.x branch. Now what is installed on our end user’s system…that’s another matter all together.

Thanks again to everyone that voted. I’m happy that I was able to attract 1000+ people to vote in this poll. My next idea for a poll is an IDE poll, partially because I’m curious as to what people are using, and partially because there is a good chance that I may have overlooked other programs on my way to selecting Geany.

Note: Python 2.7 was released (July 3rd, 2010) after the poll was created (Mach 4, 2010) this means that the value for 2.7 is probably larger now and 2.6 is probably slightly smaller.

Note: This article was first published the May 2008 issue of Python Magazine

Introducing Descriptors and Properties

Mark Mruss

New-style classes were introduced to Python with the release of Python 2.2. And with these new-style classes came descriptors and properties. This article will introduce the descriptor protocol, descriptors, and properties.

Introduction

New-style classes were introduced to Python with the release of Python 2.2. A new-style class is any class that is derived from the object base class. New-style classes give Python programmers many new (and initially confusing) features. One such feature is the descriptor protocol, and more specifically descriptors themselves.

Descriptors give Python programmers the ability to easily and efficiently create “managed attributes”. Managed attributes can be thought of as attributes that are not accessed directly. Instead their access is “managed” by something else, generally a class or a function.

If you haven’t come across this before you are probably wondering why one would want to manage attribute access? One reason might be that you don’t want people to be able to delete the attribute. Another reason may be that you need to ensure that your attribute data is always valid. Or perhaps attribute x is based on attribute y, so every time the value of y changes you want to update the value of x. From these few examples you can see the many possible cases where you might want to control access to certain attributes.

For those of you familiar with other programming languages, this type of access is often referred to as “getters and setters”. In many language, implementing “getters and setters” means using private variables and public functions that get and set the variable’s value. Since Python doesn’t (really) have private variables, the descriptor protocol is basically a built-in and Python-ic way to way to achieve something similar.

This article will introduce you to the descriptor protocol, descriptors, and properties. It will focus on demonstrating how to use them to create managed attributes. Since the descriptor protocol requires new-style classes, all of the examples in this article require Python 2.2 or newer.

A few definitions

Before moving forward, it is important to understand a few related terms. These terms will introduce some basic concepts and help you follow along with the remainder of the article.

descriptor protocol – The following three methods make up the descriptor protocol: __get__, __set__, and __delete__.

descriptor – An “object attribute with binding behavior, one whose attribute access has been overridden by methods in the descriptor protocol.” [1] In other words an “attributes whose usage resembles attribute access, but whose implementation uses method calls.”[2]

data descriptor – A descriptor with the __get__ and __set__ methods of the descriptor protocol defined.

non-data descriptor – A descriptor with only the __get__ method of the descriptor protocol defined. “Python methods (including staticmethod() and classmethod()) are implemented as non-data descriptors.” [3]

property – A built-in type that implements the descriptor protocol and allows you to easily create data descriptors.

Don’t worry if you don’t fully understand these definitions, the remainder of this article will hopefully clarify any confusion you have.

The Descriptor Protocol

Let’s take a closer look at the descriptor protocol and see how we can use it to create a descriptor. As previously mentioned, the descriptor protocol is made up of three methods: __get__, __set__, and __delete__. These methods have specific signatures and they are as follows, where self is the class that owns the methods:

__get__(self, instance, owner)
__set__(self, instance, value)
__delete__(self, instance)

These three methods represent the three basic operations that you perform on attributes in general: querying the value of that attribute; assigning a value to it; and, (very rarely) deleting it. They work as follows:

• The __get__ method is called when the attributes value is being queried. The __get__ method should return the (computed) attribute value or raise an AttributeError exception.” [4] This is where access to the attribute’s value is managed.
• The __set__ method is used in the assignment operation. It is called when we want to set the attribute value. This is where you can control what values, or types of values are being assigned to your attribute.
• Finally, the __delete__ method is called when we want to delete the attribute. Here you can (rarely) decide whether or not to delete the attribute.

There are also three different parameters passed to the three methods (excluding the standard self parameter for methods that belong to a class):

• owner – This “is always the owner class.” [5] This means that it is the actual class, and not an instance of the class. So if the descriptor is in a class called MyClass, owner will be that class.
• instance – An instance of class type owner. It is “the instance that the attribute was accessed through” [6], or None if the attribute is being accessed through the class (owner) instead of an instance.
• value- The value that the attribute is being set to.

This difference between owner and instance might be a bit confusing, so let’s look at a quick example. Let’s say we have a descriptor my_descriptor in the class MyClass, if we were to run the following code:

my_class_instance = MyClass()
print my_class_instance.my_descriptor

The second line queries the descriptors value and results in the __get__ method being called with the instance parameter being my_class_instance. The owner parameter will be the MyClass class.

Note: Notice that we treat the descriptor my_descriptor as though it is a normal attribute. We don’t call print my_class_instance.my_descriptor.__get__(my_class, MyClass). This is what was meant by: “attributes whose usage resembles attribute access, but whose implementation uses method calls.”[7]

If the following code were run:

print MyClass.my_descriptor

The __get__ method will again be called. This time the instance parameter will be None and the owner parameter will be the MyClass class.

Note: Only the __get__ method has the owner parameter. This means that it is the only function in the descriptor protocol that can be accessed through the class. Setting and deleting the descriptor through the class actually changes what the variable is. For example, if we tried to assign the numeric value 2 to a descriptor variable using the class, we would not access the descriptors __set__ method. Instead we would change the type of the variable from a descriptor to an integer with the value of 2.

A Simple Descriptor Example

A simple “transparent” descriptor example can be found in Listing 1. The first thing to notice in the code is that both the SimpleDescriptor and MyClass classes are “new-style” classes because they are derived from object. This is important because, as mentioned above, descriptors only work with “new-style” classes. The second point to notice is that the descriptor has class scope as opposed to instance scope. There are also two extra print statements included in the code. They are there to let us follow the execution a little more easily.

Listing 1

class SimpleDescriptor(object):

    def __get__(self, instance, owner):
        # Check if the value has been set
        if (not hasattr(self, "_value")):
            raise AttributeError
        print "Getting value: %s" % self._value
        return self._value

    def __set__(self, instance, value):
        print "Setting to %s" % value
        self._value = value

    def __delete__(self, instance):
        del(self._value)

class MyClass(object):
    my_value = SimpleDescriptor()

Using Listing 1 to execute the following code:

my_instance = MyClass()
my_instance.my_value = 416
print my_instance.my_value

The output would be the following:

Setting to 416
Getting value: 416
416

As you can see the second line (my_instance.data_descriptor = 416) calls the __set__ method and sets the _value attribute. When we call print my_instance.data_descriptor the __get__, method is called and the _value attribute is returned.

The Problem with the Simple Example

The previous example may look like a perfectly good descriptor, but there is something wrong with it. Take a look at what happens when we runs this new code:

my_instance = MyClass() #Create the first instance
my_instance.my_value = 416 #Set its' value
my_second_instance = MyClass() #Create the second instance
my_second_instance.my_value = 204 #Set its' value
print my_instance.my_value #What was the fist instance's value?

We get the following output:

Setting to 416
Setting to 204
Getting value: 204
204

Notice that when we set the second instance’s my_value descriptor to be 204, we are also setting the first instances. This is because my_value has class scope, so both instances (and the class itself) share the same instance of the SimpleDescriptor class. Since SimpleDescriptor only stores one value, they all actually share the same value. We will get the same results if we check what the classes value of my_value is:

my_instance = MyClass() #Create the first instance
my_instance.my_value = 416 #Set its' value
my_second_instance = MyClass() #Create the second instance
my_second_instance.my_value = 204 #Set its' value
print MyClass.my_value #What's the classes value?

We get the following results:

Setting to 416
Setting to 204
Getting value: 204
204

In the last line of the code (print MyClass.my_value #What's the classes value?) we simply use the class (ignoring both instances) in order to get the value of my_data. This will be an instance where the __get__ function will be called with the instance parameter set to None.

Note: This is not a problem if you want to share the value across all instances.

The Solution to the Problem

In order to get around this issue, you have to remember that if you want values unique to each instance your descriptors must store values that are unique to each instance. This can be in the instance itself, in dictionary in the descriptor, or perhaps in a text file. Just make sure that the value is unique to each instance. Though it seems like a simple solution, in practice a suitable solution for all cases is difficult to implement. As a result you should probably pick a specific implementation for your specific situations.

Using each instance as a key, one can store the value in a dictionary in the descriptor itself. The problem with this solution should be obvious to programmers familiar with Python dictionaries: only immutable types can be used as keys. This is fine if you know what object you are working with, but what about in the future when you want to add a descriptor to your sub-classed list?

Another solution is to store the value in the instance itself. You can do this by easily adding the value to the instance’s __dict__. The limitation is that the descriptor need to be given a suitable key so that there is no collision with anything already in the instance’s __dict__.

Listing 2 shows a solution to the problem from the previous section where the value is stored in the instance’s __dict__. Values are indexed using a key name provided during the creation of the descriptor. Aside from where we store the value, there is little difference between Listing 1 and Listing 2. Notice that the value name is a converted into a string. This is done to ensure that it can be used as a key.

Listing 2

class FixedDescriptor(object):

def __init__(self, value_name):
    self.value_name = str(value_name)

def __get__(self, instance, owner):
    if (instance is None):
        raise AttributeError
    elif (not instance.__dict__.has_key(self.value_name)):
        raise AttributeError
    return instance.__dict__[self.value_name]

    def __set__(self, instance, value):
        print "Setting to %s" % value
        instance.__dict__[self.value_name] = value

    def __delete__(self, instance):
        if (instance.__dict__.has_key(self.value_name)):
            del(instance.__dict__[self.value_name])


class MyClass(object):
    my_value = FixedDescriptor("__value")

The major usage difference between Listing 1 and Listing 2 is the fact that the __get__ method no longer works at class level; it only works at instance level. This should be obvious since this solution stores the value in the instance. If there is no instance, we have nowhere to store the value! If you attempt to access the descriptor at class level (the first if statement) an attribute error will be raised:

def __get__(self, instance, owner):
    if (instance is None):
        raise AttributeError
    elif (not instance.__dict__.has_key(self.value_name)):
        raise AttributeError
    return instance.__dict__[self.value_name]

Easy Data Descriptors with Properties

The descriptor provided in Listing 2 will work for many situations but for my money the easiest way to implement data descriptors is to use the property type. I would recommend using it unless you need the same descriptor across many different classes or attributes (i.e. for type validation). Properties can be thought of as a simple and easy way to create data descriptors. The property type implements the descriptor protocol and gets around the problem of where to store the descriptor value in an easy way: it lets the class that created the descriptor deal with it.

The full signature of the property function is as follows:

property(fget=None, fset=None, fdel=None, doc=None)

Where fget, fset, fdel are methods that will be called when the __get__, __set__, and __del__ members of the descriptor protocol are called. The doc parameter is a string that will be used as the docstring for the descriptor. If the doc parameter is not specified, the docstring of the fget method is used.

The signature of the fget, fset, fdel functions are as follows:

fget(instance)
fset(instance, value)
fdel(instance)

In the three signatures, instance is a reference to the object that owns the property attribute. This is the same instance that gets passed to the descriptor protocol. Since instance is the first parameter of each method, these are, for all intents and purposes, member methods of a class. The value parameter is the same as the value parameter that is passed to the __set__ method. It is what we are setting the attribute to.

The basic way to implement properties can be seen in Listing 3. As you can see, what we do for a property is very similar to what we did for our initial descriptor in Listing 1. We set up the three functions necessary for the descriptor protocol, and then use them to create our property attribute. Creating the property is very simple:

Listing 3

class MyClass(object):
    #Create the fget, fset, and fdel methods
    def __get_value(self):
        print "Getting value: %s" % self.__value
        return self.__value

    def __set_value(self, value):
        print "Setting to %s" % value
        self.__value = value

    def __del_value(self):
        del(self.__value)

    #Create the property
    my_value = property(__get_value
            , __set_value
            , __del_value
            , "This is my property")

    def __init__(self, value=0):
        self.my_value = value

my_value = property(__get_value
        , __set_value
        , __del_value
        , "This is my property")

Make Data Attributes Data-Descriptors

A common reason to use descriptors is to create read-only attributes. This means allowing callers to get (or access) an attribute’s value, but not allowing them to set its’ value. At first glance it seems as though one can accomplish this by using descriptors with only the __get__ method defined, i.e. a “non-data descriptor”.

While this seems to create a read-only descriptor attribute, it actually does nothing of the sort. Instead of using the descriptor’s __set__ function for the assignment operation (since no __set__ function is defined), the default Python assignment operation is used. This means that instead of stopping the assignment operation, a new attribute will be created in the instance (remember that descriptors have class scope) having the same name as the descriptor attribute and taking precedence in future operations.

This may be a bit confusing so let’s look at the example in Listing 4. It’s very much like our previous descriptor examples except it does not specify the __set__ and __del__ functions. Let’s look at what happens when we try to use this as though it were a read-only attribute:

Listing 4

class WonkyDescriptor(object):
    """This Descriptor isn't read only"""
    def __init__(self, value_name):
        self.value_name = str(value_name)

    def __get__(self, instance, owner):
        if (instance is None):
            raise AttributeError
        elif (not instance.__dict__.has_key(self.value_name)):
            raise AttributeError
        print "Getting a value"
        return instance.__dict__[self.value_name]

class MyClass(object):

    my_value = WonkyDescriptor("_value")

    def __init__(self, value):
        #initial value
        self._value = value

my_instance = MyClass(23) #Create the first instance
print my_instance.my_value
my_instance.my_value = "Don't set me!"
print my_instance.my_value

When we run this, we get the following output:

Getting a value
23
Don't set me!

As you can see we did not succeed in creating a read-only attribute, instead something else happened. The “Getting a value” string is printed out when we print out my_value for the first time. This means that we are accessing the value through the descriptor. We then set the value of the descriptor attribute and print out that new value. Since “Getting a value” isn’t printed out a second time, we know that the descriptor’s __get__ method is not accessed when the second print statement is called.

Let’s take a closer look at what is happening using the following code, which prints out the instance’s __dict__:

my_instance = MyClass(23) #Create the first instance
print my_instance.__dict__
my_instance.my_value = "Don't set me!"
print my_instance.__dict__

This results in the following, which explains what is happening:

{'_value': 23}
{'_value': 23, 'my_value': "Don't set me!"}

When we first access the __dict__ we see our value indexed by the _value key that we told the descriptor to use. Now look at the second __dict__. The old value is still there, but so is a new value my_value. When we assigned “Don’t set me!” to my_value, we didn’t overwrite the descriptor attribute at class scope, we created a new instance attribute! And it is not exactly a read-only attribute.

In order to create a read-only attribute, you need to create a data descriptor where the __set__ method raises an AttributeError exception. An easy way to do this is to create a property and only set the fget function. Instead of using the descriptor in Listing 4, we can construct a read-only attribute as follows:

def __get_value(self):
    return self._value
my_value = property(__get_value)

Conclusion

I hope that after reading this article you can see how powerful descriptors (especially data-descriptors) can be. Once you get the hang of them, they are a great way to implement “getters and setters” and read-only attributes. Data-Descriptors are also very useful for performing validation during the assignment operation, i.e. ensuring that a value remains a specific type, or in a specific form.

The more you play with descriptors and use them in your code, the more you’ll see how sophisticated and useful they can be. That being said, it’s important to remember that unless you have a good reason to use descriptors you probably don’t need them. There’s no need to sacrifice the readability of your code or the dynamism of Python just for the sake of using descriptors. But when you have a real reason for managed attributes, you’ll find that descriptors are more than up to the task.

[1] http://docs.python.org/ref/descriptor-invocation.html
[2] http://www.python.org/download/releases/2.2/descrintro/#property
[3] http://docs.python.org/ref/descriptor-invocation.html
[4] http://docs.python.org/ref/descriptors.html
[5] http://docs.python.org/ref/descriptors.html
[6] http://docs.python.org/ref/descriptors.html
[7] http://www.python.org/download/releases/2.2/descrintro/#property

I just wanted to re-point out the fact that there are some forums associated with this blog. There’s not much happening there, and recently they have become a haven for spammers, but I’m trying to clean them up and if other Python programmers read this blog maybe the forums could actually become useful!

Either way for those that didn’t know, there are learning python forums available.

Edit: Due to popular demand (well a couple of comments) I’ve decided to allow multiple answers to the poll. This should make everyone that uses two versions happy.

With two major versions of Python available to us Python programmers (2.X and 3.X) I thought it would be interesting to see which version the readers of this blog are using and targeting.

Personally I’m still using 2.5 on my Debian box because it’s the default, and 2.6 on my Windows PC. While I have used 3.X and have it installed, I’ve remained on the 2.X branch largely because many of the modules that I play around with are still focused on the 2.X branch so that’s where my focus has remained.

I voted 2.5 since I do that majority of my programing on my Debian box. So now what about you: Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.

If you want to explain your choice leave a comment below.

Note: This article was first published the March 2008 issue of Python Magazine

Mark Mruss

Over the past few years Google has expanded it’s services beyond those of a normal search engine. One of those new services is the Google Calendar. This article will provide an introduction to working with the Google Calendar using Python.

Introduction

As many of you know, Google has branched out and started offering more services besides their ubiquitous search engine. You have email, calendars, documents, spreadsheets, photos, maps, videos, source code hosting, and the list goes on. Fortunately for us Python programmers, Google released the Google data Python Client Library on March 26th, 2007, giving Python programmers easy access to some of these services.

What the Google data Python Client Library, or “gdata-python-client”, does is provide “a library and source code that makes it easy to access data through Google Data APIs.” [1] This leads to the question: “what are the Google Data APIs?” In the words of Google: “The Google data APIs provide a simple standard protocol for reading and writing data on the web. These APIs use either of two standard XML-based syndication formats: Atom or RSS.”[2]

The Google services that use the Google data APIs include many of the services that we have grown to know and love:

• Google Apps
• Google Base
• Blogger
• Google Calendar
• Google Code Search
• Google Documents
• Google Notebook
• Picasa Web Albums
• Spreadsheets
• YouTube

This tutorial will only deal with the Google Calendar service specifically, but it’s important to know that many of the techniques used here can easily be applied to other Google services.

The Google data Python Client Library requires Python 2.2 or greater and the ElementTree module to be installed. I recommend using Python 2.5 since the ElementTree module is included in that release of Python. This column assumes that you are using Python 2.5 and version 1.0.10 of the “gdata-python-client”.

Getting and Installing the “gdata-python-client”

We must first download and install the “gdata-python-client” files from the “gdata-python-client” website.[3] Once you have downloaded the compressed file, extract it’s contents to a folder. You will then need to browse to that folder and run the extracted setup.py file with the install command with root access. For me, the command looked like this:

# python2.5 setup.py install

Notice that I ran python2.5 instead of simply python. I did this to ensure that Python 2.5 was used instead of a previous version of Python.

Once you have done this, you can test to make sure that the gdata-python-client module has been installed properly by trying to import the gdata module. You can test this easily in the interactive Python shell:

$ python2.5
Python 2.5.2a0 (r251:54863, Jan  3 2008, 17:59:56)
[GCC 4.2.3 20071123 (prerelease) (Debian 4.2.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import gdata
>>>

If everything works properly you should see no errors when importing gdata. If you do run into problems, you can consult the install.txt file that was extracted with the rest of the files in the “gdata-python-client” archive that you downloaded.

Getting Started

There are some great Google Calendar examples that come with the “gdata-python-client” module. I found them to be very useful learning tools. The “Google Calendar Developers Guide”[4] is also very helpful. If you get stuck it is essential reading. The “Google Calendar API Reference Guide”[5] is also an indispensable help.

It is important to remember that when you are working with the Google Calendar service, you are actually working with XML data. You are sending and receiving XML data to and from the Google Calendar service – Atom or RSS feeds to be precise. The Python classes that wrap these feeds, or XML blocks, are dynamically “formed” around the XML. When you access something like the following:

e_link.href

You are actually accessing the “href” attribute of an XML link block that may look something like this:

<ns0 :link href="http://www.google.com/calendar/feeds/<username>/private/full" rel="alternate" type="application/atom+xml" xmlns:ns0="http:www.w3.org/2005/Atom" />

In the above example, if we had the link as an instance (perhaps an atom.Link object), e_link.rel would be equal to “alternate”.

Logging In

There are two types of Google calendars – public calendars and private calendars. Public calendars do not require any authentication while private calendars do. There are three forms of authentication for private calendars: “AuthSub proxy” authentication, “ClientLogin” authentication, and “Magic cookie” authentication.

“AuthSub proxy” authentication is meant for web applications so it will not be covered in this tutorial. “Magic cookie” authentication requires a string (the magic cookie) obtained from your Google Calendar settings page. This type of authentication gives you read only access to a private calendar. Its usage is quite specific and will not be covered in this column either.

For this column we will focus on good old-fashioned “ClientLogin” authentication. In order to perform “ClientLogin” authentication we need two things: a Google Calendar username and a password. If you do not have these, head over to the Google Calendar website [6] and get yourself signed up. If you have gmail or have signed up for other Google services, you can probably sign in using that username and password.

Note: I say username and so does the documentation, but this will actually be an email address.

We are going to read the username and password in from the command line using the raw_input function and the getpass module. I must thank the people who wrote the Google code examples for introducing me to the getpass module. Before looking through the examples, I had not heard of it.

The raw_input function simply reads a line of input from the command line and returns it with the newline stripped. The getpass.getpass function does the exact same thing as raw_input except it does not echo the input, which is handy when working with passwords.

We need to import the getpass module:

import getpass

Then, in our main function we will ask the user to input their username and password:

def main():
    username = raw_input("Enter your username: ")
    password = getpass.getpass("Enter your password: ")

if __name__ == "__main__":
    main()

Now that we have our username and password, we need to log the user in. To do this we are going to use the CalendarService class. The CalendarService “extends the GDataService to streamline Google Calendar operations.” [7] The GDataService class “provides CRUD ops. and programmatic login for GData services.” [8]. CRUD is an acronym that stands for Create, Retrieve, Update, and Delete. What the CalendarService class will allow us to do is create, retrieve, update, and delete things from a Google Calendar.

The first step in logging in is to create an instance of the CalendarService class. To do this we will pass in the username and password that we have collected:

calendar_service = gdata.calendar.service.CalendarService(username
    , password
    , "PythonMagazine-Calendar_Example-1.0")

The third parameter that we pass to the CalendarService constructor is the “source” string. This is a short “string identifying your application, for logging purposes. This string should take the form: “companyName-applicationName-versionID”"[9]

Now that we have a CalendarService instance, we need to login. To do this we will use the ProgrammaticLogin function. This will log into the Google Calendar using the CalendarService classes current username and password. The ProgrammaticLogin function can raise three possible exceptions that we want to be aware of:

1. CaptchaRequired – Raised if the login requires a “captcha” response in order to login.
2. BadAuthentication – Raised if the username and/or password were not accepted by the Google Calendar.
3. Error – Raised if a 403 error occurs that is not a “CaptchaRequired” or “BadAuthentication” error.

For this example, shown in Listing 1, we will only worry about the “BadAuthentication” exception.

Listing 1

def main():
    username = raw_input("Enter your username: ")
    password = getpass.getpass("Enter your password: ")

    calendar_service = gdata.calendar.service.CalendarService(username
        , password
        , "PythonMagazine-Calendar_Example-1.0")
    try:
        calendar_service.ProgrammaticLogin()
    except gdata.service.BadAuthentication, e:
        print "Authentication error logging in: %s" % e
        return
    except Exception, e:
        print "Error Logging in: %s" % e
        return

Working With Calendars

Now that we’ve logged into the calendar service, we can start working with the user’s available calendars. To get a list of all of the available calendars you can call the GetAllCalendarsFeed function. This will return a CalendarListFeed instance representing all of the user’s calendars. Since I want to show how to add and delete Calendars, I’m going to use the GetOwnCalendarsFeed function to get a list of all of the calendars that the “authenticated user has owner access to.”[10]

The GetOwnCalendarsFeed function returns a CalendarListFeed class instance. This contains some information (like a title) along with a list of CalendarListEntry classes. Each CalendarListEntry in this list represents a calendar. An example of a function that uses GetOwnCalendarsFeed and then prints out information about each calendar can be found in Listing 2. Sample output from this function can be found in Listing 3. The function was passed a CalendarService after logging in:

list_own_calendars(calendar_service)

Listing 2

def list_own_calendars(calendar_service):

    try:
        #Get the CalendarListFeed
        all_calendars_feed = calendar_service.GetOwnCalendarsFeed()
    except Exception, e:
        print "Error getting all calendar feed: %s" % (e)
        return
    #Print the feed's title
    print all_calendars_feed.title.text
    #Now loop through all of the CalendarListEntry items.
    for (index, cal) in enumerate(all_calendars_feed.entry):
        #Print out the title and the summary if ther is one
        if (cal.summary is not None):
            print "%d) %s - Summary: %s" % (
                index, cal.title.text, cal.summary.text)
        else:
            print "%d) %s" % (index, cal.title.text)
        #Print out the authors
        print "\tAuthors:"
        for author in cal.author:
            print "\t\t%s" % (author.name.text)
        #Print out other information
        print "\tPublished: %s \n\tUpdated: %s \n\ttimezone: %s" % (
            cal.published.text, cal.updated.text, cal.timezone.value)
        print "\tColour: %s \n\tHidden: %s \n\tSelected: %s" % (
            cal.color.value, cal.hidden.value, cal.selected.value)
        print "\tAccess Level: %s" % (cal.access_level.value)

Listing 3

Mark Mruss's Calendar List
0) Mark Mruss - Summary: Main Calendar
	Authors:
		Mark Mruss
	Published: 2008-02-07T04:15:15.701Z
	Updated: 2008-02-07T03:43:26.000Z
	timezone: America/Toronto
	Colour: #5229A3
	Hidden: false
	Selected: true
	Access Level: owner
1) PythonMagazine - Summary: Calendar for Articles
	Authors:
		PythonMagazine
	Published: 2008-02-07T04:15:15.702Z
	Updated: 2008-02-07T03:37:15.000Z
	timezone: America/Toronto
	Colour: #0D7813
	Hidden: false
	Selected: true
	Access Level: owner

Adding a Calendar

If we want to add a calendar, we need to create a CalendarListEntry class instance and then call the CalendarService classes InsertCalendar function. Let’s say I wanted to add a Banking calendar to my account, I could do the following to create the CalendarListEntry:

new_calendar = gdata.calendar.CalendarListEntry()
new_calendar.title = gdata.atom.Title(text="Banking")
new_calendar.summary = gdata.atom.Summary(text="Bills and payments")
new_calendar.timezone = gdata.calendar.Timezone(value="America/Toronto")
new_calendar.hidden = gdata.calendar.Hidden(value="false")
new_calendar.selected = gdata.calendar.Selected(value="true")

The calendar now needs to be added to the account:

try:
    created_calendar = calendar_service.InsertCalendar(new_calendar)
except gdata.service.RequestError, e:
    print "Error adding calendar: %s" % (e[0]["reason"])

This will add the new calendar to the authenticated users list of calendars, make it visible, and select it. Notice that InsertCalendar also returns a CalendarListEntry instance. This is the calendar that was actually added to the authenticated user’s list of calendars. It is wrapping the XML that represents the actual calendar, as opposed to the smaller version that we created for insertion.

Deleting a Calendar

Deleting a Calendar is very easy. You simply need to call the CalendarService classes DeleteCalendarEntry function. The DeleteCalendarEntry function takes three parameters which are documented in the source code:

1. edit_uri – The edit uri (Uniform Resource Identifier) of the Calendar that you want to delete.
2. url_params – Defaults to None. A dictionary containing URL parameters that will be included in the delete.
3. escape_params – Defaults to True. A boolean that controls whether or not the url_params will be escaped.

Note: There is another optional parameter: extra_headers, which is the second parameter, but at this point it does not seem to be used at all in the source code.

The simplest case is to ignore the optional parameters and simply pass in the edit uri of the calendar that you wish to delete. You can get the edit uri of a calendar by calling the GetEditLink function of the CalendarListEntry instance that represents the calendar that you are going to delete. An example of a function that will delete a calendar can be seen in Listing 4. This function takes a CalendarService instance and a CalendarListEntry instance as parameters.

Listing 4

def delete_calendar(calendar_service, calendar):
    e_link = calendar.GetEditLink()
    if (e_link is not None):
        try:
            calendar_service.DeleteCalendarEntry(e_link.href)
        except gdata.service.RequestError, e:
            print "Error deleting calendar: %s" % (e[0]["reason"])

Listing Events

Now let’s take a look at events. Events are items that are added to a specific calendar. If you want to remind yourself to pay your bills at the end of the month you might add that to your banking calendar. That “note” on your calendar is an event.

You can get a list of events for the primary calendar using the CalendarService classes GetCalendarEventFeed function. Since you may be working with more than one calendar, it’s probably more useful to be able to list the events for a specific calendar. You can do this in one of two ways:

1. You can pass in the optional uri parameter to the GetCalendarEventFeed function. From testing I found that a Calendar’s “alternate” link works.
2. You can use the CalendarService classes CalendarQuery function to query for a specific calendars event feed.

If you want to use the first option you can use the CalendarListEntry classes GetAlternateLink member function to get the uri, and then pass it to GetCalendarEventFeed:

a_link = calendar.GetAlternateLink()
#Make sure the link is valid
if (a_link is not None):
    event_feed = calendar_service.GetCalendarEventFeed(a_link.href)

If you use the CalendarQuery method, you need to get the calendar username (or id) of the calendar whose events you want to query. It seems strange to me that there appears to be no way to get a calendar’s username besides parsing one of the calendar’s links. The username of a calendar can be found in the alternate link after “feeds” and before the visibility and projection:

http://www.google.com/calendar/feeds/< <username>>/private/full

Note: You can use the username “default” to query the default calendar.

For the sake of simplicity I will use the alternate link method for my examples. A full example that prints out calendar data and a calendar’s events can be found in Listing 5. The method that prints out the event data is called print_event_feed. It is called near the end of the list_own_calendars method:

# Now Print out the events
print "Events:"
a_link = cal.GetAlternateLink()
if (a_link is not None):
    event_feed = calendar_service.GetCalendarEventFeed(a_link.href)
    print_event_feed(event_feed)

Listing 5

#! /usr/bin/env python

import gdata.calendar.service
import getpass

def print_event_feed(event_feed):

    for index, event in enumerate(event_feed.entry):
        print "\t%d) %s\r\n\tContent: %s" % (
                index, event.title.text, event.content.text)
        print "\t\tWho:"
        for person in event.who:
            print "\t\t\tName: %s\n\t\t\temail: %s" % (person.name
                , person.email)
        print "\t\tAuthors:"
        for author in event.author:
            print "\t\t\t%s" % (author.name.text)
        print "\t\tWhen:"
        for e_index, e_time in enumerate(event.when):
            print "\t\t\t%d) Start time: %s\n\t\t\tEnd time: %s" % (
                e_index
                , e_time.start_time
                , e_time.end_time)

def list_own_calendars(calendar_service):

    try:
        #Get the CalendarListFeed
        all_calendars_feed = calendar_service.GetOwnCalendarsFeed()
    except Exception, e:
        print "Error getting all calendar feed: %s" % (e)
        return
    #Print the feed's title
    print all_calendars_feed.title.text
    #Now loop through all of the CalendarListEntry items.
    for (index, cal) in enumerate(all_calendars_feed.entry):
        #Print out the title and the summary if there is one
        if (cal.summary is not None):
            print "%d) %s - Summary: %s" % (
                index, cal.title.text, cal.summary.text)
        else:
            print "%d) %s" % (index, cal.title.text)
        #Print out the authors
        print "\tAuthors:"
        for author in cal.author:
            print "\t\t%s" % (author.name.text)
        #Print out other information
        print "\tPublished: %s \n\tUpdated: %s \n\ttimezone: %s" % (
            cal.published.text, cal.updated.text, cal.timezone.value)
        print "\tColour: %s \n\tHidden: %s \n\tSelected: %s" % (
            cal.color.value, cal.hidden.value, cal.selected.value)
        print "\tAccess Level: %s" % (cal.access_level.value)
        # Now Print out the events
        print "\tEvents:"
        a_link = cal.GetAlternateLink()
        if (a_link is not None):
            event_feed = calendar_service.GetCalendarEventFeed(a_link.href)
            print_event_feed(event_feed)

def main():
    username = raw_input("Enter your username: ")
    password = getpass.getpass("Enter your password: ")

    calendar_service = gdata.calendar.service.CalendarService(username
        , password
        , "PythonMagazine-Calendar_Example-1.0")
    try:
        calendar_service.ProgrammaticLogin()
    except gdata.service.BadAuthentication, e:
        print "Authentication error logging in: %s" % e
        return
    except Exception, e:
        print "Error Logging in: %s" % e
        return

    list_own_calendars(calendar_service)

if __name__ == "__main__":
    main()

An example of the output produced by Listing 5 can be seen in Listing 6. Don’t mind the formatting – it’s not pretty. It’s merely meant to give you an example of some of the data that you can mine from an event feed.

Listing 6

Mark Mruss's Calendar List
0) Mark Mruss - Summary: Main Calendar
	Authors:
		Mark Mruss
	Published: 2008-02-12T02:30:18.731Z
	Updated: 2008-02-08T04:03:10.000Z
	timezone: America/Toronto
	Colour: #5229A3
	Hidden: false
	Selected: true
	Access Level: owner
	Events:
	0) Dinner at the Drake
	Content: Dinner
		Who:
			Name: Mark Mruss
			email: mark.mruss@gmail.com
		Authors:
			Mark Mruss
		When:
			0) Start time: 2008-02-15T21:00:00.000-05:00
			End time: 2008-02-15T22:30:00.000-05:00
	1) Cezanne's Closet
	Content: Cezanne's Closet
		Who:
			Name: Mark Mruss
			email: mark.mruss@gmail.com
		Authors:
			Mark Mruss
		When:
			0) Start time: 2008-02-09
			End time: 2008-02-11

Adding an Event

Adding a new event to a calendar is very similar to creating a new calendar. You need to create a new CalendarEventEntry instance. You then populate it with options, and pass it to the CalendarService classes InsertEvent function, along with the alternate link of the calendar to which you would like to add the event.

If you take a look at Listing 7 you can see a simple example of how to add an all day event. If you want to specify an event that lasts for less then a day, or include a specific start and end time, you need to use the “RFC 3339 timestamp” format for your start_time and end_time values. Also notice that just like adding a Calendar, the newly created event is returned by the InsertEvent function.

Listing 7

a_link = calendar.GetAlternateLink()
if (a_link is not None):

    new_event = gdata.calendar.CalendarEventEntry()
    new_event.title = gdata.atom.Title(text="New Article")
    new_event.content = gdata.atom.Content(text="Write Article")
    new_event.when.append(gdata.calendar.When(
        start_time="2008-02-20"
        , end_time="2008-02-21"))
    try:
        created_event = calendar_service.InsertEvent(new_event
            , a_link.href)
    except gdata.service.RequestError, e:
        print "Error adding event: %s" % (e[0]["reason"])

Deleting an Event

Deleting an event is (you guessed it) almost identical to deleting an entire calendar. This is because both helper functions wrap the same GDataService base class function. However the 'DeleteEvent event function does a little bit more work than the DeleteCalendarEntry function by making sure that your edit_uri is in the correct format.

To delete an event you simply need to call the DeleteEvent function of the CalendarService class. An example of a function that will delete an event can be seen in Listing 8. This function takes a CalendarService instance and a CalendarEventEntry instance as parameters. As with the majority of these actions, the CalendarService instance should already be authenticated.

Listing 8

def delete_event(calendar_service, event):
    e_link = event.GetEditLink()
    if (e_link is not None):
        try:
            calendar_service.DeleteEvent(e_link.href)
        except gdata.service.RequestError, e:
            print "Error deleting event: %s" % (e[0]["reason"])

Conclusion

Hopefully this article has given you a small taste of what is possible with the gdata-python-client module and Google Calendars. Maybe you can already imagine a use for these features in your next Python application? Please keep in mind that there is much more that can be done with Google Calendars, including updating existing Calendars and Events (CalendarService member functions: UpdateCalendar and UpdateEvent), and almost any other task you can perform using the “online” version of the Google Calendar service.

Also, it is important to note that the Python documentation for the Google Calendar service is sparse but growing. A lot of what is contained in this article was found through browsing the examples, the gdata-python-client source code, other people’s examples, as well as a whole bunch of trial and error on my part. Since this is the case, there may be other ways, or more preferred methods of accomplishing what I have shown here.

[1] http://code.google.com/p/gdata-python-client/
[2] http://code.google.com/apis/gdata/index.html
[3] http://code.google.com/p/gdata-python-client/
[4] http://code.google.com/apis/calendar/developers_guide_python.html
[5] http://code.google.com/apis/calendar/reference.html
[6] http://www.google.com/calendar
[7] gdata.calendar.service.py (From the source comments)
[8] gdata.service.py (From the source comments)
[9] http://code.google.com/apis/accounts/AuthForInstalledApps.html
[10] http://code.google.com/apis/calendar/developers_guide_python.html

By: Mark Mruss

Note: This article was first published the February 2008 issue of Python Magazine

Of all the tasks assigned to programmers, commenting code and writing documentation are among the most disliked. This article introduces you to Python’s documentation strings. While they won’t make commenting your code any more enjoyable, they will provide a systematic approach to doing it, as well as access to additional tools for documentation generation and testing.

You’ve just finished your new Python module. You can’t wait to upload it to the Web and let all the other Python hackers start using it. The only step left is the most dreaded for many programmers: documentation. Unless you’ve been commenting and documenting your code as you wrote it, you’re going to have to go back through each source file, class, and function, trying to remember exactly what your code was supposed to do. Not an enjoyable task.

Sound familiar to you? If it does, you’re probably not using documentation strings, commonly known as doc strings or docstrings (I prefer docstrings, without the space), or any of the handy tools that work with docstrings. This article will introduce you to Python’s docstrings, and a few of the tools that make them a great addition to your code.

Docstrings

If you are not already using docstrings in your Python code, you really should. They provide a standard way to comment your code, giving you and other developers (who might want to use your code at some point) easy access to descriptions of the modules, classes, and functions found within.

At the heart of it, docstrings are simply comments placed in special locations in your Python source code. These comments can then be looked at by tools designed to work with docstrings or other Python programmers using your code. Note that I said using// your code, not reading your code. This is because docstrings are accessible via a Python object’s __doc__ attribute. This is very helpful is you are testing out a new module in Python’s interactive shell and really need to know what sort of parameters a certain function needs.

Pep 257 has a good definition of docstrings: “A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. Such a docstring becomes the __doc__ special attribute of that object.”[1] Definitions are nice but it might be easier to look at a quick example of some docstrings:

def add(x, y):
    """This is the add function's docstring."""
	return x + y

def subtract(x, y):
    """This is the subtract function's docstring.
    It is longer then the add functions, it goes
    on and on and on and on."""
    return x - y

Since docstrings are comments they can be in any format that you want, however since this is Python there are a few style guidelines that you should probably be aware of. You don’t have to follow these guidlines but if you do it will be easier for other Python programmers to understand and work with your code.

In general there are two types of docstrings: one-line docstrings and multi-line docstrings. The difference between the two should be fairly obvious: one-line docstrings are only one line in length and multi-line docstrings are more then one line in length. One-line docstrings and multi-line docstrings each have different style guidelines that will be explained in more detail in the following two sections.

One-Line Docstrings

Let’s look at a quick example of a one-line docstring for a simple function:

def add(x, y):
    """Return the sum of two numbers."""
    return x + y

In this example """Return the sum of two numbers.""" is the docstring of the add function. If you were to run the following:

print add.__doc__

The output would look like this:

Return the sum of two numbers.

In his “Python Style Guide” Guido van Rossum has a few notes on the preferred style of one line docstrings:

• Triple quotes are used even though the string fits on one line. This makes it easy to later expand it.
• The closing quotes are on the same line as the opening quotes. This looks better for one-liners.
• There’s no blank line either before or after the doc string.
• The doc string is a phrase ending in a period. It prescribes the function’s effect as a command (“Do this”, “Return that”), not as a description: e.g. don’t write “Returns the pathname …” [2]

One-line docstrings should only be used to document the simplest of cases. If what you are documenting does anything complicated, accepts input, or returns a value, it’s probably a good idea to use a multi-line docstring.

Multi-Line Docstrings

Multi-line docstrings should be used to document the majority of your modules, classes, functions, and methods. This is because most of what you are programming performs tasks more complicated then that which can fit into a single sentence summary. Multi-line docstrings should be used to documents the input, output, and complex behaviour of your objects. Like one-line docstrings, multi-line docstrings should start with a single sentence summary. After that there should be a blank line and then a more detailed description. The blank line separating the one line summary and the additional information is important as certain tools will use the blank line to separate the summary from the rest of the docstring. An example of a multi-line comment can be found in Listing 1.

Listing 1

def subtract(x, y):
    """Return the difference between two numbers.

    Arguments:
    x -- The minuend.
    y -- The subtrahend.

    Returns:
    A number, the difference between x and y (ie. x - y)

    """
    return x - y

What to Document

There are also style guidelines that dictate what you should include in your multi-line docstrings when documenting different sections of your code. These are general guidelines but following them will help ensure that your code is well documented and easily understood by other Python programmers. For more information on this please see Guido’s “Python Style Guide”.[2]

Modules – Document the module and provide a one line summary of everything that is exported by the module. (eg. classes, exceptions, and functions).

Classes – Summarize the functionality of the class. List all public methods and data members of the class. If there is any addition information needed to subclass the class, or if there is an additional interface for subclasses provide a description.

Functions and Methods – Summarize the functionality of the function and “document its arguments, return value(s), side effects, exceptions raised, and restrictions on when it can be called (all if applicable).”[3]

An example of all of these can be found in Listing 2.

Listing 2

#!/usr/bin/env python
"""A simple math module.

Exported Classes:

Math -- A simple math class with mathematical functions.

"""

class Math(object):
    """A simple math class with mathematical functions.

    Public functions:
    add -- Adds two numbers together and
    returns the result.

    subtract -- Returns the difference between
    two numbers.

    """

    def subtract(self, x, y):
        """Return the difference between two numbers.

        Arguments:
        x -- The minuend.
        y -- The subtrahend.

        Returns:
        A number, the difference between x and y (ie. x - y)

        """
        return x - y

    def add(self, x, y):
        """Return the sum of two numbers.

        Arguments:
        x -- Number to be summed.
        y -- Number to be summed.

        Returns:
        A number, the sum of x and y (ie. x + y)

        """
        return x + y

Documentation Generation

Generally if you have a complicated module or API people would rather read a help file, or online documentation, as opposed to constantly having find and browse the source code. Thankfully for us there are many different documentation generation tools out there that make use of docstrings. So when you are writing your docstrings, you aren’t just commenting your source code you’re also writing your help file!

The easiest tool to use is PyDoc, it’s a module, and a stand-alone application, that has been included in the Python standard library since version 2.1. There is much that can be done with PyDoc but for this column we are going to focus on it’s documentation generation. PyDoc can take the docstrings found within a module and output them as either simple text documentation (much like UNIX or Linux man pages) or HTML documentation.

Creating HTML documentation of the SimpleMath module used in Listing 2 is very easy. On a UNIX like computer (Linux, OS X, etc.) it can be accomplished using the following command:

$ pydoc -w /home/selsine/python/SimpleMath.py

On Windows the command will look something like this:

C:>c:Python25Libpydoc.py -w c:pythonSimpleMath.py

This will create an HTML file named SimpleMath.html in the current folder. A sample of what the generated HTML looks like can be seen in Figure 1.

Figure 1 - pydoc

While PyDoc is a great tool and easy to use because it is in the standard library, there are a few other tools out there that you might consider using. If you look at the HTML file that PyDoc generates you will notice that it does not mark up your docstrings. It simply reads them from your source code and spits them back out. If you are looking for something a little bit fancier with a few more options you might consider Epydoc[4] or docutils[5].

Both Epydoc and docutils use simple markup languages to give the documentation generated a bit more punch. Docutils uses the reStructuredText markup language. While Epydoc uses the Epytext markup language, as well as being able to work with Javadoc and reStructuredText. An example of the HTML that Epydoc produces can be seen in Figure 2. The code that was used to generate the HTML can be found in Listing 3.

Figure 2 – Epydoc

Listing 3

#!/usr/bin/env python
"""A simple math module.

Exported Classes:

Math -- A simple math class with mathematical functions.

"""

class Math(object):
    """A simple math class with mathematical functions.

    Public functions:
    add -- Adds two numbers together and
    returns the result.

    subtract -- Returns the difference between
    two numbers.

    """

    def subtract(self, x, y):
        """Return the difference between two numbers.

        @type   x: number
        @param  x: The minuend.
        @type   y: number
        @param  y: The subtrahend.

        @rtype: number
        @returns: A number, the difference between x and y (ie. x - y)

        """
        return x - y

    def add(self, x, y):
        """Return the sum of two numbers.

        @type   x: number
        @param  x: Number to be summed.
        @type   y: number
        @param  y: Number to be summed.

        @rtype: number
        @returns: A number, the sum of x and y (ie. x + y)

        """
        return x + y

def _test():
    import doctest
    doctest.testmod()

if __name__ == "__main__":
    _test()

Doctest

One of the most interesting uses of docstrings is unit testing using the doctest module. As we all know testing our code is important, and (as many of us have begun to learn) writing unit tests for large projects is a great way to ensure that changes in one area of a project don’t cause problems elsewhere in the code. If you don’t know what a unit test is it’s basically a simple test case to prove whether a specific area of your code is functioning properly. Generally at least one unit test is created for each object in order to test the correctness of the project as a whole.

In a nutshell the the doctest module searches your dostrings "for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown."[6] This means that it will search your docstrings for lines that start with >>> or with ... if they are the continuation of a statement (i.e. the inside of a function). If there is output generated by the statements it “must immediately follow the final ‘>>> ‘ or ‘… ‘ line containing the code, and … extends to the next ‘>>> ‘ or all-whitespace line.”[7] Adding doctests doesn’t just add unit tests to your code it also provides working examples in your docstrings and documentation.

Since doctests are formatted to look like interactive Python sessions a simple way to write them is to use Python’s interactive shell. You do this by simply executing your code in the interactive shell, and then copying and pasting the resulting test into your docstrings. For example, if we were to use this method to write a doctest for our subtract method we could do something like the following in the interactive shell:

>>> from SimpleMath import Math
>>> simple_math = Math()
>>> simple_math.subtract(10, 7)
3
>>>

We don’t need to import our module for the doctest so all we will need to copy from the interactive shell are the middle three lines:

>>> simple_math = Math()
>>> simple_math.subtract(10, 7)
3

As you can see, what we have here is a test compromising of two lines of Python code and then the expected result. The SimpleMath module containing a doctest for each function can be found in Listing 4.

Listing 4

#!/usr/bin/env python
"""A simple math module.

Exported Classes:

Math -- A simple math class with mathematical functions.

"""

class Math(object):
    """A simple math class with mathematical functions.

    Public functions:
    add -- Adds two numbers together and
    returns the result.

    subtract -- Returns the difference between
    two numbers.

    """

    def subtract(self, x, y):
        """Return the difference between two numbers.

        >>> simple_math = Math()
        >>> simple_math.subtract(10, 7)
        3

        @type   x: number
        @param  x: The minuend.
        @type   y: number
        @param  y: The subtrahend.

        @rtype: number
        @returns: A number, the difference between x and y (ie. x - y)

        """
        return x - y

    def add(self, x, y):
        """Return the sum of two numbers.

        >>> simple_math = Math()
        >>> simple_math.add(10, 7)
        17

        @type   x: number
        @param  x: Number to be summed.
        @type   y: number
        @param  y: Number to be summed.

        @rtype: number
        @returns: A number, the sum of x and y (ie. x + y)

        """
        return x + y

def _test():
    import doctest
    doctest.testmod()

if __name__ == "__main__":
    _test()

If you look at Listing 4 you will also notice the following code at the end of the source:

def _test():
    import doctest
    doctest.testmod()

if __name__ == "__main__":
    _test()

This is the code that will be executed if the module is launched directly from the command line. The code will import the doctest module and then use the testmod method to test the current module. Now when we run our SimpleMath.py file directly we will get the following:

$ python SimpleMath.py
$

Since nothing was written out to the command line we know that all of the doctests were successful. If you want to get more information you can use the -v flag to produce verbose output:

$ python SimpleMath.py -v

If you were to encounter an error it would look something like Listing 5.

Listing 5

**********************************************************************
File "SimpleMath.py", line 26, in __main__.Math.subtract
Failed example:
    simple_math.subtract(10, 7)
Expected:
    4
Got:
    3
**********************************************************************
1 items had failures:
   1 of   2 in __main__.Math.subtract
***Test Failed*** 1 failures.

As an added bonus if you are using Epydoc or docutils to generate your documentation, both tools will recognize doctest sections and highlight them. An example of what it looks like when Epydoc does this can be seen in Figure 3.

Figure 3 - doctest

Conclusion

Hopefully by this point you can see how useful docstrings can be to any Python code that you write. They give you a structured way to document your source code, the ability to easily generate great looking documentation, and a simple way to add unit tests to your code. But that's not all, as more and more people start using doctrings you can be sure that the list of docstring tools will continue to grow.

We all know that commenting source code and writing documentation are among the least enjoyable tasks a programmer can face. In fact the only thing worse then commenting and documenting is trying to use code with no comments and poor documentation! So do me, and yourself, a favour: start writing those docstrings.

[1] http://www.python.org/dev/peps/pep-0257/#what-is-a-docstring
[2] http://www.python.org/doc/essays/styleguide.html
[3] http://www.python.org/doc/essays/styleguide.html
[4] http://epydoc.sourceforge.net/
[5] http://docutils.sourceforge.net/
[6] http://docs.python.org/lib/module-doctest.html
[7] http://docs.python.org/lib/doctest-finding-examples.html

By: Mark Mruss

Note: This article was first published the January 2008 issue of Python Magazine

Iterators, iterables, and generators are features handled so wall by Python that people programming in other languages cannot help but drool over. Fortunately for us, creating iterators, iterables and generators is a relatively simple task. This article introduces the concepts of iterators, iterables, and generators and illustrates how easy it is to add them to your code.

1. Introduction
2. Iteration in Python
3. An Initial Example
4. Creating An Iterator
5. Looking More Closely At The Iterator
6. The Upside And Downside Of Iterators
7. Generators
8. Looking Closely At The Generator
9. But What About Iterables?
10. Creating An Iterable Object
11. Conclusion

Introduction

In this article I’m going to introduce three related Python features: iterators, iterables, and generators. Generators are easy to define, they are functions that create and return an iterator. Iterators and iterables on the other hand, are easier to use than they are to define. An iterable object is a “container object capable of returning its members one at a time.”[1] An iterator object is “An object representing a stream of data. Repeated calls to the iterator’s next() method return successive items in the stream. When no more data is available a StopIteration exception is raised instead. At this point, the iterator object is exhausted and any further calls to its next() method just raise StopIteration again.”[1] You can think of the difference between the two in this way: an iterable object can be iterated over multiple times, whereas an iterator object can only be iterated over once. In general an iterable produces an iterator every time something wants to iterate over its data.

Note: Classes that define the __getitem__ function are also considered iterables, but since that falls outside the scope of this article, it will not be covered here.

In this tutorial, I will begin by discussing iterators, the most basic concept. Then I will move onto generators, and finish by discussing iterables, the most wide open topic of the three.

Iteration in Python

Iterators objects are used in Python in order to iterate over an objects data. For example, we all know how to do this in Python when we work with lists:

my_list = [1,2,3]
for num in my_list:
	print num

This code will iterate over the list object my_list and print out all of the list items , i.e., the numbers 1, 2, and 3. Iterating over sequences in this simple and transparent manner happens to be one of my favourite features of Python.

According to our definition above, lists are iterables since you can iterate over them multiple times. In fact, each time you iterate over a list you are actually using a listiterator iterator object produced by the list.

It may not be immediately clear to you when you should add an iteration support to a class. However, the more you work with Python the more you’ll find instances when doing just this is very useful, sometimes the only advantage is cleaner looking code. One nice thing about iterators (and generators too) is that the processing for each item happens as you need it. Instead of collecting all of the data into a list and then running through the list, you will collect each item as you need it. This might not seem like a large difference but imagine if there were tens of thousands of items to process? What if you were collection your data from an online source? Performing all of your processing up front may take a very long time, especially if you only wanted the first few items.

An initial example

In order to explain iteration further, let’s look at a simple example task where we might use iterators. For this example we will create a class that takes a string of characters as input and then converts each character into its byte value. If we were NOT going to use iterators we might do something like what is found in Listing 1.

Listing 1

class ByteValue(object):

	def __init__(self, data):
		self.data = data

	def to_bytes(self):
		bytes = []
		for char in self.data:
			bytes.append(ord(char))
		return bytes

This code is pretty simple. We have a data member, named data, that we use to store the string that was used to initialize the class. In the to_bytes function we loop through the string, converting each character to its byte value using the built in ord function. We store each byte value in a list and once we have collected all of the values we return that list.

When we run the following:

bv = ByteValue("abcdef")
for byte in bv.to_bytes():
	print byte

we would get this as our output:

97
98
99
100
101
102

Creating an Iterator

Let’s convert this into an iterator. Making your class into an iterator requires adding “two methods, which together form the iterator protocol.”[2] The two functions needed are: 1) the __iter__ function; and, 2) the next function. The __iter__ function will return the object itself, while the next function will return the next item. The next function is where the actual iteration work occurs. The next function iterates by returning the next item in the “sequence” each time it is called for as long as there is a “next” item. When there are no more items to iterate over, the next function must raise the StopIteration exception to halt the iteration.

To be clear, in order to make your class an iterator you need to do two things:

1) Add an __iter__ function that returns the object itself (self)

2) Add a next function that returns the next item in the sequence each time it is called. When there are no more items in the sequence, the next function raises a StopIteration exception signal the end of the iteration.

For those of you still confused, the following example will help illustrate how iterators work. If we were to convert our ByteValue class into an iterator object, it might look something like Listing 2.

Listing 2

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0

	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value

Let’s compare the code in Listings 1 and Listing 2 in detail, focusing on the iterator in Listing 2. The first difference is the addition of the data member current_item to the class, initialized in the __init__ function. current_item serves as a counter and keeps track of the current character in the string while we iterate over it. The counter must have class scope since the iterator works through successive calls to the next function. If current_item were local to the next function, its value would be reset with each subsequent call, and would not be of much use.

The second difference between the listings is the addition in Listing 2, of the __iter__ function where we return self.

The final addition to Listing 2 is the next function, where we first check to see if current_item is equal to the length of our string. If current_item is equal to the strings length we raise the StopIteration exception to signal the end of the iteration because we have no more characters left to iterate over. If there are more characters to iterate over we calculate the byte value of the current character, increase our current_item counter, and then return the byte value.

Note: Notice that current_item is only initialized when the ByteValue object is created. This happens because according to the iteration protocol, “once an iterator’s next() method raises StopIteration, it will continue to do so on subsequent calls.”[2] If we were to re-initialize current_item we would then be able to iterate over the iterator more than once breaking the iteration protocol.

Now that we have converted our class into an iterator we can use it as follows:

for byte in ByteValue("abcdef"):
	print byte

Doing so would result in:

97
98
99
100
101
102

Notice that we do not store the instance of our ByteValue class in a variable. Doing so would be useless because since ByteValue is an iterator it is only good for one pass of the data. If ByteValue were an iterable (returning an iterator object when __iter__ was called) it would make sense to keep an instance around because we could iterate over the instance more than once. We will look at creating iterables later on in this article.

Looking more closely at the Iterator

Let’s look at what is happening in more detail by examining what is happening behind the scenes during the iteration process. In order to illustrate what is happening in the for loop I will demonstrate the order in which things are being called behind the scenes.

The first step in the iteration process is to call to the __iter__ function in order to get the iterator object that will perform the iteration. Notice that this works on iterator and iterable objects, since iterators returns themselves and iterables return an iterator.

bv = ByteValue("abcdef")
iterator = bv.__iter__()

Now that we have the iterator object, we start iterating by calling the next function in order to get the next value:

print iterator.next()

This executes the next function which will return the byte value of character in the string with which we are currently working. Since this is the first call to the next function current_item will be zero and we will calculate the byte value of the first character in our string (‘a’) resulting in:

97

If we continue the iteration process by calling the next function six more times we would get the results shown in Listing 3. Notice that we have now made a total of seven calls to the next function, one more then the number of characters in our string. I’m running the python code from a file (iter.py found in Listing 4). Depending on how you are running it, you might get slightly different results. However, the most important thing to observe in this example is the exception that is raised.

Listing 3

97
98
99
100
101
102
Traceback (most recent call last):
  File "Listing4.py", line 34, in ?
    main()
  File "Listing4.py", line 30, in main
    print iterator.next()
  File "Listing4.py", line 13, in next
    raise StopIteration
StopIteration

Listing 4

#!/usr/bin/env python

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0
	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value
		return self.data

def main():
    for v in ByteValue("abc"):
        if v in ByteValue("abc"):
            print "We have a %d" % v

    bv = ByteValue("abcdef")
    iterator = bv.__iter__()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()
    print iterator.next()

if __name__ == "__main__":
	# Someone is launching this directly
	main()

For the most part you won’t be calling an iterator object’s __iter__ or next functions manually, instead you’ll probably just let the for loop do it all for you.

The upside and downside of Iterators

Now that our class is an iterator object we can use any of the built-in functions and methods that work on iterators and iterables, such as: the sum, tuple, sorted, and list functions, to name a few.

In the first example the bytes function returned a list object that we could work with. If we want a list instead of an iterator for any reason, it will be as simple as using the list function, which takes an iterator as a parameter and returns a list:

list(ByteValue("abcdef"))

If we want a sorted list:

sorted(ByteValue("abcdef"))

If we want the sum of all of the bytes:

sum(ByteValue("abcdef"))

While very useful, iterators do have some downsides to them. The most obvious is that they are only good for one pass over the data. They also generally require you to add extra data members to your class in order to keep track of your current iterator position. Depending on what you are iterating over, this process can become quite complex. Iterators also only allow you to perform one “type” of iteration, i.e., in one direction or over one piece of internal data. You might address this by adding flags to your class but this will further clutter the class and decrease readability. So how can you simply add multiple types of iteration to a class? Enter the generator!

Generators

A generator is a function that creates, or generates, an iterator. In order for a function to become a generator it must return a value using the yield keyword.

Generators are interesting because they are functions, yet execution does not run through them as it does in a normal function. The first time execution enters a generator function it will start at the beginning of the function and continue until the yield keyword is encountered. When the iteration continues, execution will continue in the generator function on the statement immediately following the yield keyword. All local variables in the function will remain intact. If the yield statement occurs within a loop, execution will continue within the loop as though execution had not been interrupted.

Continuing with the ByteValue example, let’s add a generator function named reverse that can be used to iterate through the byte vales of the string in reverse order:

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		yield ord(self.data[current_item])

So what’s going on here? The first thing to notice is that we did not add anymore data members to our class, this generator is a self-contained unit. Secondly, since this is a generator, our counter current_item can be a local variable.

In the reverse function the first step is to initialize current_item, which represents the current character in the string, to be equal to the length our string. We initialize it to the length of the string instead of zero since we are iterating through the string in reverse. Next we have a while loop that loops while current_item is greater than zero. We then subtract one from our counter, to give us the current character to process. Finally, we yield the byte value of the current character.

Note: We subtract one from our counter the first time through the loop because Python is zero-based and the length of a list minus one gives us the position of the last item in the list. In our examples we have used the following string:

abcdef
012345

When we calculate the length of the string we get 6. We then subtract 1 from that number, leaving 5, which is the index of the last number in the string.

Making use of our new generator function, we run the following:

bv = ByteValue("abcdef")
for byte in bv.reverse():
	print byte

We get our favourite byte values in reverse:

102
101
100
99
98
97

Looking closely at the Generator

Let’s take a detailed look at what is happening in the generator in the same way that we did earlier with the iterator object. The first thing that happens when you call a generator function is NOT the execution of the actual function, rather, it is the creation of a generator object. Running the following:

bv = ByteValue("abcdef")
gen = bv.reverse()
print gen

will result in:

<generator object at 0xb7d8e04c>

This demonstrates that, as stated above, the first call to our generator function does not return the byte value of that last character in the string, instead it creates a generator object. In fact the first time a generator function is called, the actual function is not executed at all. A generator object is “what Python uses to implement generator iterators. They are normally created by iterating over a function that yields values”.[3]

Once we have a generator object we can start calling its next function (sound familiar?) to perform the action iteration. An example of this can be found in Listing 5. The results of this execution can be found in Listing 6. The results will seem very familiar to you, especially the StopIteration exception.

Listing 5

#!/usr/bin/env python

class ByteValue(object):

	def __init__(self, data):
		self.data = data
		self.current_item = 0
	def __iter__(self):
		return self

	def next(self):
		if (self.current_item == len(self.data)):
			raise StopIteration
		else:
			byte_value = ord(self.data[self.current_item])
			self.current_item += 1
			return byte_value
		return self.data

	def reverse(self):
		current_item = len(self.data)
		while (current_item > 0):
			current_item -= 1
			yield ord(self.data[current_item])

def main():

	bv = ByteValue("abcdef")
	gen = bv.reverse()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()
	print gen.next()

if __name__ == "__main__":
	# Someone is launching this directly
	main()

Listing 6

102
101
100
99
98
97
Traceback (most recent call last):
  File "Listing5.py", line 40, in ?
    main()
  File "Listing5.py", line 36, in main
    print gen.next()
StopIteration

If we want to look at the contents of the generator object we could use the following code:

print dir(gen)

And getting the following results:

['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'gi_frame', 'gi_running', 'next']

Notice that the generator object contains the __iter__ and next functions making the generator object itself an iterator. Because the generator itself is an iterator object, the same built-in iterator functions I mentioned earlier can also be used on our generators:

print sum(bv.reverse())

It is also important to remember that your generator or iterator does not have to perform only “dumb” iteratation, simply moving through some sort of an internal list. Rather, it can make decisions just like any other block of code.

For example, let’s say that in our reverse generator we want to use the byte value 99 as an end condition. We can do something similar to the example found in Listing 7.

Listing 7

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		value = ord(self.data[current_item])
		if (value == 99):
			return
		yield value

In Listing 7 if the byte value equals 99 we return from our generator function using the return keyword. Since we didn’t yield anything this will cause the StopIteration exception to be fired halting the iteration.

Be careful about getting too smart with your iteration because you cannot return any information from a generator you can only yield it. So if you tried to execute the code in Listing 8, in an attempt to return one last value before quitting (or if you wanted to return a success or failure code) you will get the following error when the return value line of code is executed:

SyntaxError: 'return' with argument inside generator

Listing 8

def reverse(self):
	current_item = len(self.data)
	while (current_item > 0):
		current_item -= 1
		value = ord(self.data[current_item])
		if (value == 99):
			return value
		yield value

But what about iterables?

I’m sure that by now you are wondering about the iterable objects that I mentioned at the start of this tutorial. As you have probably guessed, simply making your class an iterator isn’t that useful unless it is only performing one task like our ByteValue example. Since your classes will generally be performing more than one task, you will likely want to make your class an iterable object rather than an iterator object. To recap, iterable objects return an iterator object when their __iter__ function is called, which allows for multiple passes over their data.

Creating an Iterable object

Since the definition of an iterable object is an object that returns an iterator object when its __iter__ function is called, creating an iterable object can be done in a variety of ways. Two options that come to mind are: 1)creating an iterator helper class to perform the iteration and, 2) using a generator function. I prefer using a generator function since it keeps the functionality within the main class.

See Listing 9 for an example of creating an iterarable object using a generator. You will see that we have replaced the next function with the forward function. The forward function is a generator that iterates through the data in the “forward” direction. In the __iter__ function we return the results of a call to the forward function, a generator object. Since generator objects contain the iterator protocol and are, in fact iterators, by returning one from our __init__ function we have successfully created an iterable.

Listing 9

class ByteValue(object):

	def __init__(self, data):
		self.data = data

	def __iter__(self):
		#We are an iterable, so return our iterator
		return self.forward()

	def forward(self):
		#The forward generator
		current_item = 0
		while (current_item < len(self.data)):
			byte_value = ord(self.data[current_item])
			current_item += 1
			yield byte_value

	def reverse(self):
		#The reverse generator
		current_item = len(self.data)
		while (current_item > 0):
			current_item -= 1
			yield ord(self.data[current_item])

def main():
    bv = ByteValue("abc")
    for v in bv:
        if v in bv:
            print "We have a %d" % v

if __name__ == "__main__":
    main()

Now that we have an iterable object, we can iterate over it as many times as we want. We can even have fun with nested iteration:

bv = ByteValue("abcdef")
for value in bv:
	print value
	for second_value in bv:
		print second_value

Conclusion

This concludes our introduction to iterators, iterables, and generators. I hope I have demonstrated the immense power and flexibility that they provide. In general, making your object an iterable object and/or using generators allows for more flexibility than simple iterator objects provide. For most complex classes or complex sets of data, multiple iterations are a given.

With all of the praise that I have heaped upon iterators, iterables, and generators, it’s important to remember that they are not a panacea and should not be used in every case where a sequence of items is needed. There are many instances where a returning a list is the desired result. This being said, iterators, iterables, and generators are extremely useful and provide a great way to loop through data.

[1] http://docs.python.org/tut/node18.html
[2] http://docs.python.org/lib/typeiter.html
[3] http://docs.python.org/api/gen-objects.html

So just to prove that I’m a masochist and that the Dodger Editor is not dead (even though I don’t think that anyone has been using it) I thought I’d post a quick update. No the editor is not dead and no neither am I.

I’m still writing for Python Magazine and working on the editor in some of the free time that I get. I just added a selection cursor mode to the editor which makes it easier for me to use. You can see it in the second row of the pallet in the following screen shot.

Now with a Select Mode!

If you want access to the newest code you can grab it from the mercurial repository: http://freehg.org/u/selsine/dodger/

If anyone is at all interested in this project, whether it’s using or helping to develop, or just some suggestions please let me know. I’m always open to opinions. If I get the time I’d like to work on a short post showing you how you can use the Dodger Editor to create the level files for a game, when that will be I don’t know but hopefully soon.