ff

Time ago I started writing a tutorial for the XMPP support in Twisted, but for technical problems I had to suspend it. I will resume it as soon as I can.

The problem, I noticed from several requests I've received recently, is that this library is not very friendly for beginners. Twisted is not the easiest framework to learn, since you need to modify many of you programming habits; the word.protocols.jabber package has gone under heavy development and there is a bit of confusion about which version to use for which XMPP server. And how can I do with Google Talk? Is it possible to connect to? Are there any examples?

First few words about XMPP and Google Talk.  Historically XMPP, when the protocol was named Jabber, had a different authentication method, based on <iq/> messages in the "jabber:iq:auth" namespace. Now that method is deprecated and maintained by few servers for legacy reasons, the only supported authentication method is SASL, which is well described in RFC 3920. SASL indeed is a general framework for plugging in different authentication mechanisms, and among them the most used in XMPP are PLAIN and MD5-DIGEST. Google Talk is 100% XMPP and it supports two authentication mechanisms: PLAIN and GOOGLE-TOKEN, no digest since Google Talk allows TLS connections only. With Google Talk you must only pay attention to one detail: differently from many XMPP servers the realm you are authenticating to (gmail.com) is different from the physical server you are connecting to. Which is the server? It's talk.google.com, port 5222, but you can't rely on this. There is standard way to discover the host, and you should use it: clients should query the "xmpp-client" SRV record of the desired domain (e.g. gmail.com). 

So, getting back to Twisted, where do I start from?  Though many Linux distributions still come with Twisted 2.4 your choice is Twisted 2.5, which is the only one implementing SASL and being able to connect to standard compliant servers, such as Google Talk. This version has an example that shows one possible way to write a client: xmpp_client.py, in the words package examples. This example uses a SRV dns query in order to discover the host to connect to, so it can connect to gmail. So just download Twisted 2.5, install it and the try the xmpp_client.py with this command line:

python xmpp_client.py user@gmail.com yourpasswword 

If everything works well, you should see a dump of the exchanged packets with server.

As you can see, picking the right version and the right example, the startup time for using   and understanding Twisted is almost as low as for the other XMPP libraries. And once you have become familiar, I think you'll start loving Twisted as I do.

This is the first part of a tutorial in which we will learn how to use Python and Twisted for making Jabber services. Besides showing how to use the great Twisted support for XMPP, the tutorial will embrace few aspects that go beyond instant messaging and that could make Jabber the backbone of the real time Internet:

• data forms

• an initial implementation of ad hoc commands
  

• transport of RSS and Atom feeds in XMPP stanzas

Moreover we will also learn something about Twisted and databases, in particular embedded databases such as SQLite and the object based ZopeDB.

In order to follow this tutorial you need some basic knowledge of Python and Twisted.

In this first installation we take a look at the basics of twisted.words.protocols.jabber, and we will write our first Jabber client. The examples will use the code from the xmpp-subprotocols branch of Twisted, which is far better than the code in trunk. In order to obtain it issue the following command and then run setup.py:

svn co svn://svn.twistedmatrix.com/svn/Twisted/branches/xmpp-subprotocols-2178-2

The business logic of XMPP clients is implemented as an XMPPHandler, an object that has the xmlstream attribute allowing to send packets and to register observers for receiving incoming stanzas. This object is notified when the connection is ready and the client has been successfully authenticated by calling the connectionInitialized() method. This is an example of the code needed for starting listening to presence stanzas:

from twisted.words.protocols.jabber import xmlstream

class GMailChecker(xmlstream.XMPPHandler):

    def connectionInitialized(self):
        self.xmlstream.addObserver(
            "/presence",
            self.got_presence
        )

    def got_presence(self):
        pass

Form the example you can that all you need for receiving packets is ti register an “observer” using an XPath query. Therefore got_presence will be called each time a stanza whose root element is /presence is received. Easy isn't it? No need to write receiving threads or complex state machines for taking in account all the possible packets. Just register the observers for the stanzas you are interested in and you will be called when a packet is ready. All the observers are also thread safe, since basically there aren't threads: in most cases you can live (a better life) without them. I haven't written “all cases” just because there still exist some limited situations in which you need some thread, and we will consider also them in this tutorial.

And what about sending packets? It's as easy as receiving them. The xmlstream attribute has a send() method allowing you sending chunks of XML. You can build those chunks with the Element objects supplied by the twisted.words.xish.domish package. Here is the code for building a message and sending it:

 def connectionInitialized(self):

        .... 

        msg = Element(("jabber:client", "message"))
        msg["to"] = "ff@jabber.bluendo.com"
        body = msg.addElement("body", content = "Hi Fabio!")
        self.xmlstream.send(msg)

Now we know the very basics for handling an XMPP connection, but we still need to create it. Twisted will take care of all the details, we just need to tell it to start connecting and use our handler once logged in. In order to accomplish this, we need three objects (in this aspect Twisted is a bit complicated, but the separation of roles that is introduced gives a lot flexibility): a factory for building a client xml stream, the manager of handlers for associating them to the stream, and Internet client for physically connecting to the server. Here is the resulting code:

 if __name__ == "__main__":
    from twisted.words.protocols.jabber import client, jstrports
    from twisted.words.protocols.jabber.jid import JID
    from twisted.internet import reactor

    jid = JID("xxx@jabber.bluendo.com")
    cf = client.XMPPClientFactory(jid, "xxx")
    sm = xmlstream.StreamManager(cf)
    sm.addHandler(GMailChecker())

    client_svc = jstrports.client("tcp:%s:5222"%("jabber.bluendo.com"), cf)
    client_svc.startService()
    reactor.run()

Just replace the jabber id and the password with your account, and you be able to login and send you message. Here you may find the full code of the example: http://www.bluendo.com/~fabio/tutorial/part1.py

In the next installation we will improve our client, making it a bit more Twisted like and we will start adding some functionality.

Take a look at TGWebservices: TGWebServices provides a super simple API for creating web services that are available via SOAP, HTTP->XML, and HTTP->JSON. The SOAP API generates WSDL automatically for your Python and even generates enough type information for statically typed languages (Java and C#, for example) to generate good client code on their end.

Do I hear somebody whispering "where are ad-hoc commands"?

It's a pity that Twisted hasn't a decent support for asynchronous db api. All you can do is to wrap transactions in separate thread taken from a thread pool, and you if you are used to the cleanness of twisted APIs it is really pain in the ass.

 Update: an interesting link about ZODB vs SQLite: ZODB is far superior in terms of locking capabilities, though it has some drowbacks too. Take a look also at the comments, since there are many interesting links.

While "connecting things" one of the most requested features is automatic configurations. Why setting up accounts, contacts and server addresses, when you continously add or remove nodes or when you move them to different places? Jabber already has the answer, and its name is rendevouz, used in many client implementations.

Here are some interesting readings:

•  Bonjour iChat's use of XMPP(Gareth Simpson)
•  Zeroconf and Python
•  iChat and Jabber(Julian Missing)