ff

It's common understanding that XMPP needs compression for mobile connections and periodically somebody starts shouting "we need binary XML"! Reactions are always mixed and at the end almost no step forward is taken, since the subject is really tough and XML and binary don't match well. Moreover there are common misunderstandings always firing the usual flames:

• most binary encodings for XML work well since they already know in advance the vocabulary of the tokens used in the stream; with XMPP this is only partially true, because XMPP is an extensible stream of XML elements and only few - the core - are know in advance;
• the more a binary encoding based on lookup tables is able to compress, the more is the memory that it uses, and this is not friendly to mobile devices;
• some binary encodings such as EXI do use zlib in order to be effective, so the gain is mainly due to the fact that they rearrange the data in order to be more compressible (therefore we have a double overhead of memory);
  
• it's not completely true that binary encoding is much simpler than an XML parser, since if you want to keep both extensibility and compression your state machine becomes pretty complex (see EXI specs...)
• almost no binary encoding takes in account that also some attribute values could be read from a lookup table (e.g. JIDs, which are always the same, and they are where you find most of XMPP redundancy),  thus adding further complexity to the parser.

So what? Are "binary" and "XMPP" incompatible? Do we need binary parsers when zlib + XML behaves pretty well (less then 10%  of the original size) and other improvements such as partial roster synchronization a privacy lists can cut much more traffic? The answer to the second question, I think, is yes if you think of mobile devices: they are very limited, and XML + zlib is a combination taking a lot of memory and cpu cycles (i.e. battery!). This is one of the reasons for which the guys of Android ditched XMPP for the moment: (accordingly to them) the power you save sending less data is lost by compression. 

Hence my idea: what about a binary encoding that only simplifies the parsing, without any compression, and then we apply zlib at the top of it? How does it perform in terms of bandwidth, memory, computational complexity?

The encoding is very trivial, it's inspired to ASN.1, and any token is sent using this encoding: (type, length, value). There are few optimization such as packing type and length in a single byte when values are short, but I'll describe the details in another post (the encoding still needs many improvements). What is important, is that with this encoding you don't compress a lot (you just save the ending tags and xml punctuation is compensated by the type/length bytes), but the parser is super simple and fast. Just to give the figures: a j2me parser, with a class defining basic dom elements is less then 2.9 KB of bytecode and it has been written in an 1h trip on the train ;) Moreover, since the encoding isn't aimed at compression, it's possible to send any element in any namespace, thus preserving all the nice features of XMPP.

Now the  results with a set of about 25K stanzas taken from a server log (j2me parser):

And the winner is... bin + zlib!

bin + zlib compresses almost as well  as xml + zlib, but with the incredible advantage of being much faster than raw xml without compression! 

Last word about memory: I haven't measured it, but the binary parser should perform much better , since it doesn't use dynamic memory blocks, because it knows in advance the size of each token it is going to read (no more vectors and string buffers, but only arrays and fixed strings)

Here at Bluendo Labs we are always committed to offering a great experience in XMPP for mobile clients and for this reason we embrace the best available technologies. Since everybody knows that XMPP shouldn't be used in mobile connections, for the verboseness of XML,  Pedro Melo  at last DevCon in February had the brilliant idea of a new binary transport for XMPP.  We loved the idea since the very beginning, and during the last month we have worked hard (together with Pedro and the invaluable guide of Peter) in order to offer you the first implementation of this technology. Take a look at the Google Code project for trying it and for learning more about this technological breakthrough.

I'm reading the developer's guidelines for Ning platform (yep I'm studying and experimenting for this) and the first point is the sum-up of the usual defect of all web 2.0 collaboratives sites, i.e. lack of realtimess:

Require user interaction for information/data reload. Scripts or other types of mechanisms (including Flash widgets, META refresh tags, and similar) generate load regardless of whether a user is navigating the site or not. For example, at this time, we do not support polling applications running on the platform. Pull of information based on user actions should be the default behavior of your code on Ning.

Users must be logged in do something in order to pull data, that's to say they must waste time just to see if there is something interesting... Luckily there are also technologies which are more time friendly, taking your time only when there is something new interesting you have expressed interest in, and that you can access from anywhere.

Few weeks ago I wrote about xmpp widgets, which are a concept similar to Pedro Melo's Xiclets, or to the scriplets  you can find in  Massimiliano Mirra's Sameplace, and about which Adam Nemeth is brainstorming too.

Since then few things happened. These criminal minds met at the DevCon at FOSDEM and they decided that some common action should taken. The scope is not writing a XEP or a set of XEPS, since we are not dealing with protocol aspects. What we would like to investigate is the definition of a set of API and of an execution environment allowing embedding XMMP applications in clients (either in desktop or in web based clients).

The reference technology should be XHTML + Javascript, while XMPP will have the role of transport. This is not, as it may appear, the reinvention of the wheel using XMPP at the place of HTTP, but we will try to bring in presence, real time messaging and contact lists (the social graph) in order exploit the real potential of XMPP in social applications.

If we were fond of buzzwords we could have called them Web 3.0, instead we agreed with the less ambitious temporary name of xiclets, as Pedro named them. So, if you are interested, you are welcome to the XMPP API mailing list (a even more under cover name ;)) that Peter has kindly created for us.