ff

It's common understanding that XMPP needs compression for mobile connections and periodically somebody starts shouting "we need binary XML"! Reactions are always mixed and at the end almost no step forward is taken, since the subject is really tough and XML and binary don't match well. Moreover there are common misunderstandings always firing the usual flames:

• most binary encodings for XML work well since they already know in advance the vocabulary of the tokens used in the stream; with XMPP this is only partially true, because XMPP is an extensible stream of XML elements and only few - the core - are know in advance;
• the more a binary encoding based on lookup tables is able to compress, the more is the memory that it uses, and this is not friendly to mobile devices;
• some binary encodings such as EXI do use zlib in order to be effective, so the gain is mainly due to the fact that they rearrange the data in order to be more compressible (therefore we have a double overhead of memory);
  
• it's not completely true that binary encoding is much simpler than an XML parser, since if you want to keep both extensibility and compression your state machine becomes pretty complex (see EXI specs...)
• almost no binary encoding takes in account that also some attribute values could be read from a lookup table (e.g. JIDs, which are always the same, and they are where you find most of XMPP redundancy),  thus adding further complexity to the parser.

So what? Are "binary" and "XMPP" incompatible? Do we need binary parsers when zlib + XML behaves pretty well (less then 10%  of the original size) and other improvements such as partial roster synchronization a privacy lists can cut much more traffic? The answer to the second question, I think, is yes if you think of mobile devices: they are very limited, and XML + zlib is a combination taking a lot of memory and cpu cycles (i.e. battery!). This is one of the reasons for which the guys of Android ditched XMPP for the moment: (accordingly to them) the power you save sending less data is lost by compression. 

Hence my idea: what about a binary encoding that only simplifies the parsing, without any compression, and then we apply zlib at the top of it? How does it perform in terms of bandwidth, memory, computational complexity?

The encoding is very trivial, it's inspired to ASN.1, and any token is sent using this encoding: (type, length, value). There are few optimization such as packing type and length in a single byte when values are short, but I'll describe the details in another post (the encoding still needs many improvements). What is important, is that with this encoding you don't compress a lot (you just save the ending tags and xml punctuation is compensated by the type/length bytes), but the parser is super simple and fast. Just to give the figures: a j2me parser, with a class defining basic dom elements is less then 2.9 KB of bytecode and it has been written in an 1h trip on the train ;) Moreover, since the encoding isn't aimed at compression, it's possible to send any element in any namespace, thus preserving all the nice features of XMPP.

Now the  results with a set of about 25K stanzas taken from a server log (j2me parser):

And the winner is... bin + zlib!

bin + zlib compresses almost as well  as xml + zlib, but with the incredible advantage of being much faster than raw xml without compression! 

Last word about memory: I haven't measured it, but the binary parser should perform much better , since it doesn't use dynamic memory blocks, because it knows in advance the size of each token it is going to read (no more vectors and string buffers, but only arrays and fixed strings)