vanitasvitae's blog

Archive for the ‘xmpp’ Category

GSoC Week 11.5: Success!

Wednesday, August 16th, 2017

Newsflash: My Jingle File Transfer is compatible with Gajim!

The Gajim developers recently fixed a bug in their Jingle Socks5 transport implementation and now I can send and receive files without any problems between Gajim and my test client. Funny side note: The bug they fixed was very familiar to me *cough* :P

Happy Hacking!

GSoC Week 11: Practical Use

Monday, August 14th, 2017

You know what makes code of a software library even better? Client code that makes it do stuff! I present to you xmpp_sync!

Xmpp_sync is a small command line tool which allows you to sync file from one devices to one or more other devices via XMPP. It works a little bit like you might now it from eg. owncloud or nextcloud. Just drop the files into one folder and they automagically appear on your other devices. At the moment it works only unidirectional, so files get synchronized in one direction, but not in the other.

The program has to modes; master mode and slave mode. In general, a client started in master mode will send files to all clients started in slave mode. So lets say we want to mirror contents from one directory to another. We start the client on our master machine and give it a path to the directory we want to monitor. On the other machines we start the client in slave mode, and then add them to the master client. Whenever we now drop a file into the directory, it will automatically be sent to all registered slaves via Jingle File Transfer. Files do also get send when they get modified by the user. I registered a FileWatcher in order to get notified of such events. For this purpose I got in touch with java NIO again.

Currently the transmission is made unencrypted (as described in XEP-0234), but I plan to also utilize my Jingle Encrypted Transports (JET) code/spec to send the files OMEMO encrypted in the future. My plan for the long run is to further improve JET, so that it might get implemented by other clients.

Besides that I found the configuration error in my ejabberd configuration which prevented my Socks5 proxy from working. The server was listening at by default, so the port was not reachable from the outside world. Now I can finally test on my own server :D

I also tested my code against Gajims implementation and found some more mistakes I made, which are now fixed. The Jingle InBandBytestream Transport is sort of working, but there are some more smaller things I need to change.

Thats all for the week.

Happy Hacking :)

GSoC Week 10: Finding that damn little bug

Monday, August 7th, 2017

Finally!! I found a bug which I was after for the last week. Now I finally got that little bas****.

The bug happened in my code for the Jingle SOCKS5 Bytestream Transport (XEP-0260). SOCKS5 proxies are used whenever the two endpoints can’t reach one another directly due to firewalls etc. In such a case, another entity (eg. the XMPP server) can jump in to act as a proxy between both endpoints. For that reason, the initiator (Alice) first collects available proxies, and sends them over to the responder (Bob). The responder does the same and sends its candidates back to the initiator. Both then try to connect to the candidates (in this case proxies) they got sent from their peer. In order for the proxy to know, who wants to talk to whom, both include a destination address, which is calculated as SHA1(sid, providerJid, targetJid), where the provider is the party which sent the candidates to the target.

The alert reader will know, that there are two different destination addresses in the game by now. The first one being SHA1(sid, Alice, Bob) and the second one SHA1(sid, Bob, Alice). The issue is that somewhere in the logs I ended up with 3 different destination addresses. How the hell did that happen. For the answer lets look at an example stanza:

<iq from=’romeo@montague.lit/orchard’
<jingle xmlns=’urn:xmpp:jingle:1′
<content creator=’initiator’ name=’ex’>
<description xmlns=’urn:xmpp:example’/>
<transport xmlns=’urn:xmpp:jingle:transports:s5b:1′
<candidate cid=’hft54dqy’

Here we have a session initiation with a Jingle SOCKS5 Bytestream transport. The transport exists of one candidate. Now where was my error?

You might have noted, that there are two attributes with the name ‘sid’ in the stanza. The first one is the so called session id, the id of the session. This should not be of interest for our case. The second one however is the stream id. Thats the sid that gets crunched in the SHA1 algorithm to produce the destination address.

Well, yeah… In one tiny method used to update my transport with the candidates of the responder, I used session.getSid() instead of transport.getSid()… That was the bug, that cost me a week.

Now it wasn’t too bad. While I searched for the error, I read through the XEPs again and again, discovering some more issues which I discussed with other developers. Also I began testing my implementation against Gajim and I’m happy to tell you that the InBand Bytestream code is already working sort of. Sometimes a few bytes get lost, but we live in times of Big Data, so thats not too bad, am I right :P ?

In the last 3 weeks I plan to stabilize the API some more. Currently you can only receive data into files, but I plan to add another method which gives back a bytestream instead.

Also I need more tests. Things like that nasty sid bug can be prevented and found using junit tests, so I’ll definitely stock up on that front.

Thats all for now :) Happy Hacking!

GSoC Week 9: Bringing it back to life.

Wednesday, August 2nd, 2017

The 9th week of GSoC is there. I’m surprised of how fast time went by, but I guess that happens when you enjoy what you do :)

I’m happy to report that the third iteration of my Jingle code is working again. There are still many bugs and Socks5Transport is still missing, but all in all I’m really happy with how it turned out. Next I’ll make the implementation more solid and add more features like transport replacing etc.

Apart from normal Jingle File Transfer I also started working on my JET protocol. JET is short for Jingle Encrypted Transfers which is my approach to combining Jingle sessions with end-to-end encryption. My focus lays on modularity and easy extensibility. Roughly JET works as follows:

Lets assume, Alice wants to send an encrypted file to Bob. Luckily Alice and Bob already do have a secure OMEMO session. Alice now sends a JET File transfer request to Bob, which includes a security element which contains an OMEMO key transport message. Bob can decrypt the key transport message with his OMEMO session to retrieve an AES key. That key will be used to encrypt/decrypt the file Alice sends to Bob as soon as the jingle session negotiation is finished.

This protocol should in theory work with any end-to-end encryption method, for example also with OpenPGP. Also JET is in theory not limited to file transfer, but could also be used to secure other types of Jingle sessions, eg. Audio/Video calls. Since the development is in a very early state, it would be nice to get some feedback from more experienced developers and members of the XMPP community. A rendered version of the JET specification can be found here.

I’m very happy that encrypted File transfer already works in my implementation. I created an integration test for that, which transports the encryption key using OMEMO. Apropos tests: I created a basic JingleTransport test, which tests transport methods. Currently SOCKS5 is still failing, but I’m very close to a solution.

During the week I opened another PR against the XEPs repo, which adds a missing attribute to a XML schema in the Jingle File Transfer XEP.

GSoC Week 8: Reworking

Wednesday, July 26th, 2017

The 8th. week of GSoC is there. The second evaluation phase has begun.

This week I was not as productive as the weeks before. This is due to me having my last (hopefully :D ) bachelors exam. But that is now done, so I can finally give 100% to coding :) .

I made some more progress on my Jingle code rework. Most of the transports code is now finished. I started to rework the descriptions code which includes the base classes as well as the JingleFileTransfer code. I’m very happy with the new design, since it is way more modular and less interlocked than the last iteration. Below you can find an UML-like diagram of the current structure.

UML diagram of the jingle code

UML-like diagram of the current jingle implementation

During the rework I stumbled across a slight ambiguity in the Jingle XEP(s), which made me wonder: There are multiple jingle actions, which denote the purpose of a Jingle stanza (eg. transport-replace to replace the used transport message, session-initiate to initiate a session (duh) and so forth). Now there is the session-info Jingle action, which is used to announce session specific events. This is used for example in RTP sessions to let the peers client ring during a call, or to send the checksum of a file in a file transfer session. My problem with this is, that such use-cases are in my opinion highly description related and instead the description-info action should be used. The description part of a Jingle session is the part that represents the actual purpose of the session (eg. file transfer, video calls etc.).

The session itself is description agnostic, since it only bundles together a set of contents. One content is composed of one description, one transport and optionally one security element. In a content you should be able to combine different description, transport and security components in an arbitrary way. Thats the whole purpose of the Jingle protocol. In my opinion it does no make much sense to denote description related informational stanzas with the session-info action.

My proposal to make more use of the description-info element is also consistent with other uses of *-info actions. The transport-info action for example is used to denote transport related stanzas, while the security-info action is used for security related information.

But why do I even care?

Lets get back to my implementation for that :) . As you can see in the diagram above, I split the different Jingle components into different classes like JingleTransport, JingleSecurity, JingleDescription and so on. Now I’d like to pass component-related jingles down to the respective classes (a transport-info usually only contains information valuable to the transport-component). I’d like to do the same for the JingleDescription. At the moment I have no real “recipient” for the session-info action. It might contain session related infos, but might also be interesting for the description. As a consequence I have to make exceptions for those actions, which make the code more bloated and less logical.

Another point is, that such session-info elements (due to the fact that they target a single content in most cases) often contain a “name” attribute that matches the name of the targeted content. I’d propose to not only replace session-info with description-info, but also specify, that the description-info MUST have one or more content child elements that denote the targeted contents. That would make parsing much easier, since the parser always can expect content elements.

That’s all for now :)

Happy Hacking!

GSoC: Week 7

Tuesday, July 18th, 2017

This is my update post for the 7th week of GSoC. The next evaluation phase is slowly approaching and it is still a lot of work to do.

This week I started to work on integrating encryption in my Jingle File Transfer code. I’m very pleased, that only very minor changes are required to my OMEMO code. The OmemoManager just has to implement a single interface with two methods and that’s it. The interface should be pretty forward to implement in other encryption methods as well.

Unfortunately the same is not true for the code I wrote during the GSoC. There are many things I haven’t thought about, which require major changes, so it looks like I’ll have to rethink the concept once again. My goal is to make the implementation so flexible, that description (eg. video chat, file transfer…), transport (eg. Socks5, IBB…) and security (XTLS, my Jet spec etc.) can be mixed in arbitrary ways without adding in glue code for new specifications. Flow told me, this is going to get complicated, but I want to try anyway :D . For “safety” reasons, I’ll keep a seperate working branch in case the next iteration does not turn out as I want.

Yesterday Flow found an error in smack-omemo, which caused bundles not getting fetched. The mistake I made was, that a synchronous CarbonListener was registered in the packet-reader thread. This caused the packet-reader thread to timeout on certain messages, even though the stanzas arrived. It is nice to have this out of the way and it was a good lesson about the pitfalls of parallel programming.

While reading the Jingle File Transfer XEP I also found some missing XML schemes and proposed to replace the usage of xs:nonNegativeInteger and xs:positiveNumber with xs:unsignedLong to simplify/unify the process of implementing the XEP.

Thats basically it for this week. Unfortunately I have an upcoming exam at university next week, which means even less free time for me, but I’ll manage that. In the worst case I always have a second try on the exam :)

Happy Hacking!

GSoC Week 6 – Tests and Excitement

Monday, July 10th, 2017

Time is flying by. The sixth week is nearly over. I hope I didn’t miscounted so far :)

This week I made some more progress working on the file transfer code. I read the existing StreamInitialization code and found some typos which I fixed. I than took some inspiration from the SI code to improve my Jingle implementation. Most notably I created a class FileTransferHandler, which the client can use to control the file transfer and get some information on its status etc. Most functionality is yet to be implemented, but at least getting notified when the transfer has ended already works. This allowed me to bring the first integration test for basic Jingle File transfer to life. Previously I had the issue, that the transfer was started as a new thread, which was then out of scope, so that the test had no way to tell if and when the transfer succeeded. This is now fixed :)

Other than that integration test, I also worked on creating more junit tests for my Jingle classes and found some more bugs that way. Tests are tedious, but the results are worth the effort. I hope to keep the code coverage of Smack at least on a constant level – already it dropped a little bit with my commits getting merged, but I’m working on correcting that again. While testing, I found a small bug in the SOCKS5 Proxy tests of Smack. Basically there were simultaneous insertions into an ArrayList and a HashSet with a subsequent comparison. This failed under certain circumstances (in my universities network) due to the properties of the set. I fixed the issue by replacing the ArrayList with a LinkedHashSet.

Speaking of tests – I created a “small” test app that utilizes NIO for non-blocking IO operations. I put the word small in quotation marks, because NIO blows up the code by a factor of at least 5. My implementation consists of a server and a client. The client sends a string to the server which than 1337ifies the string and sends it back. The goal of NIO is to use few Threads to handle all the connections at once. It works quite well I’d say. I can handle around 10000 simultaneous connections using a single thread. The next steps will be working NIO into Smack.

Last but not least, I once again got excited about the XMPP community :)
As some of you might now, I started to dig into XMPP roughly 8 months ago as part of my bachelors thesis about OMEMO encryption. Back then I wrote a mail to Daniel Gultsch, asking if he could give me some advice on how to start working on an OMEMO implementation.
Now eight months later, I received a mail from another student basically asking me the same question! I’m blown away by how fast one can go from the one asking to the one getting asked. For me this is another beautiful example of truly working open standards and free software.

Thank you :)

GSoC Week 5: Tests, fallbacks and politics

Tuesday, July 4th, 2017

This is my blog post for the 5th week of the Google Summer of Code. I passed the first evaluation phase, so now the really hard work can begin.
(Also the first paycheck came in, and I bought a new Laptop. Obviously my only intention is to reduce compile times for Smack to be more productive *cough*).

The last week was mostly spent writing JUnit tests and finding bugs that way. I found that it is really hard to do unit tests for certain methods, which might be an indicator that there are too many side effects in my code and that there is room to improve. Sometimes when I need to save a state as a variable from within a method, I just go the easy way and create a new attribute for that. I should really try to improve on that front.

Also I did try to create an integration test for my jingle file transfer code. Unfortunately the sendFile method creates a new Thread and returns, so I have no real way of knowing when the file transfer is complete for now, which hinders me from creating a proper integration test. My plans are to go with a Future task to solve this issue, but I’ll have to figure out the most optimal way to bring Futures, Threads, (NIO) and the asynchronous Jingle protocol together. This will probably be the topic of the second coding phase :)

The implementation of the Jingles transport fallback functionality works! When a transport method (eg. SOCKS5) fails for some reason (eg. proxy servers are not reachable), then my implementation can fallback to another transport instead. In my case the session switches to an InBandBytestream transport since I have no other transports implemented yet, but theoretically the fallback method will try all available transports in the future.

I started on creating a small client/server application that will utilize NIO to handle multiple connections on a single thread as a small apprentice piece. I hope to get more familiar with NIO to start integrating non-blocking IO into the jingle filetransfer code.

In the last days I got some questions on my OMEMO module, which very nicely illustrated to me, that developing a piece of software does not only mean to write the code, but also maintain it and support the people that end up using it. My focus lays on my GSoC project though, so I mostly tell those people how to fix their issues on their own.

Last but not least a sort of political remark: In the end of the GSoC, students will receive a little gift from Google (typically a Tshirt). Unfortunatelly not all successful students will receive one due to some countries being “restricted”. Among those countries are Russia, Ukraine, Kazakhstan, but also Mexico. It is sad to see, that politics made by few can affect so many.

Working together, regardless of where we come from, where we live and of course who we are, that is something that the world can and should learn from free software development.

Happy Hacking.

GSoC Week 4.1

Wednesday, June 28th, 2017

This is a quick update post to my post from yesterday. The GSoC part came a little short because of my OMEMO reflections and because I only worked on one topic the last week, so this is intended as a kind of supplement.

Today I got my SOCKS5 code working again! I can send and receive both from and to local and external proxy servers. It was previously not working because I missread the XEP. I thought that one party only connects to a proxy after the activation element was received, while in fact the connection must be established beforehand. Also on the sender side I closed my OutputStream after writing the data on it. This caused the receiver to fail opening an InputStream because the socket was closed. It took some time and Googling to find out that closing the OutputStream also closes the socket. Unfortunately I’m not having a “established” implementation which I can test my code against. If you know of an app/messenger which supports Jingle FT 5, please let me know :)

My next steps will be writing integration tests. At some point I lost track of tests, so I’m a little bit behind on that front. Creating those tests should greatly improve the stability of the code and make breaking changes more obvious. Also I’ll create a little NIO test application to make myself familiar with the NIO framework. Currently my focus lays on getting things to work in the first place, but in the end it would be nice to have data streams implemented using NIO. Another thing that I’ll tackle this week is getting jingles fallback mechanism in case of a failed transport to work. This should be pretty straight forward.

Then again I’ll hopefully soon spent some more time again on my plans for Jingle Encrypted Transfers. I hope to get a working prototype after the general Jingle File Transfer is finished.

Thats all for now. Happy Hacking :)

Fourth week of GSoC and OMEMO thoughts

Wednesday, June 28th, 2017


Evaluation phase is there! Time went by faster than I expected. That’s a good sign, I really enjoy working on Smack (besides when my code does not work :D ).

I spent this week to work on the next iteration of my Jingle code. IBB once again works, but SOCKS5 still misses a tiny bit, which I struggle to find. For some reason the sending thread hangs up and blocks just before sending begins. Its probably a minor issue, which can be fixed by changing one line of code, nevertheless I’m struggeling to find the solution.

Apart from that it turned out, that a bug with smack-omemo which I was earlier (unsuccessfully) trying to solve resurrected from the land of closed bug reports. Under very special conditions pubsub results seem to only arrive in Smack exactly after the result listener timed out. This keeps smack-omemo from fetching device bundles on some servers. I originally thought this was a configuration issue with my server, but it turned out that aTalk (a Jitsi for Android fork which is working on including OMEMO support) faces the exact same issue. Looks like I’ll have to investigate this issue once more.

OMEMO – Reflections

It appears that soon the council will decide on the future of OMEMO. I really hope, that the decision will make everyone happy and that the XMPP community (and – perhaps more important – the users) will benefit from the outcome.

There are multiple options for the direction, which the development of OMEMO can take. While everybody aggrees, that something should happen, it is not clear what.

OMEMO is already rather well deployed in the wild, so it is obviously not a good idea to break compatibility with existing implementations. A few months ago, OMEMO underwent a very minor protocol change to mitigate a vulnerability and even though Conversations (birthplace to OMEMO) made a smooth transition over multiple versions, things broke and confused users. While things like this probably cannot be avoided in a protocol which is alive and changing, it is desirable to avoid breaking stuff for users. Technology is made for users. Keeping them happy should be highest priority.

However, at the moment not every user can benefit from OMEMO since the current specification is based on libsignal, which is licensed under the GPLv3. This makes the integration in premissively licensed software either expensive (buying a license from OWS), or laborious (substitute libsignal with other double ratchet library). While the first option is probably unrealistic, the second option has been further investigated during discussions on the standards mailing list.

Already the “official” specification of OMEMO is based on the Olm library instead of libsignal. Olm more or less resembles OWSs double ratchet specification, which has been published by OWS roughly half a year ago. Both Olm and libsignal got audited, while libsignal got significantly more attention both in the media and among experts. One key difference between both libraries is, that libsignal uses the Extended Triple Diffie Hellman (X3DH) key exchange using the XEdDSA signature scheme. The later allows a signature key to be derived from an encryption key, which spares one key. In order to use this functionality, a special conversion algorithm is used. While apparently this algorithm is not too hard to implement, there is no permissive implementation available in any established, trusty crypto library.

In order to work around this issue, one proposal suggests to replace X3DH completely and switch to another form of key exchange. Personally I’m not sure, whether it is desirable to change a whole (audited) protocol in order to replace a single conversion algorithm. Given the huge echo of the Signal protocol in the media, it is only a matter of time until the conversion algorithm makes its way into approved libraries and frameworks. Software follows the principle of supply and demand and as we can conclude from the mailing list discussion, there is quite a lot of demand even alone in the world of XMPP.

I guess everybody aggrees that it is inconvenient, that the “official” OMEMO XEP does not represent, what is currently implemented in the real world (siacs OMEMO). It is open to debate now, how to continue from here on. One suggestion is to document the current state of siacs OMEMO in a historical XEP and continue development of the protocol in a new one. While starting from scratch offers a lot of new possibilities and allows for a quicker development, it also most definitely implies to completely break compatibility to siacs OMEMO at users expense. XMPP is nearly 20 years old now. I do not believe that we are in a hurry :) .

I think siacs OMEMO should not get frozen in time in favor of OMEMO-NEXT. Users are easily confused with names, so I fear that two competing but incompatible OMEMOs are definitely not the way to go. On the other hand, a new standard with a new name that serves the exact same use case is also a bad idea. OMEMO works. Why sould users switch? Instead I’d like to see a smooth transition to a more developer friendly, permissively implementable protocol with broad currency. Already there are OTR and OpenPGP as competitors. We don’t need even more segmentation (let alone the same name for different protocols). I propose to ditch the current official OMEMO XEP in favor of the siacs XEP and from there on go with Andreas’ suggestion, which allows both libsignal as well as Olm to be used simultaneously. This allows permissive implementations of OMEMO, drives development of a permissively licensed conversion algorithm and does not keep users standing in the rain.

To conclude my reflections: Users who use OMEMO will use OMEMO in two or three years. It is up to the community to decide, whether it will be frozen “historical” siacs OMEMO, or collectively developed, smoothly transitioned and unified OMEMO.

Thank you for your time :)