Paul Boddie's Free Software-related blog

Daniel Pocock asks whether supporting the Software Freedom Conservancy is the right thing to do, given the recent announcement of a fund-raising drive inviting individuals to sustain the organisation’s activities. The short answer is “yes it is”, but the question and the longer answer are still worth thinking about.

An Overview

A certain focus has been placed on the Conservancy’s licensing compliance activities, which are valuable for a number of reasons that we shall consider in a moment, but let us also consider the other work done by the organisation:

• It acts as a kind of umbrella organisation for a variety of Free Software projects, offering services and support in areas such as finance and law.
• Together with other organisations, it provides licence compliance advice and best practices for Free Software developers.
• Also together with other organisations, it attempts to remove uncertainty and risk around the way certain Free Software distributions have been shared, as well as seeking to neutralise the hostile legislative environment that continues to threaten customer and end-user freedoms.
• It also seeks to address the lack of Free Software accounting solutions for non-profit organisations.

Although other organisations exist to look after Free Software projects in certain ways, many only offer technical facilities to those projects, whereas others rely on copyright assignments or comparable instruments in order to act as stewards for those projects. Unusually, the Conservancy instead offers a framework where projects may delegate responsibility for activities that would otherwise take time away from the vital work of developing software, rather than assuming all responsibility and leadership for a project as a starting point for cooperation.

So, by working with the Conservancy, developers may retain their project’s autonomy while being able to get help from the Conservancy when they need it. Indeed, the merits of the Conservancy’s offerings complement the offerings of other organisations in such a way that Debian has chosen to work more closely with the Conservancy to safeguard the interests of those developers making their work available via the Debian software distribution.

The image of member project logos gives a representative indication of the organisation’s influence and importance in the Free Software world today. Many of these projects provide vital infrastructure and tools that Free Software users and developers rely upon every single day.

Compliance and Enforcement

But what about those licensing compliance or licensing enforcement activities undertaken by the Conservancy? Some people might wonder whether there is a real need to ensure that individuals and organisations adhere to Free Software licences, and if they do not, whether it is worthwhile taking those parties to task on such matters. Others, arguably with their own agenda, may even dislike the very idea of bringing anyone to account for not respecting the Free Software licensing of various works.

First of all, we must ask whether Free Software licences are being violated. The sad answer to this is “on an industrial scale“, given the glut of products being manufactured on the back of Free Software and then sold without even notifying customers of their rights. When companies are approached about the source code for the copyleft-licensed software provided in their products, it is by all accounts a rare occurrence to be directed to a well-managed repository of code that can be built, installed and used on the product. If one is lucky, a hastily-prepared bundle of sources might be thrown over the wall, leaving the enquirer with the task of verifying that it really does generate the originally-shipped software.

And beyond those more favourable outcomes is the case of the mystery “original design manufacturer”, who was merely passing on stuff concocted by the platform vendor, with everybody else insisting that they hardly touched anything and that the software is someone else’s responsibility. Or the manufacturer who declares that they are not affected by the software licensing and that anything they find on the Internet is presumably fair game to use as they please.

Now, some people would advise Free Software developers not to expect too much after contributing anything to a project. Such people would probably also advise developers to use permissive licences: that way, they won’t build up any expectations around what people might do with their work, nor hold out any hopes that others might benefit from seeing the source code down the line.

Certainly, it rather suits some of those people to cultivate the notion that getting one’s code out there into widespread use should be the principal reward for a Free Software developer, not because it actually encourages generosity or delivers a sense of satisfaction or recognition, but because it keeps those developers in their place and discourages them from expecting anything more. Meanwhile, various companies do very nicely out of repackaging and selling such code, denying end-users any insight into – or control over – the code they end up using, and (of course) denying them the right to give away or sell such code to others themselves.

When choosing to use a copyleft licence, Free Software developers are making a valid statement: they are actively stating that anyone who receives their software should enjoy the benefits of being able to modify, install, run and redistribute it to others who would also benefit from it. This is nothing that anyone should be ashamed of, nor should it be something that people should be forced to abandon because others (for whatever reason) do not share the same goals or vision. But at the same time, it perhaps requires more attention to be paid towards those redistributing the software. If others fail to uphold the licence, there needs to be some mechanism in place to demand a remedy.

Some people are obviously never going to like the idea of licence enforcement. For a start, licence violators are not going to like it: it means that they can no longer get away with their shoddy engineering practices and turning a quick profit on code they happened to find online. It has previously been made very apparent that apologists for licence violators are likely to claim that licence enforcement will only “scare away business from open source” (being of the ideological persuasion that considers “open source” as a business productivity tool, as opposed to Free Software which is about end-user freedoms), and they also tend to advocate for more permissively-licensed software so that it will be virtually impossible for the average software outfit not to accidentally clear the resulting lower threshold for licence compliance.

But why should Free Software developers care about the convenience of blatantly profiteering, inept and often hostile companies? No-one forces those companies to use the software, and if they don’t like the licensing terms, they can always go and use something else. The problem here is firmly with the companies in question (and their apologists): they really want to use such software, but they also want to behave as if they own it, all so that they get to decide what kind of licence it might have, and all without having done the hard work of actually writing it themselves. In short, they want it all! Well, forgive the rest of us for not giving them the ingredients of a charmed existence on a silver platter!

Investments of Time and Money

Unfortunately, chasing up licence violators is costly in terms of time and money. The Conservancy actually takes a very gentle approach to seeking licensing compliance when you consider that other people accused of copyright infringement can expect hostile industry bodies working law-enforcement agencies like puppets and performing on-the-spot “audits” (not to mention the endless barrage of messaging about “piracy” aimed at individuals).

Here, Daniel gets on to something fairly important. While certain figures promote the virtues of volunteering one’s own time to write “open source” software, presumably around a day job which does not reward the average developer for writing Free Software, certain aspects of software development and distribution cannot be so easily covered by spontaneous volunteer contributions. Money is required, but that money has to come from somewhere. Again, people can be persuaded to donate their own money (alongside their own time) to help make things happen, but that money also has to come from somewhere.

Sadly, with the cultivation of the notion of the noble volunteer, together with the misguided idea that “open source” be promoted as the cheap or free-of-charge alternative to “commercial” software, the realm of Free Software development – as far as community-centred projects, not corporate projects, are concerned – has been left chronically underfunded. And when many corporate participants prioritise their own interests, the result is a funding gap that leaves vital projects undone or unfinished and a more general sustainability problem around how such projects may be started, staffed and supported.

Lately, I have read a few articles about people burning out, perhaps because they took on too much work, and perhaps because they believed that their “marketable skills” would be enhanced by a heavy portfolio of volunteer responsibilities, making them attractive to potential employers. Again, the interests of profit-making businesses are put before the needs or values of the individual, with the individual even feeling obliged to make this so. Indeed, there are commercial interests which gain from Free Software remaining perpetually underfunded: proprietary software vendors can portray Free Software solutions as being less capable and somehow worth less. That results in Free Software projects, whose offerings would be improved and more competitive with more available revenue, actually getting less and less funding, interest and support over time.

We should not be pandering to the interests of those who are effectively impoverishing us, degrading our life quality, or forcing us to choose between the things we believe in and the means to be able to live a decent-enough life. Quite how we can develop a sustainable stream of funding for projects that would benefit everybody, along with forms of organisation where such actual work may be undertaken, is a topic for another time. However, one way of stopping the exploitation of developers is to uphold the licences through which those developers have shared their contributions, and that requires us to realise that such efforts also require ongoing funding to become viable and to remain so.

So, of course, I believe that supporting the Software Freedom Conservancy is the right thing to do. And beyond the good work that is done by that organisation, sustained by what is effectively an investment in the continued viability of Free Software in a hostile world, I hope that people will gradually realise that investment is also more generally needed to sustain the creation and maintenance of Free Software as well.

I thought it might be helpful if I wrote a quick note about my previous work on Debian packaging and Kolab. As my blog can attest, I have written a few articles about packaging and the effort required to make Kolab somewhat more amenable to a convenient and properly functioning installation on Debian systems. Unfortunately, perhaps due to a degree of overambition and perhaps due to me being unable to deliver a convincing and/or palatable set of modifications to achieve these goals, no progress was made over the year or so I spent looking at the situation. I personally do not feel that there was enough of a productive dialogue about aligning these goals with those of the core developers of Kolab, and despite a certain level of interest from others, I no longer have the motivation to keep working on this problem.

Occasionally, I receive mails from people who have read about my experiments with Debian packaging or certain elements of Kolab configuration that became part of the packaging work. This message is intended to communicate that I am no longer working on such things. Getting Kolab to work with other mail transport/delivery/storage systems or other directory systems is not particularly difficult for those with enough experience (and I am a good example of someone who has been able to gain that experience relatively quickly), but integrating this into setup-kolab in an acceptable fashion ultimately proved to be an unrealisable goal.

Other people will presumably continue their work packaging various Kolab libraries for Debian, and some of these lower-level packages may even arrive in the stable release of Debian fairly soon, perhaps even delivering a recent version of those libraries. I do not, however, see any progress on getting other packages into Debian proper. I continue to have the opinion that this unfortunate situation will limit wider adoption of the Kolab technologies and does nobody but the proprietary competition any good.

Since I do not believe in writing software that will never be used – having had that experience in a professional setting where I at least had the consolation of getting paid for such disappointing outcomes (and for the waste of my time) – my current focus is on developing a low-impact form of standards-based calendaring for existing mail systems, without imposing extensive infrastructure requirements when adopting such a solution, and I hope to have something useful to show in the fairly near future. This time last year, I was much more upbeat about the prospect of getting Kolab into Debian and into more people’s hands. Now, I only wish that I had changed course earlier and started on my current endeavour considerably sooner.

But as people like to say: better late than never. I look forward to sharing my non-Kolab groupware developments in the coming months.

In my last post about Kolab I hinted about unbundling the LDAP server and using a remote server for LDAP service. Since then, I’ve been looking at making the IMAP server an “unbundled” component – IMAP service still being required, of course – and even at supporting Dovecot in setup-kolab as well. After all, choice is an important motivation for adopting Free Software, and we should at least try and make that choice convenient to exercise where possible. Supporting a choice of IMAP servers gives everyone a bit more flexibility and should make Kolab a bit easier to adopt. The Dovecot work is still very much in progress, however.

As you may recall, I wanted to deploy an XMPP server – ejabberd – on one host and the rest of the Kolab stack on another host, and yet retain the possibility of configuring these different components using setup-kolab. To support this specific situation, and to eventually move beyond it to support other architectural configurations, I have chosen to introduce a new metapackage called kolab-minimal: the glue components of Kolab plus the Web components minus the “infrastructure” components, those being the components providing the LDAP, IMAP and mail transport services. Here’s where this takes us:

A dependency diagram for Kolab featuring a kolab-minimal package and the different setup-kolab invocations

If you want to install the complete stack, the kolab metapackage will bring everything you need into the installation and hopefully provide you with a self-contained solution. But now, the kolab metapackage is formulated in terms of the “infrastructure” components plus kolab-minimal. And if you want to install something that gets its services from other computers, kolab-minimal should be all you need.

Obviously, this work deviates from the “official” Kolab packages, but since it appears that those packages aren’t being refreshed with upstream fixes as often as might be desirable, I’ve decided to put some source packages online. If you should be tempted to build them and try them out, please remember to do so in a test environment and that you do so at your own peril! And if you want to track my packaging changes, I’ve put a collection of repositories online, too.

Note that the XMPP dependencies (Converse and ejabberd) are integrated into the above, but more testing is really needed to make sure that they become robust additions. Indeed, as part of the Converse plugin package there’s a patch that fixes an annoying bug that makes deploying the plugin almost like an “all or nothing” affair. So I’ve put them under a new metapackage – kolab-extra – for now.

And the way the packaging invokes the configuration program – setup-kolab – probably needs some review, particularly the way setup-kolab wants to edit or replace configuration files of other packages. Some insights into the proper Debian way of doing this would be very useful indeed. I can imagine Kolab using configuration files in slightly different locations and changing references to where the configuration lives in only one key place (or a few key places), leaving the originals intact, but I haven’t been able to look at this in depth.

And with that, I’ll give my modestly equipped desktop computer a rest from running two User Mode Linux instances and pdebuild in 1GB RAM (of the semiconductor variety, plus some swap).

I had another chance to look at Kolab and the dependency graph recently. Having been inspired by the prospect of chat integration within Roundcube, I set out to install a suitable XMPP server, and it seemed that ejabberd was the most likely choice on Debian systems such as my own. Here is what the configuration would ideally look like:

Kolab, Roundcube and ejabberd running in a User Mode Linux environment

But I then discovered that ejabberd did not seem to work with the User Mode Linux environment in which I test my packages. This then gave me an excuse to contemplate the relationship between the different components. LDAP is central to the way Kolab manages credentials and is employed by ejabberd to authenticate users, but the LDAP service does not need to be located on the same server. Indeed, it is likely that in a larger organisation the services would reside on a number of different computers.

Repositioning LDAP

Since I was interested in writing a component to configure ejabberd for integration with the other Kolab components, but that this service would have to be installed outside my own User Mode Linux environment (within which Roundcube happens to reside), I therefore needed to find a way of teaching setup-kolab (Kolab’s setup script) about remote LDAP services as an alternative to any such service running on the same machine. And from this perspective I realised that the dependency on LDAP is a “soft” one: it is entirely possible to want to install Kolab without also installing an LDAP server suite, but the need for LDAP service remains. It thus falls on other computers to provide LDAP services to the computer running the chat service (and to the other Kolab services, too).

A bit of adjustment to the setup_ldap module in pykolab and it became possible to choose a local directory or a remote one accessible via LDAP. At this point, running ejabberd outside User Mode Linux (UML) and connecting to the LDAP service running inside UML looked feasible, and I developed a setup-kolab component to propagate Kolab settings to ejabberd’s configuration file, but my desktop environment’s chat program didn’t seem interested in joining the testing effort. That meant that I really had to get the Converse plugin working within Roundcube, thus enabling chat within the webmail environment.

Enlisting Converse

Naturally, this meant figuring out a reliable way of configuring Converse, and thus another setup-kolab component was created for this purpose. So far, so straightforward: get Converse to talk to an XMPP service and the job is done. But now in my arrangement, the XMPP service – ejabberd – is situated in a remote location from Converse and in a separate location from Roundcube, and is thus not accessible without some additional measures. Converse runs JavaScript in the browser, but that code needs to “bind” to the XMPP service in order to be able to use it, and a general security measure enforced in browsers is that scripts aren’t allowed to talk to any location on the Internet just because they want to: instead, they may be restricted to only being capable of sending information to the server that delivered them to the browser in the first place. Here is a diagram illustrating the problem:

Kolab, Roundcube and ejabberd, but with Converse wanting to communicate with two different hosts

Since Converse wants to talk to the XMPP service, but given that the XMPP service is not located in the same place as the Web server that sent it to the browser, a proxy must be deployed to listen to Converse within the Web environment and then relay the communications to ejabberd. This involves configuring Apache to receive requests and pretend to be the “connection manager”, and then Apache forwards such requests to the real connection manager provided by ejabberd. Thus, the following diagram illustrates the solution to this distribution of services problem:

Kolab, Roundcube and ejabberd, with the latter being reached via a proxy from Roundcube

Thus, the task of setting up chat in Roundcube, integrated with Kolab, involves the following:

• The configuration of ejabberd to authenticate users using Kolab account details stored in the LDAP directory
• The configuration of Roundcube to enable the Converse plugin and…
• The deployment of a proxy site in Apache to forward Converse’s chat requests to ejabberd

The State of Play

There seems to be plenty of integration work still to be done. Although Converse can obtain contact details supplied by ejabberd from the LDAP service and thus provide immediate access to other users in the same organisation, the level of integration with the rest of the interface is still fairly loose: you cannot find a chat button in the address book for each contact, for example. Even so, the level of convenience probably already matches various other groupware solutions.

I can’t wait to see what kind of communication or collaboration technology will be next, even if there will be a degree of work to make it a bit easier to set up with Kolab. And that reminds me to get the configuration nuts and bolts packed off and sent upstream so that everybody else can try it out.

A few weeks ago, I published a suggested dependency arrangement for Kolab in Debian. For completeness, here is a reasonably nice diagram showing the dependency arrangement used by the existing “vendor” packages:

The dependencies used by the current "vendor" packages for Kolab

As noted before, the experiment I have been performing reorganises the packages so that they can be configured one at a time, and I even managed to get the debconf system to ask the user any necessary installation-related questions.

(Once again, the slightly fancier than normal output is thanks to the notugly.xsl stylesheet – described here – and updated further by myself to fix a few things that seemed to go wrong. At some point, I’ll try and send my changes to vidarh and see if he likes them.)

My recent diversion via pykolab may appear to have rather little to do with the matter of improving the Debian packaging situation for Kolab, but after initial tidying exercises with the pykolab code, I started to consider how the setup-kolab program should behave in the context of Debian packaging operations: what it should do when a package is installed, if anything, and whether it can rely on anything having been configured in advance. Until now, setup-kolab has been regarded as a tool that is run once the Kolab software is mostly or completely present on a system, but this is a somewhat different approach to the way many service-providing Debian packages are set up.

Take MySQL as an example. Upon installing the appropriate Debian package providing the MySQL server, the user is prompted to set up the administrative credentials for the server. After this brief interaction, MySQL should be available for general use without further configuration work, although it may be the case that some tuning might be beneficial. It seems to me that Kolab could be delivered with the same convenience in Debian.

The Different Ways of Configuring Kolab

Until now, many of the individual Kolab meta-packages – those grouping together individual components as functional units providing mail transport (MTA), storage (IMAP) or directory (LDAP) functionality – have only indirectly relied on the presence of the kolab-conf package providing the setup-kolab program. Indeed, if the top-level kolab meta-package is installed, kolab-conf will be pulled in as a dependency and setup-kolab will be available, and as noted above, since setup-kolab has been the program that configures a complete system, this makes some sense. However, this causes the work of configuring Kolab to be put off until the very end and thus to pile up so that a user may have to undertake a lengthy question-and-answer process to get the software working, and this can lead to mistakes and a non-working installation.

But setup-kolab is familiar with the notion of configuring individual components. Although the Kolab documentation has emphasised a “one shot” configuration of everything at once, setup-kolab can be asked to configure different individual things, such as the LDAP integration or the integration with Roundcube (the webmail software supported by Kolab). Thus, it becomes interesting to consider whether individual packages can be configured one by one using setup-kolab until they have all been configured, and whether this produces the same desired result of everything having been set up (and hopefully without the same intensity of questioning).

To make setup-kolab available to packages during their configuration, the kolab-conf package providing the program must be a dependency of each of the packages concerned. Here is a nice little diagram illustrating the result of a few adjustments to the dependency graph:

A revised dependency graph for Kolab (produced using Graphviz and tidied up using a modified version of vidarh's notugly.xsl stylesheet)

(Here, the pykolab packages are coloured in green and have been allowed to float freely to make the layout cleaner: pykolab contributes in a number of ways and the introduction of kolab-conf as a dependency of a few packages means that it is very much “in demand” by all parts of the graph.)

Now, with kolab-conf available when, for example, kolab-imap is being installed, the post-installation script of kolab-imap will be able to invoke setup-kolab and to make sure that a usable configuration is available by the time the Debian packaging system is done with the installation activity. But one thing might be bothering the attentive reader: why is kolab-ldap not dependent on kolab-conf? In fact, the configuration process has to start somewhere, and since information related to the LDAP directory is rather central to further configuration, this initial configuration takes place when the kolab-conf package is itself installed. And since this initial configuration needs access to the LDAP system, the dependency relationship is “inverted” for this single case, and kolab-conf populates the basic configuration so that subsequent package configuration can take advantage of this initial work.

Communicating with the User

Command line usage of setup-kolab involves some fairly straightforward console-style prompting and input, but during Debian packaging activities, it is widely regarded as a bad thing to have packaging scripts just ask for things from standard input and just write out messages to standard output or standard error: people may be using graphical package management tools that offer alternative facilities for user interaction, and it is possible that console-style operations go completely unnoticed by both the user and such tools. Thus, there is a need to make setup-kolab aware of the facility known as debconf (which is not to be confused with the DebConf series of conferences that may well appear in any searches made to try and discover documentation about the debconf system).

It would appear that debconf is used primarily within shell scripts, as the packaging scripts most commonly use the shell command language, but a goal of this work is surely to avoid replicating the activities of setup-kolab in other programs: it is not particularly desirable to have to rewrite the component-specific parts of pykolab in shell command language just to be able to use debconf to talk to users. Fortunately, the debconf package provides a Python wrapper for the debconf system, and although the documentation and examples are not particularly substantial or enlightening, a bit of experimentation and some consultation of the debconf specification has led to a workable level of interaction between setup-kolab and debconf so that the latter can prompt the user and supply the user’s input to the former without too much complaint.

What Next?

As previously stated, the aim now is to try and get the pykolab changes upstream – regardless of the Debian-related modifications, pykolab has been enhanced in other ways – and to try and reach consensus about whether this way of structuring the Debian dependencies is sensible or not. A substantial number of iterations involving pykolab adjustments, packaging improvements, pbuilder sessions and eventual deployment of the new packages in a virtual machine seem to indicate that the approach is not completely impractical, but it is possible that there are better ways of doing some of this work.

But certainly, my confidence in the Debian packaging situation for Kolab is significantly higher than it was before. I can now see a time when the matter of installing and configuring Kolab (at least for common deployment situations) will be seen as entirely uninteresting and a largely solved problem. Then, people will be able to concentrate on more interesting matters instead.

After my previous efforts at building Kolab packages using more traditional Debian methods, my attention then turned to the nature of the software itself, particularly since the Debian Kolab packages (as currently available from the Kolab project’s own repository) do not configure the software so that it is immediately usable: that job is assigned to a program called setup-kolab, which is provided in the kolab-conf Debian package but really lives in the pykolab software distribution. Thus, my focus shifted to what pykolab is and does, and to any ways I might be able to improve or adjust that software.

A quick examination of the Debian packages produced by the pykolab source package gives an indication of the different things pykolab provides: a command framework for configuration (kolab-conf), a command framework for administration (kolab-cli), and an underlying library that supports these and other packages (pykolab). It surprised me slightly that there was so much Python involved with Kolab – my perception had been that the project was mostly KDE-related C++ libraries combined with PHP Web applications (sitting on top of some fairly well-known and established programs and libraries) – and since my primary programming language has been Python for quite some time, I took the opportunity to go through the code and see if there were any trivial but worthwhile adjustments that could be made to make more substantial further improvements more convenient.

Upon emerging from some verbose tidying efforts, hopefully being merged into the upstream code in the near future, I turned my attention to the way setup-kolab behaves. Although the tool mostly does the job already, there are things about it that are perhaps not quite as convenient as they could be. For example, until now, setup-kolab has made the user specify database credentials to access the MySQL database system (unless an explicit component other than MySQL or Roundcube has been indicated as an argument to the program), and in many cases this is not really necessary: the required database may already exist, and on a Debian system there are established ways to query the database system about existing databases without asking the user for the login details. Another desirable feature is that of being able to run setup-kolab and not have it try and perform configuration tasks that have already been done: although prior LDAP directory instances are detected and setup-kolab refuses to go any further unless a special option is given as a command argument, it would be nice if setup-kolab just mentioned that no work needs to be done, at least for most of the components.

Ultimately, setup-kolab might itself be called from packaging scripts invoked during the installation of various different packages, perhaps bringing in the debconf framework to present a standardised interface on Debian systems, but hopefully not demanding the user’s attention at all. Until then, I hope that it can offer a convenient way of setting Kolab up and/or checking that the different components are ready to support Kolab in use.

More Fun with Packaging

Changing the actual pykolab source code did also involve building the different binary packages from the pykolab source package, but this meant that doing other packaging-related things was largely ignored until someone asked a question on the #kolab IRC channel about the free/busy component and how it should be configured to function. I had not looked very much at this component at all, and I discovered that I did not even have it installed. The kolab-freebusy package sits on top of the other Kolab-related packages, so as I had been installing the kolab package and watching various dependencies being pulled in, kolab-freebusy managed to avoid being installed. (This is understandable from a packaging perspective as the free/busy functionality is an extension to the basic services provided by Kolab.)

Upon inspecting the configuration of the kolab-freebusy package, I noticed that it had not been set up right, either in its default configuration or by setup-kolab. Some further investigation revealed that setup-kolab was trying to configure a file that the kolab-freebusy package had renamed, thus making configuration somewhat ineffective. Fortunately, although not without some duplicated work before I realised, this particular bug had already been fixed in a later version of the package. Still, one or two other things also needed changing and thus a trip via pbuilder was in order.

One of the things that needed changing, or at least refining, concerns the relationship between kolab-freebusy and kolab-utils. The latter package provides the “free/busy daemon”, and one may indeed wonder what this is when the free/busy functionality must surely be provided by the package with the more obvious name. In fact, this is where a picture can be very descriptive:

The basic free/busy architecture in Kolab (made with Graphviz and made nicer using vidarh's notugly.xsl template)

So, since Kolab stores calendar information in IMAP but wants to be able to publish it in a relatively efficient way, the daemon runs periodically and exports free/busy resources for calendar users. Such information is then provided upon request when clients access the free/busy Web service provided by the kolab-freebusy package.

Although one can install only the Web service (in the kolab-freebusy package) and ask for free/busy information, the exercise is rather pointless: there will be no information to retrieve, at least in the default configuration of the system and with the default functionality being offered; only after obtaining and running the daemon (in the kolab-utils package) will the Web service have any use. I am inclined to think that if someone wants to install and use kolab-freebusy, they should also have kolab-utils pulled in for their convenience (or perhaps a package that only provides the daemon).

In Conclusion

All in all, I think Kolab offers plenty of things to explore, review, and – on occasion – fix or improve, but it has to be mentioned that once the packages provided the right directories for logging and free/busy resource storage, and once the Web service started to use the correct configuration, everything seemed to work as intended. I hope that with some more polishing, we will see something that more people will be able to try out and perhaps feel comfortable using in their own organisation.

Some things I would like to see, as I noted on the mailing list, include…

• Getting some of my changes upstream and into the packaging. On the latter front, the Kolab packaging effort in Debian itself is apparently being revived: this will be highly beneficial in a number of ways, not least that of using Debian-native packaging tools instead of the less-than-optimal Open Build Service.
• A revival of the Kolab Wiki. It is nice to be able to write things up somewhere, evolve a consensus about how things might be done, and collaboratively describe how things are already done: it is rather hard to do this on mailing lists or using reference documentation tools like Sphinx, and blogging about it only achieves so much.
• Actually improving the functionality and not just the nuts and bolts of getting Kolab installed and set up.

Obviously, I aim to continue doing these things and not just ask for things and expect to see others doing them, and I am quite sure that there are others in the community who are similarly motivated. If your organisation could benefit from Kolab or does so already, or if it benefits from any Free Software groupware solution, please consider supporting the developers of those solutions in their work.

Without actively developed Free Software groupware, there is a risk that, over time, other Free Software deployed in the average organisation may be under threat as proprietary software vendors and uninformed decision-makers use excuse after excuse to flush out diversity and replace it with their own favourite monoculture of proprietary products and solutions. By supporting Free Software groupware and establishing basic infrastructure founded on Free Software and open standards in your own organisation, you establish an environment of fair competition, nurture a healthy and often necessary technological diversity, and you give your organisation the strategic freedom that would otherwise be denied to you.

To have the choice of Free Software in any particular application or field, we need to sustain those giving us that choice and not just say that we would use it “if only it were better” and then pay a proprietary vendor for something nicer or shinier instead. Please keep such things in mind when evaluating solutions for your organisation’s immediate and future needs.

My recent excursion into Debian packaging with Kolab has involved a tour of lots of different tools and services, but it started out with a brief attempt to build the existing packages with pbuilder: a tool that has become fairly familiar to me in the process of experimenting with Debian packages and even contributing one to “Debian proper”. After some initial frustrations that prevented me from building packages using my normal workflow, I decided to familiarise myself with the infrastructure that the Kolab project itself uses to make packages, if only to reassure myself that the packages really could be built and didn’t require any special magic to do so. I won’t go into this because Timotheus has already done so in sufficient depth.

After some playing around with osc and the Development project in the Kolab OBS (Open Build Service), building packages, installing them in Debian root filesystems and User Mode Linux instances (administered by some scripts I’ve developed over the years), I persuaded myself to have another go at feeding the packages to pbuilder via the pdebuild tool, determined to overcome any build issues and to demonstrate that it could be done. One fairly good reason for doing this is that even though pbuilder-based builds can be sluggish as pbuilder decides to unpack an existing filesystem, install build dependencies, and do other housekeeping before actually building the package, it seems to be more efficient and quicker than osc, which I found took as long as 9 minutes before it was ready to build a relatively straightforward Python-based package. Another reason for going with pbuilder is that it is what the Debian project itself will be using, more or less, if – hopefully when – the packages get accepted back into Debian, whereas the OBS infrastructure seems to be based on other technologies for the management of the build environment.

Packages of the Future

One challenge posed by Kolab is that some of its packages depend on some of the other Kolab-provided packages when being built – those other packages are so-called “build dependencies” – but tools like pdebuild rely on such build dependencies already being available as installable packages from the Debian archive. For things like make and gcc, this isn’t a problem at all: they were packaged a very long time ago (although I suppose you could get caught out with very recent versions of such packages). But when a yet-to-be-submitted package is required as a build dependency for another package, the issue of satisfying that dependency arises, and this was something I hadn’t encountered before.

A perusal of the Debian Wiki provided a solution: after building the packages that the other ones rely upon, put them in a directory, expose them in a repository, and then make pbuilder consult that repository when deciding how it can satisfy those build dependencies. This involves three things:

• A directory with newly built packages, obviously, together with the necessary package metadata.
• A hook script that will refresh the metadata and perform the necessary update action that lets the pbuilder environment know about the packages.
• A special configuration for pbuilder.

The Package Directory

At first, this will contain nothing at all, but as you add packages it will contain both .deb package files and the package metadata. We will call this directory deps.

The Hook Script

As the Debian Wiki page describes, a hook called D05deps can be placed in a directory called, say, hooks and populated with the following code:

#!/bin/sh
(cd /path-to-kolab-packaging/deps; apt-ftparchive packages . > Packages)
apt-get update

I did find that the permissions on the hook file were crucial, fixing them as follows:

chmod a+x hooks/D05deps

Otherwise, no mention of the script will be made in the copious output from pbuilder and it will simply be ignored. I also experienced some initial problems with the package metadata, but running the first line of actual commands from the script and manually producing the metadata was enough to get pbuilder on the right track:

cd deps; apt-ftparchive packages . > Packages

After that, it didn’t have any difficulties seeing the new packages as I added them to the deps directory.

The Configuration File

All this has to do is to point to the deps and hooks directories. You can more or less copy the contents of /etc/pbuilderrc and add the following to it (customising it for your own choices, of course):

OTHERMIRROR="deb [trusted=yes] file:///path-to-kolab-packaging/deps ./"
BINDMOUNTS="/path-to-kolab-packaging/deps"
HOOKDIR="/path-to-kolab-packaging/hooks"
EXTRAPACKAGES="apt-utils"

Yes, there really isn’t anything different about this than the example on the Debian Wiki page. I put this configuration file alongside the deps and hooks directories and called it pbuilderrc.

Actually Building Packages

With the above extra stuff in place, the process of building packages is slightly different: you have to tell pbuilder to use this alternative configuration, and then you just hope that all the different aspects of it are consistent and that pbuilder is able to take notice of it. The command that will eventually be run inside a directory containing “Debianized” sources is the following:

pdebuild -- --distribution wheezy --override-config --configfile ../pbuilderrc

Obviously, the pbuilderrc file resides in the parent directory after you change into a package’s sources directory.

Build Order

Above, I mentioned that some packages depend on others in order to be built. Finding out which packages are affected involves consulting their build dependencies which are conveniently listed in their .dsc files. Doing the following permits a general overview to be obtained and the basis of a suitable build order to be worked out:

grep ^Build-Depends *.dsc

Obviously, it makes sense to start with packages that do not depend on others that are also being built for this exercise. Devising or discovering an automated approach for this is left as an exercise for the reader, but Kolab is relatively uncomplicated and I used the following build order:

python-icalendar pykolab libkolabxml libcalendaring libkolab kolab kolab-freebusy kolab-schema kolab-syncroton kolab-utils kolab-webadmin mozilla-ldap-sdk chwala irony pyasn1-modules php-http-request2 roundcubemail roundcubemail-plugin-contextmenu roundcubemail-plugin-dblog roundcubemail-plugin-threadingasdefault roundcubemail-plugins-kolab smarty3

The General Workflow

To obtain, unpack and build the packages I used the following workflow:

1. Visit the package downloads page (found via the Development project’s repository overview) and obtain a list of package URLs.
2. Get each package using dget from the devscripts package. This will probably give a warning about unsigned or unverifiable packages and not unpack the sources. (I suppose that importing the repository key using apt-key fixes this. One should obviously consider the risks and recommendations around downloading code from the Internet.)
3. Unpack each package using dpkg-source. For example:

   dpkg-source -x python-icalendar_3.4-1.dsc

4. Change into the source directory.
5. Run the pdebuild command given above.
6. Copy or move the resulting package files from /var/cache/pbuilder/result into the deps directory.
7. Optional but tidy: move the other artefacts of building from the parent directory into some other place for future reference.

There are probably much more efficient and cleaner ways of building lots of packages, but this allowed me to inspect them and to consider a few changes.

Making Changes

I took the opportunity to make some changes to the python-icalendar package because I saw that it wanted me to install python-setuptools before pdebuild would even launch pbuilder. I have little confidence in setuptools generally and would prefer not to have it on my system, and it is an unfortunate but recurring phenomenon that one finds the setup.py script commonly used for the preparation or installation of Python software packages using setuptools when its functionality only requires the older and less disruptive distutils library. Regardless of whether such changes are desired in the eventual Kolab packaging, I took the opportunity to investigate how changes should be made to the package.

Modern Debian packages prefer such “upstream patching” – where the code being changed originates with the actual developers of the software, as opposed to people packaging it for different distributions – to be done using a tool called quilt. I have some experience using quilt for my previous packaging work, but it’s easy to forget how to use it. Fortunately, the Debian Wiki came to the rescue once again. Here’s what I did in the sources directory for the package:

quilt new setup.py.patch # tell quilt about my patch
quilt add setup.py # tell quilt that I will patch setup.py
# Now, I edited setup.py to replace setuptools with distutils.
quilt refresh # update the patch within quilt
quilt pop -a # go back to the way everything was (but remember the patch for later)

When running pdebuild, the tool will notice such patches and apply them. Upon finishing they should be provided in a file containing all the necessary patches that allow the software to be built as a package. In this case, a file called python-icalendar_3.4-1.debian.tar.gz was produced.

One or two things are probably necessary to make this work:

• A suitable quilt configuration as described on the Debian Wiki page referenced above.
• A suitable debian/source/format file containing the 3.0 (quilt) value.

What Next?

Some of the previously encountered pitfalls had very little to do with the actual packaging of Kolab in terms of getting the software installed, but were more to do with the way it behaved once configured (and perhaps how the configuration gets done). I intend to look a bit more closely at the configuration process and to see if some of the awkward situations that may arise can’t be diagnosed and remedied by some helpful enhancements to the tools. On the way, I expect to find areas of improvement in the ways some things are done – that’s just the way things are with software – but with regard to the packaging itself most of the hard work has already been done and it seems to hold up rather well. So thanks are obviously due to Paul and Jeroen (and others) for allowing me to join in at this fairly late stage in the game.

Well, after my recent blog post highlighting some surprising problems with my Kolab installation – not at all a complaint about the packages, really, but more of a contribution towards improving the packaging situation, as I see it at least – some more interest in the situation around Kolab packaging for Debian has been shown:

• HowTo improve Debian packages using OBS describes the documented packaging-related process in more detail for a specific improvement.
• A Call for Participation invites those interested in Debian packaging to get involved.

Packaging for Debian can be a challenge. My own experience involved a pure-Python tool and still required lots of iterations to satisfy the Debian gatekeepers; this is understandable given that they try to virtually guarantee a coherent experience and provide a large selection of software whose copyright and licensing status must be clear, acceptable and without nasty surprises. I respect the effort that has gone into Kolab packaging for Debian already: without that effort, I probably wouldn’t even have tried the software.

The plan now must surely involve input from the Debian groupware initiative, especially as the Kolab architecture presumably resembles some of the other packaged solutions, and those who have contributed to the existing packaging work, as well as some discussion on the Kolab development mailing list, and some effort with the Open Build Service tools (with the “build commander” tool fortunately being available as a Debian package).

It is unfortunate that as Torsten points out, “Currently, there’s only one volunteer working on the Debian packages in his limited spare time, but hundreds of people who want to use reliable Debian packages.” Meanwhile, Timotheus points out, “Since there seems to be no corporate funding available for the Debian packages, we all need to pull together as a community and get it done!” It seems to me that those organisations that stand to benefit from more adoption of Free Software groupware, especially those using Debian as their foundation, might do well to assist this work instead of waiting for people to get it done in their free time.

Hugo Roy has been trying to install Kolab and not getting on particularly well with it. His experiences persuaded me to take another look at my Kolab installation done back in June, and to my surprise it didn’t seem to work any more. I eventually discovered some things that will probably need fixing in the packaging, and these are mentioned below. I suppose I’ll try and pursue these with the developers and packagers.

The LDAP server (provided by the 389-ds suite of packages, but actually started when the ns-slapd program is run, and known as the dirsrv service – yes, all very confusing stuff) doesn’t want to run until the permissions are fixed on the /var/run/dirsrv and /var/lock/dirsrv directories so that the ns-slapd program can create pid and lock files.

The kolab-saslauthd service won’t be running if the LDAP server isn’t running. (You can check this using service --status-all and seeing what is running and what isn’t.) Some Kolab programs seem to get upset when they can’t connect to the LDAP or IMAP servers, and if the LDAP server is brought up, there’s a frequent, recurring error from a Python program complaining about IMAP server connections failing…

Traceback (most recent call last):
 File "/usr/lib/python2.7/dist-packages/kolabd/process.py", line 44, in synchronize
 auth.synchronize()
 File "/usr/lib/python2.7/dist-packages/pykolab/auth/__init__.py", line 243, in synchronize
 self._auth.synchronize()
 File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 860, in synchronize
 callback=self._synchronize_callback,
 File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 2151, in _search
 secondary_domains
 File "", line 10, in
 File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 1895, in _persistent_search
 secondary_domains=secondary_domains
 File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 1735, in _synchronize_callback
 eval("self._change_none_%s(entry, change_dict)" % (entry['type']))
 File "", line 1, in
 File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 1389, in _change_none_user
 self.imap.connect(domain=self.domain)
 File "/usr/lib/python2.7/dist-packages/pykolab/imap/__init__.py", line 144, in connect
 self._imap[hostname].login(admin_login, admin_password)
 File "/usr/lib/python2.7/dist-packages/pykolab/imap/cyrus.py", line 133, in login
 cyruslib.CYRUS.login(self, *args, **kw)
 File "/usr/lib/python2.7/dist-packages/cyruslib.py", line 416, in login
 self.__doexception("LOGIN", error)
 File "/usr/lib/python2.7/dist-packages/cyruslib.py", line 359, in __doexception
 self.__doraise( function.upper(), msg )
 File "/usr/lib/python2.7/dist-packages/cyruslib.py", line 368, in __doraise
 raise CYRUSError( idError[0], mode, msg )
 CYRUSError: (10, 'LOGIN', 'generic failure')

Restarting the kolab-saslauthd service fixes this; maybe restarting the cyrus-imapd service also helps. Restarting the kolab-server service should apparently synchronise the constituent services, but I’m not sure it helps if you get the above Python error. You may also see an LDAP-related error which just appears to be the same program or a related one getting even more upset about the LDAP server.

Also, if you don’t update for a while, the clamav-freshclam service uses a lot of CPU and bandwidth performing updates. Such stuff needs turning off if you value your computer’s interactivity, in my experience.