Paul Boddie's Free Software-related blog

A long time ago, when lots of people complained all the time about how it was seemingly “impossible” to write Python programs that use more than one CPU or CPU core, and given that Unix has always (at least in accessible, recorded history) allowed processes to “fork” and thus create new ones that run the same code, I decided to write a library that makes such mechanisms more accessible within Python programs. I wasn’t the only one thinking about this: another rather similar library called processing emerged, and the Python core development community adopted that one and dropped it into the Python standard library, renaming it multiprocessing.

Now, there are a few differences between multiprocessing and my own library, pprocess, largely because my objectives may have been slightly different. First of all, I had been inspired by renewed interest in the communicating sequential processes paradigm of parallel processing, which came to prominence around the Transputer system and Occam programming language, becoming visible once more with languages like Erlang, and so I aimed to support channels between processes as one of my priorities. Secondly, and following from the previous objective, I wasn’t trying to make multithreading or the kind of “shared everything at your own risk” model easier or even possible. Thirdly, I didn’t care about proprietary operating systems whose support for process-forking was at that time deficient, and presumably still is.

(The way that the deficiencies of Microsoft Windows, either inherently or in the way it is commonly deployed, dictates development priorities in a Free Software project is yet another maddening thing about Python core development that has increased my distance from the whole endeavour over the years, but that’s a rant for another time.)

User Stories

Despite pprocess making it into Debian all by itself – or rather, with the efforts of people who must have liked it enough to package it – I’ve only occasionally had correspondence about it, much of it regarding the package falling out of Debian and being supported in a specialised Debian variant instead. For all the projects I have produced, I just assume now that the very few people using them are either able to fix any problems themselves or are happy enough with the behaviour of the code that there just isn’t any reason for them to switch to something else.

Recently, however, I heard from Kai Staats who is using Python for some genetic programming work, and he had found the pprocess and multiprocessing libraries and was trying to decide which one would work best for him. Perhaps as a matter of chance, multiprocessing would produce pickle errors that he found somewhat frustrating to understand, whereas pprocess would not: this may have been a consequence of pprocess not really trying very hard to provide some of the features that multiprocessing does. Certainly, multiprocessing attempts to provide some fairly nice features, but maybe they fail under certain overly-demanding circumstances. Kai also noted some issues with random number generators that have recently come to prominence elsewhere, interestingly enough.

Some correspondence between us then ensued, and we both gained a better understanding of how pprocess could be applied to his problem, along with some insights into how the documentation for pprocess might be improved. Eventually, success was achieved, and this article serves as a kind of brief response to his conclusion of our discussions. He notes that the multiprocess model inhibits the sharing of global variables, almost as a kind of protection against “bad things”, and that the processes must explicitly communicate their results with each other. I must admit, being so close to my own work that the peculiarities of its use were easily assumed and overlooked, that pprocess really does turn a few things about Python programs on its head.

Update: Kai suggested that I add the following, perhaps in place of the remarks about the sharing of global variables…

Kai notes that pprocess allows passing more than one variable through the multi-core portal at once, identical to a standard Python method. This enables direct translation of a single CPU for-loop into a multiple-CPU model, with nearly the same number of lines of code.

Two Tales of Concurrency

If you choose to use threads for concurrency in Python, you’ll get the “shared everything at your own risk” model, and you can have your threads modifying global variables as they see fit. Since CPython employs a “global interpreter lock”, the modifications will succeed when considered in isolation – you shouldn’t see things getting corrupted at the lowest level (pointers or references, say) like you might if doing such things unsafely in a systems programming language – but without further measures, such modifications may end up causing inconsistencies in the global data as threads change things in an uncoordinated way.

Meanwhile, pprocess also lets you change global variables at will. However, other processes just don’t see those changes and continue happily on their way with the old values of those globals.

Mutability is a key concept in Python. For anyone learning Python, it is introduced fairly early on in the form of the distinction between lists and tuples, perhaps as a curiosity at first. But then, when dictionaries are introduced to the newcomer, the notion of mutability becomes somewhat more important because dictionary keys must be immutable (or, at least, the computed hash value of keys must remain the same if sanity is to prevail): try and use a list or even a dictionary as a key and Python will complain. But for many Python programmers, it is the convenience of passing objects into functions and seeing them mutated after the function has completed, whether the object is something as transparent as a list or whether it is an instance of some exotic class, that serves as a reminder of the utility of mutability.

But again, pprocess diverges from this behaviour: pass an otherwise mutable object into a created process as an argument to a parallel function and, while the object will indeed get mutated inside the created “child” process, the “parent” process will see no change to that object upon seeing the function apparently complete. The solution is to explicitly return – or send, depending on the mechanisms chosen – changes to the “parent” if those changes are to be recorded outside the “child”.

Awkward Analogies

Kai mentioned to me the notion of a portal or gateway through which data must be transferred. For a deeper thought experiment, I would extend this analogy by suggesting that the portal is situated within a mirror, and that the mirror portrays an alternative reality that happens to be the same as our own. Now, as far as the person looking into the mirror is concerned, everything on the “other side” in the mirror image merely reflects the state of reality on their own side, and initially, when the portal through the mirror is created, this is indeed the case.

But as things start to occur independently on the “other side”, with things changing, moving around, and so on, the original observer remains oblivious to those changes and keeps seeing the state on their own side in the mirror image, believing that nothing has actually changed over there. Meanwhile, an observer on the other side of the portal sees their changes in their own mirror image. They believe that their view reflects reality not only for themselves but for the initial observer as well. It is only when data is exchanged via the portal (or, in the case of pprocess, returned from a parallel function or sent via a channel) that the surprise of previously-unseen data arriving at one of our observers occurs.

Expectations and Opportunities

It may seem very wrong to contradict Python’s semantics in this way, and for all I know the multiprocessing library may try and do clever things to support the normal semantics behind the scenes (although it would be quite tricky to achieve), but anyone familiar with the “fork” system call in other languages would recognise and probably accept the semantic “discontinuity”. One thing I’ve learned is that it isn’t always possible to assume that people who are motivated to use such software will happen to share the same motivations as I and other library developers may have in writing that software.

However, it has also occurred to me that the behavioural differences caused in programs by pprocess could offer some opportunities. For example, programs can implement transactional behaviour by creating new processes which may or may not return data depending on whether the transactions performed by the new processes succeed or fail. Of course, it is possible to implement such transactional behaviour in Python already with some discipline, but the underlying copy-on-write semantics that allow “fork” and pprocess to function make it much easier and arguably more reliable.

With CPython’s scalability constantly being questioned, despite attempts to provide improved concurrency features (in Python 3, at least), and with a certain amount of enthusiasm in some circles for functional programming and the ability to eliminate side effects in parts of Python programs, maybe such broken expectations point the way to evolved forms of Python that possibly work better for certain kinds of applications or systems.

So, it turns out that user feedback doesn’t have to be about bug reports and support questions and nothing else: it can really get you thinking about larger issues, questioning long-held assumptions, and even be a motivation to look at certain topics once again. But for now, I hope that Kai’s programs keep producing correct data as they scale across the numerous cores of the cluster he is using. We can learn new things about my software another time!

It’s an odd-enough question. After all, the Firefox browser is surely called “Mozilla Firefox” if you use its full name, and the organisation behind it is called “Mozilla Corporation“. Mozilla has been responsible for various products and projects over the years, but if you actually go to the Mozilla site and look around now, it’s all Firefox, Firefox and, digging deeper, Firefox. Well, there’s also a mention of something called Webmaker, “apps”, and some developer-related links, presented within a gallery of pictures of the cool people working for Mozilla.

Now, I use Iceweasel, which is Debian’s version of Firefox, and it’s a good browser. But what concerns me is what has happened to certain other products produced by Mozilla that people also happen to be using. In the buzz that Mozilla are trying to create around their Firefox-centred strategy, with Firefox-the-browser, Firefox-the-mobile-OS, and whatever else the Firefox name will soon be glued onto, what treatment do things like Thunderbird get? Go to the Mozilla site and try and find the page for it: it’s easier to just use a search engine instead.

And once you’ve navigated to the product page for Thunderbird, the challenge of finding useful, concrete information continues. It may very well be the case that most people just want a download button and to be in and out of the site as fast as possible, on their way to getting the software installed and running. (If so, one really hopes that they did use a search engine and didn’t go in via Mozilla’s front page.) But what if you want to find out more about the code, the community, the development processes? Dig too deep in the support section – a reasonable path to take – and you’ll be back in Firefox country where there are no Thunderbirds to be found.

Now, I don’t use Thunderbird for my daily e-mail needs: given that I’ve used KDE for a decade and a half, I’ve been happy with Kontact for my modest e-mail retrieving, reading, writing and sending activities. But Thunderbird is used by quite a few other people, and I did indeed use it for a few years in a former workplace. I didn’t always like how it worked, especially compared to Kontact, but then again Kontact needed quite a bit of tuning to work to my tastes, especially when I moved over to KDE 4 (or Plasma, if you insist) and had to turn off all sorts of things that were bolted on but didn’t really work. Generally, however, both products do their job well enough.

When Mozilla announced that Thunderbird would take a back seat to other activities (which looks more like being pushed off the desk now, but anyway), people complained a lot about it. One would have thought that leveraging the common Mozilla codebase to keep delivering a cross-platform, Free Software e-mail client would help uphold the kind of freedom and interoperability in messaging that the organisation seeks to uphold on the Web generally. But I suppose the influencers think that webmail is enough, not least because the browser remains central in such a strategy. Unfortunately, webmail doesn’t exactly empower end-users with things like encryption and control over their own data, at least in the traditional sense. (Projects like Mailpile which deliver a Web-based interface locally via the browser are different, of course.)

So, given any need to remedy deficiencies with Thunderbird, where should one go? Fortunately, I did some research earlier in the year – maybe Mozilla’s site was easier to navigate back then – and found the Thunderbird page on the Mozilla Wiki. Looking again, I was rather surprised to see recent activity at such a level that it apparently necessitates weekly status meetings. Such meetings aren’t really my kind of thing, but the fact that they are happening does give me a bit more confidence about a product that one might think is completely abandoned if one were only paying attention to the official Mozilla channels. My own interests are more focused on the Lightning calendar plugin, and its official page is more informative than that of Thunderbird, but there’s also a collection of wiki pages related to it as well.

Once upon a time, there was a company called Mosaic Communications Corporation that became Netscape Communications Corporation, both of these names effectively trading on the familiarity of the Mosaic and Netscape product names. Given Mozilla’s apparent focus on “Firefox”, it wouldn’t surprise me if they soon went the other way and called themselves Firefox Corporation. But I would rather they sought to deliver a coherent message through a broad range of freedom-upholding and genuinely useful products than narrowing everything to a single brand and one-and-a-bit products that – in case those things don’t work for you – leave you wondering what your options are, especially in this day and age of proprietary, cloud-based services and platforms that are increasingly hostile to interoperability.

Without even a peripheral Mozilla Messaging organisation to block the tidal flow towards “convenient” but exploitative cloud services, one has to question Mozilla’s commitment in this regard. But those responsible could at least fix up the incoherent Web site design that would leave many wondering whether Thunderbird and other actively-supported Mozilla products were just products of their own vivid and idealistic imagination.

I was interested to read the recent announcement about source code availability for the first Fairphone device. I’ve written before about the threat to that device’s continued viability and Fairphone’s vague position on delivering a device that properly supports Free Software. It is nice to see that the initiative takes such matters seriously and does not seem to feel that letting its partners serve up what they have lying around is sufficient. However, a few questions arise, starting with the following quote from the announcement:

We can happily say that we have recently obtained a software license from all our major partners and license holders that allows us to modify the Fairphone 1 software and release new versions to our users. Getting that license also required us to obtain rights to use and distribute Mentor Graphics’s RTOS used on the phone. (We want to thank Mentor Graphics in making it possible for us to acquire the distribution license for their RTOS, as well as other partners for helping us achieve this.)

I noted before that various portions of the software are already subject to copyleft licensing, but if we ignore those (and trust that the sources were already being made available), it is interesting to consider the following questions:

• What is “the Fairphone 1 software” exactly?
• Fairphone may modify the software but what about its customers?
• What role does the Mentor Graphics RTOS have? Can it be replaced by customers with something else?
• Do the rights to use and distribute the RTOS extend to customers?
• Do those rights extend to the source code of the RTOS, and do those rights uphold the four freedoms?

On further inspection, some contradictions emerge, perhaps most efficiently encapsulated by the following quote:

Now that Fairphone has control over the Fairphone 1 source code, what’s next? First of all, we can say that we have no plans to stop supporting the Fairphone hardware. We will continue to apply security fixes as long as it is feasible for the years to come. We will also keep exploring ways to increase the longevity of the Fairphone 1. Possibilities include upgrading to a more recent Android version, although we would like to manage expectations here as this is still very much a longshot dependent on cooperation from license holders and our own resources.

If Fairphone has control over the source code, why is upgrading to a more recent Android version dependent on cooperation with licence holders? If Fairphone “has control” then the licence holders should already have provided the necessary permissions for Fairphone to actually take control, just as one would experience with the four freedoms. One wonders which permissions have been withheld and whether these are being illegitimately withheld for software distributed under copyleft licences.

With a new device in the pipeline, I respect the persistence of Fairphone in improving the situation, but perhaps the following quote summarises the state of the industry and the struggle for sustainable licensing and licence compliance:

It is rather unusual for a small company like Fairphone to get such a license (usually ODMs get these and handle most of the work for their clients) and it is uncommon that a company attempts and manages to obtain such a license towards the end of the economic life cycle of the product.

Sadly, original design manufacturers (ODMs) have a poor reputation: often being known for throwing binaries over the wall whilst being unable or unwilling to supply the corresponding sources, with downstream manufacturers and retailers claiming that they have no leverage to rectify such licence violations. Although the injustices and hardships of those working to supply the raw materials for products like the Fairphone, along with those of the people working to construct the devices themselves, make other injustices seem slight – thinking especially of those experienced by software developers whose copyright is infringed by dubious industry practices – dealing with unethical and untidy practices wherever they may be found should be part of the initiative’s objectives.

From what I’ve seen and heard, Fairphone 2 should have a better story for chipset support and Free Software, but again, an inspection of the message raises some awkward questions. For example:

In the coming months we are going to launch several programs that address different aspects of creating fairer software. For now, one of the best tools for us to reach these goals is to embrace open source principles. With this in mind and without further ado, we’re excited to announce that we are going to release the complete build environment for Fairphone OS on Fairphone 2, which contains the full open source code, all the tools and the binary blobs that will allow users to build their own Fairphone OS.

To be fair, binary blobs are often difficult to avoid: desktop computers often use them for various devices, and even devices like the Neo900 that emphasise a completely Free Software stack will end up using them for certain functions (mitigating this by employing other technical measures). Making the build environment available is a good thing: frequently, this aspect is overlooked and anyone requesting the source code can be left guessing about build configuration details in an exercise that is effectively a matter of doing the vendor’s licence compliance work for them. But here, we are left wondering where the open source code ends, where binary blobs will be padding out the distribution, and what those blobs are actually for.

We need to keep asking difficult questions about such matters even if what Fairphone is doing is worthy in its own right. Not only does it safeguard the interests of the customers involved, but it also helps Fairphone to focus on doing the right thing. It does feel unkind to criticise what seems like a noble initiative for not doing more when they obviously try rather hard to do the right thing in so many respects. But by doing the right thing in terms of the software as well, Fairphone can uphold its own reputation and credibility: something that all businesses need to remember, as certain very large companies have very recently discovered.

Recently, I was alerted to a blog post by Bradley Kuhn of the Software Freedom Conservancy where he describes the way in which proprietary e-mail infrastructure not only withholds the freedoms end-users should expect from their software, but where the operators of such infrastructure also stifle the free exchange of information by refusing to deliver mail, apparently denying delivery according to seemingly arbitrary criteria (if we are to be charitable about why, for example, Microsoft might block the mails sent by an organisation that safeguards the rights of Free Software users and developers).

The article acknowledges that preventing spam and antisocial activities is difficult, but it rightfully raises the possibility that if things continue in the same way they have been going, one day the only way to use e-mail will be through subscribing to an opaque service that, for all we as customers would know, censors our messages, gathers and exploits personal information about us, and prevents people from contacting each other based on the whims of the operator and whatever agenda they might have.

Solved and Boring

Sadly, things like e-mail don’t seem to get the glory amongst software and solutions developers that other methods of online communication have experienced in recent years: apparently, it’s all been about real-time chat, instant messaging, and social networking. I had a conversation a few years ago on the topic of social networking with an agreeable fellow who was developing a solution, but I rather felt that when I mentioned e-mail as the original social networking service and suggested that it could be tamed for a decent “social” experience, he regarded me as being somewhat insane to even consider e-mail for anything serious any more.

But anyway, e-mail is one of those things that has been regarded as a “solved problem“, meaning that the bulk of the work needed to support it is regarded as having been done a long time ago, and the work needed to keep it usable and up-to-date with evolving standards is probably now regarded as a chore (if anyone really thinks about that work at all, because some clearly do not). Instead of being an exciting thing bringing us new capabilities, it is at best seen as a demanding legacy that takes time away from other, more rewarding challenges. Still, we tell ourselves, there are solid Free Software solutions available to provide e-mail infrastructure, and so the need is addressed, a box is ticked, and we have nothing to worry about.

Getting it Right

Now, mail infrastructure can be an intimidating thing. As people will undoubtedly tell you, you don’t want to be putting a mail server straight onto the Internet unless you know what you are doing. And so begins the exercise of discovering what you should be doing, which either entails reading up about the matter or just paying someone else to do the thinking on your behalf, which in the latter case either takes the form of getting some outside expertise to get you set up or instead just paying someone else to provide you with a “mail solution”. In this day and age, that mail solution is quite likely to be a service – not some software that you have to install somewhere – and with the convenience of not having to manage anything, you rely completely on your service provider to do the right thing.

So to keep the software under your own control, as Bradley points out, Free Software needs to be well-documented and easy to adopt in a foolproof way. One might regard “foolproof” as an unkind term, but nobody should need to be an expert in everything, and everybody should be able to start out on the path to understanding without being flamed for being ignorant of the technical details. Indeed, I would go further and say that Free Software should lend itself to secure-by-default deployment which should hold up when integrating different components, as opposed to finger-pointing and admonishments when people try and figure out the best practices themselves. It is not enough to point people at “how to” documents and tell them to not only master a particular domain but also to master the nuances of specific technologies to which they have only just been introduced.

Thankfully, some people understand. The FreedomBox initiative is ostensibly about letting people host their own network services at home, which one might think is mostly a matter of making a computer small and efficient enough to sit around switched on all the time, combined with finding ways to let people operate such services behind potentially restrictive ISP barriers. But the work required to achieve this in a defensible and sustainable way involves providing software that is easily and correctly configured and which works properly from the moment the system is deployed. It is no big surprise that such work is being done in close association with Debian.

Signs of the Times

With regard to software that end-users see, the situation could be a lot worse. KDE’s Kontact and KMail have kept up a reasonably good experience over the years, despite signs of neglect and some fairly incoherent aspects of the user interface (at least as I see it on Debian); I guess Evolution is still out there and still gets some development attention, as is presumably the case with numerous other, less well-known mail programs; Thunderbird is still around despite signs that Mozilla presumably thought that people should have been using webmail solutions instead.

Indeed, the position of Mozilla’s leadership on Thunderbird says a lot about the times we live in. Web and mobile things – particularly mobile things – are the new cool, and if people aren’t excited by e-mail and don’t see the value in it, then developers don’t see the value in developing solutions for it, either. But sometimes those trying to focus on current fashions fail to see the value in the unfashionable, and a backlash ensued: after all, what would people end up using at work and in “the enterprise” if Thunderbird were no longer properly supported? At the same time, those relying on Thunderbird’s viability, particularly those supplying it for use in organisations, were perhaps not quite as forthcoming with support for the project as they could have been.

Ultimately, Thunderbird has kept going, which is just as well given that the Free Software cross-platform alternatives are not always obvious or necessarily as well-maintained as they could be. Again, a lesson was given (if not necessarily learned) about how neglect of one kind of Free Software can endanger the viability of Free Software in an entire area of activity.

Webmail is perhaps a slightly happier story in some ways. Roundcube remains a viable and popular Web-hosted mail client, and the project is pursuing an initiative called Roundcube Next that seeks to refactor the code in order to better support new interfaces and changing user expectations. Mailpile, although not a traditional webmail client – being more a personal mail client that happens to be delivered through a Web interface – continues to be developed at a steady pace by some very committed developers. And long-established solutions like SquirrelMail and Horde IMP still keep doing good service in many places.

Attitude Adjustment

In his article, Bradley points out that as people forsake Free Software solutions for their e-mail needs, whether deciding to use an opaque and proprietary webmail service for personal mail, or whether deciding that their organisation’s mail can entirely be delegated to a service provider, it becomes more difficult to make the case for Free Software. It may be convenient to “just get a Gmail account” and if your budget (of time and/or money) doesn’t stretch to using a provider that may be friendlier towards things like freedom and interoperability, you cannot really be blamed for taking the easiest path. But otherwise, people should be wary of what they are feeding with their reliance on such services.

And those advocating such services for others should be aware that the damage they are causing extends far beyond the impact on competing solutions. Even when everybody was told that there is a systematic programme of spying on individuals, that industrial and competitive espionage is being performed to benefit the industries of certain nations, and that sensitive data could end up on a foreign server being mined by random governmental and corporate agencies, decision-makers will gladly exhibit symptoms of denial dressed up in a theatrical display of level-headedness: making a point of showing that they are not at all bothered by such stories, which they doubt are true anyway, and will with proud ignorance more or less say so. At risk are the details of other people’s lives.

Indicating that privacy, control and sustainability are crucial to any organisation will, in the face of such fact-denial, indeed invite notions that anyone bringing such topics up is one of the “random crazy people” for doing so. And by establishing such a climate of denial and marginalisation, the forces that would control our communications are able to control the debate about such matters, belittling concerns and appealing to the myth of the benign corporation that wants nothing in return for its “free” or “great value” products and who would never do anything to hurt its customers.

We cannot at a stroke solve such matters of ignorance, but we can make it easier for people to do the right thing, and to make it more obvious and blatant when people have chosen to do the wrong thing in the face of more conveniently and appropriately doing the right thing. We can evolve our own attitudes more easily, making Free Software easier to deploy and use, and in the case of e-mail not perpetuate the myth that nothing more needs to be done.

We will always have work to do to keep our communications free and unimpeded, but the investment we need to make is insignificant in comparison to the value to society of the result.

One of my other interests, if you can even regard it as truly separate to my interests in Free Software and open hardware, involves the microcomputer systems of the 1980s that first introduced me to computing and probably launched me in the direction of my current career. There are many aspects of such systems that invite re-evaluation of their capabilities and limitations, leading to the consideration of improvements that could have been made at the time, as well as more radical enhancements that unashamedly employ technology that has only become available or affordable in recent years. Such “what if?” thought experiments and their hypothetical consequences are useful if we are to learn from the strategic mistakes once made by systems vendors, to have an informed perspective on current initiatives, and to properly appreciate computing history.

At the same time, people still enjoy actually using such systems today, writing new software and providing hardware that makes such continuing usage practical and sustainable. These computers and their peripherals are certainly “getting on”, and acquiring or rediscovering such old systems does not necessarily mean that you can plug them in and they still work as if they were new. Indeed, the lifetime of magnetic media and the devices that can read it, together with issues of physical decay in some components, mean that alternative mechanisms for loading and storing software have become attractive for some users, having been developed to complement or replace the cassette tape and floppy disk methods that those of us old enough to remember would have used “back in the day”.

My microcomputer of choice in the 1980s was the Acorn Electron – a cut-down, less expensive version of the BBC Microcomputer hardware platform – which supported only cassette storage in its unexpanded form. However, some expansion units added the disk interfaces present on the BBC Micro, while others added the ability to use ROM-based software. On the BBC Micro, one would plug ROM chips directly into sockets, and some expansion units for the Electron supported this method, too. The official Plus 1 expansion chose instead to support the more friendly expansion cartridge approach familiar to users of other computing and console systems, with ROM cartridges being the delivery method for games, applications and utilities in this form, providing nothing much more than a ROM chip and some logic inside a convenient-to-use cartridge.

The Motivation

A while ago, my brother, David, became interested in delivering software on cartridge for the Electron, and a certain amount of discussion led him to investigate various flash memory integrated circuits (ICs, chips), notably the AMD Am29F010 series. As technological progress continues, such devices provide a lot of storage in comparison to the ROM chips originally used with the Electron: the latter having only 16 kilobytes of capacity, whereas the Am29F010 variant chosen here has a capacity of 128 kilobytes. Meanwhile, others chose to look at EEPROM chips, notably the AT28C256 from Atmel.

Despite the manufacturing differences, both device types behave in a very similar way: a good idea for the manufacturers who could then sell products that would be compatible straight away with existing products and the mechanisms they use. In short, some kind of de-facto standard seems to apply to programming these devices, and so it should be possible to get something working with one and then switch to the other, especially if one kind becomes too difficult to obtain.

Now, some people realised that they could plug such devices into their microcomputers and program them “in place” using a clever hack where writes to the addresses that correspond to the memory provided by the EEPROM (or, indeed, flash memory device) in the computer’s normal memory map can be trivially translated into addresses that have significance to the EEPROM itself. But not routinely using such microcomputers myself, and wanting more flexibility in the programming of such devices, not to mention also avoiding the issue of getting software onto such computers so that it can be written to such non-volatile memory, it seemed like a natural course of action to try to do the programming with the help of some more modern conveniences.

And so I considered the idea of getting a microcontroller solution like the Arduino to do the programming work. Since an Arduino can be accessed over USB, a ROM image could be conveniently transferred from a modern computer and, with a suitable circuit wired up, programmed into the memory chip. ROM images can thus be obtained in the usual modern way – say, from the Internet – and then written straight to the memory chip via the Arduino, rather than having to be written first to some other medium and transferred through a more convoluted sequence of steps.

Breadboarding

Being somewhat familiar with Arduino experimentation, the first exercise was to make the circuit that can be used to program the memory device. Here, the first challenge presented itself: the chip employs 17 address lines, 8 data lines, and 3 control lines. Meanwhile, the Arduino Duemilanove only provides 14 digital pins and 6 analogue pins, with 2 of the digital pins (0 and 1) being unusable if the Arduino is communicating with a host, and another (13) being connected to the LED and being seemingly untrustworthy. Even with the analogue pins in service as digital output pins, only 17 pins would be available for interfacing.

So, a way of multiplexing the Arduino pins was required, where at one point in time the Arduino would be issuing signals for one purpose, these signals would then be “stored” somewhere, and then at another point in time the Arduino would be issuing signals for another purpose. Ultimately, these signals would be combined and presented to the memory device in a hopefully coherent fashion. We cannot really do this kind of multiplexing with the control signals because they typically need to be coordinated to act in a timing-sensitive fashion, so we would be concentrating on the other signals instead.

So which signals would be stored and issued later? Well, with as many address lines needing signals as there are available pins on the Arduino, it would make sense to “break up” this block of signals into two. So, when issuing an address to the memory device, we would ideally be issuing 17 bits of information all at the same time, but instead we take approximately half of the them (8 bits) and issue the necessary signals for storage somewhere. Then, we would issue the other half or so (8 bits) for storage. At this point, we need only a maximum of 8 signal lines to communicate information through this mechanism. (Don’t worry, I haven’t forgotten the other address bit! More on that in a moment!)

How would we store these signals? Fortunately, I had considered such matters before and had ordered some 74-series logic chips for general interfacing, including 74HC273 flip-flop ICs. These can be given 8 bits of information and will then, upon command, hold that information while other signals may be present on its input pins. If we take two of these chips and attach their input pins to those 8 Arduino pins we wish to use for communication, we can “program” each 74HC273 in turn – one with 8 bits of an address, the other with another 8 bits – and then the output pins will be presenting 16 bits of the address to the memory chip. At this point, those 8 Arduino pins could even be doing something else because the 74HC273 chips will be holding the signal values from an earlier point in time and won’t be affected by signals presented to their input pins.

Of all the non-control signals, with 16 signals out of the way, that leaves only 8 signals for the memory chip’s data lines and that other address signal to deal with. But since the Arduino pins used to send address signals are free once the addresses are sent, we can re-use those 8 pins for the data signals. So, with our signal storage mechanism, we get away with only using 8 Arduino pins to send 24 pieces of information! We can live with allocating that remaining address signal to a spare Arduino pin.

That now leaves us with the task of managing the 3 control signals for the memory chip – to make it “listen” to the things we are sending to it – but at the same time, we also need to consider the control lines for those flip-flop ICs. Since it turns out that we need 1 control signal for each of the 74HC273 chips, we therefore need to allocate 5 additional interfacing pins on the Arduino for sending control signals to the different chips.

In the end, we don’t even need all the available pins on the Arduino, but the three going spare wouldn’t be enough to save us from having to use the flip-flop ICs.

With this many pins in use, and the need to connect them together, there are going to be a lot of wires in use:

The breadboard circuit with the Arduino and ICs

The result is somewhat overwhelming! Presented in a more transparent fashion, and with some jumper wires replaced with breadboard wires, it is slightly easier to follow:

An overview of the breadboard circuit

The orange wires between the two chips on the right-hand breadboard indicate how the 8 Arduino pins are connected beyond the two flip-flop chips and directly to the flash memory chip, which would sit on the left-hand breadboard between the headers inserted into that breadboard (which weren’t used in the previous arrangement).

Making a Circuit Board

It should be pretty clear that while breadboarding can help a lot with prototyping, things can get messy very quickly with even moderately complicated circuits. And while I was prototyping this, I was running out of jumper wires that I needed for other things! Although this circuit is useful, I don’t want to have to commit my collection of components to keeping it available “just in case”, but at the same time I don’t want to have to wire it up when I do need it. The solution to this dilemma was obvious: I should make a “proper” printed circuit board (PCB) and free up all my jumper wires!

It is easy to be quickly overwhelmed when thinking about making circuit boards. Various people recommend various different tools for designing them, ranging from proprietary software that might be free-of-charge in certain forms but which imposes arbitrary limitations on designs (as well as curtailing your software freedoms) through to Free Software that people struggle to recommend because they have experienced stability or functionality deficiencies with it. And beyond the activity of designing boards, the act of getting them made is confused by the range of services in various different places with differing levels of service and quality, not to mention those people who advocate making boards at home using chemicals that are, shall we say, not always kind to the skin.

Fortunately, I had heard of an initiative called Fritzing some time ago, initially in connection with various interesting products being sold in an online store, but whose store then appeared to be offering a service – Fritzing Fab – to fabricate individual circuit boards. What isn’t clear, or wasn’t really clear to me straight away, was that Fritzing is also some Free Software that can be used to design circuit boards. Conveniently, it is also available as a Debian package.

The Fritzing software aims to make certain tasks easy that would perhaps otherwise require a degree of familiarity with the practice of making circuit boards. For instance, having decided that I wanted to interface my circuit to an Arduino as a shield which sits on top and connects directly to the connectors on the Arduino board, I can choose an Arduino shield PCB template in the Fritzing software and be sure that if I then choose to get the board made, the dimensions and placement of the various connections will all be correct. So for my purposes and with my level of experience, Fritzing seems like a reasonable choice for a first board design.

Replicating the Circuit

Fritzing probably gets a certain degree of disdain from experienced practitioners of electronic design because it seems to emphasise the breadboard paradigm, rather than insisting that a proper circuit diagram (or schematic) acts as the starting point. Here is what my circuit looks like in Fritzing:

The breadboard view of my circuit in Fritzing

You will undoubtedly observe that it isn’t much tidier than my real-life breadboard layout! Having dragged a component like the Arduino Uno (mostly compatible with the Duemilanove) onto the canvas along with various breadboards, and then having dragged various other components onto those breadboards, all that remains is that we wire them up like we managed to do in reality. Here, Fritzing helps out by highlighting connections between things, so that breadboard columns appear green as wires are connected to them, indicating that an electrical connection is made and applies to all points in that column on that half of the breadboard (the upper or lower half as seen in the above image). It even highlights things that are connected together according to the properties of the device, so that any attempt to modify to a connection that leads to one of the ground pins on the Arduino also highlights the other ground pins as the modification is being done.

I can certainly understand criticism of this visual paradigm. Before wiring up the real-life circuit, I typically write down which things will be connected to each other in a simple table like this:

If I were not concerned with prototyping with breadboards, I would aim to use such information directly and not try and figure out which size breadboard I might need (or how many!) and how to arrange the wires so that signals get where they need to be. When one runs out of points in a breadboard column and has to introduce “staging” breadboards (as shown above by the breadboard hosting only incoming and outgoing wires), it distracts from the essential simplicity of a circuit.

Anyway, once the circuit is defined, and here it really does help that upon clicking on a terminal/pin, the connected terminals or pins are highlighted, we can move on to the schematic view and try and produce something that makes a degree of sense. Here is what that should look like in Fritzing:

The schematic for the circuit in Fritzing

Now, the observant amongst you will notice that this doesn’t look very tidy at all. First of all, there are wires going directly between terminals without any respect for tidiness whatsoever. The more observant will notice that some of the wires end in the middle of nowhere, although on closer inspection they appear to be aimed at a pin of an IC but are shifted to the right on the diagram. I don’t know what causes this phenomenon, but it would seem that as far as the software is concerned, they are connected to the component. (I will come back to how components are defined and the pitfalls involved later on.)

Anyway, one might be tempted to skip over this view and try and start designing a PCB layout directly, but I found that it helped to try and tidy this up a bit. First of all, the effects of the breadboard paradigm tend to manifest themselves with connections that do not really reflect the logical relationships between components, so that an Arduino pin that feeds an input pin on both flip-flop ICs as well as a data pin on the flash memory IC may have its connectors represented by a wire first going from the Arduino to one of the flip-flop ICs, then to the other flip-flop IC, and finally to the flash memory IC in some kind of sequential wiring. Although electrically this is not incorrect, with a thought to the later track routing on a PCB, it may not be the best representation to help us think about such subsequent problems.

So, for my own sanity, I rearranged the connections to “fan out” from the Arduino as much as possible. This was at times a frustrating exercise, as those of you with experience with drawing applications might recognise: trying to persuade the software that you really did select a particular thing and not something else, and so on. Again, selecting the end of a connection causes some highlighting to occur, and the desired result is that selecting a terminal highlights the appropriate terminals on the various components and not the unrelated ones.

Sometimes that highlighting behaviour provides surprising and counter-intuitive results. Checking the breadboard layout tends to be useful because Fritzing occasionally thinks that a new connection between certain pins has been established, and it helpfully creates a “rats nest” connection on the breadboard layout without apparently saying anything. Such “rats nest” connections are logical connections that have not been “made real” by the use of a wire, and they feature heavily in the PCB view.

PCB Layout

For those of us with no experience of PCB layout who just admire the PCBs in everybody else’s products, the task of laying out the tracks so that they make electrical sense is a daunting one. Fritzing will provide a canvas containing a board and the chosen components, but it is up to you to combine them in a sensible way. Here, the circuit board actually corresponds to the Arduino in the breadboard and schematic views.

But slightly confusing as the depiction of the Arduino is in the breadboard view, the pertinent aspects of it are merely the connectors on that device, not the functionality of the device itself which we obviously aren’t intending to replicate. So, instead of the details of an actual Arduino or its functional equivalent, we instead merely see the connection points required by the Arduino. And by choosing a board template for an Arduino shield, those connection points should appear in the appropriate places, as well as the board itself having the appropriate size and shape to be an Arduino shield.

Here’s how the completed board looks:

The upper surface of the PCB design in Fritzing

Of course, I have spared you a lot of work by just showing the image above. In practice, the components whose outlines and connectors feature above need to be positioned in sensible places. Then, tracks need to be defined connecting the different connection points, with dotted “rats nest” lines directly joining logically-connected points needing to be replaced with physical wiring in the form of those tracks. And of course, tracks do not enjoy the same luxury as the wires in the other views, of being able to cross over each other indiscriminately: they must be explicitly routed to the other side of the board, either using the existing connectors or by employing vias.

The lower surface of the PCB design in Fritzing

Hopefully, you will get to the point where there are no more dotted lines and where, upon selecting a connection point, all the appropriate points light up, just as we saw when probing the details of the other layouts. To reassure myself that I probably had connected everything up correctly, I went through my table and inspected the pin-outs of the components and did a kind of virtual electrical test, just to make sure that I wasn’t completely fooling myself.

With all this done, there isn’t much more to do before building up enough courage to actually get a board made, but one important step that remains is to run the “design checks” via the menu to see if there is anything that would prevent the board from working correctly or from otherwise being made. It can be the case that tracks do cross – the maze of yellow and orange can be distracting – or that they are too close and might cause signals to go astray. Fortunately, the hours of planning paid off here and only minor adjustments needed to be done.

It should be noted that the exercise of routing the tracks is certainly not to be underestimated when there are as many connections as there are above. Although an auto-routing function is provided, it failed to suggest tracks for most of the required connections and produced some bizarre routing as well. But clinging onto the memory of a working circuit in real three-dimensional space, along with the hope that two sides of a circuit board are enough and that there is enough space on the board, can keep the dream of a working design alive!

The Components

I skipped over the matter of components earlier on, and I don’t really want to dwell on the matter too much now, either. But one challenge that surprised me given the selection of fancy components that can be dragged onto the canvas was the lack of a simple template for a 32-pin DIP (dual in-line package) socket for the Am29F010 chip. There were socket definitions of different sizes, but it wasn’t possible to adjust the number of pins.

Now, there is a parts editor in Fritzing, but I tend to run away from graphical interfaces where I suspect that the matter could be resolved in more efficient ways, and it seems like other people feel the same way. Alongside the logical definition of the component’s connectors, one also has to consider the physical characteristics such as where the connectors are and what special markings will be reproduced on the PCB’s silk-screen for the component.

After copying an existing component, ransacking the Fritzing settings files, editing various files including those telling Fritzing about my new parts, I achieved my modest goals. But I would regard this as perhaps the weakest part of the software. I didn’t resort to doing things the behind-the-scenes way immediately, but the copy-and-edit paradigm was incredibly frustrating and doesn’t seem to be readily documented in a way I could usefully follow. There is a Sparkfun tutorial which describes things at length, but one cannot help feeling that a lot of this should be easier, especially for very simple component changes like the one I needed.

The Result

With some confidence and only modest expectations of success, I elected to place an order with the Fritzing Fab service and to see what the result would end up like. This was straightforward for the most part: upload the file created by Fritzing, fill out some details (albeit not via a secure connection), and then proceed to payment. Unfortunately, the easy payment method involves PayPal, and unfortunately PayPal wants random people like myself to create an account with them before they will consider letting me make a credit card payment, which is something that didn’t happen before. Fortunately, the Fritzing people are most accommodating and do support wire transfers as an alternative payment method, and they were very responsive to my queries, so I managed to get an order submitted even more quickly than I thought might happen (considering that fabrication happens only once a week).

Just over a week after placing my order, the board was shipped from Germany, arriving a couple of days later here in Norway. Here is what it looked like:

The finished PCB from Fritzing

Now, all I had to do was to populate the board and to test the circuit again with the Arduino. First, I tested the connections using the Arduino’s 5V and GND pins with an LED in series with a resistor in an “old school” approach to the problem, and everything seemed to be as I had defined it in the Fritzing software.

Given that I don’t really like soldering things, the act of populating the board went about as well as expected, even though I could still clean up the residue from the solder a bit (which would lead me onto a story about buying the recommended chemicals that I won’t bother you with). Here is the result of that activity:

The populated board together with the Arduino

And, all that remained was the task of getting my software running and testing the circuit in its new form. Originally, I was only using 16 address pins, holding the seventeenth low, and had to change the software to handle these extended addresses. In addition, the issuing of commands to the flash memory device probably needed a bit of refinement as well. Consequently, this testing went on for a bit longer than I would have wished, but eventually I managed to successfully replicate the programming of a ROM image that had been done some time ago with the breadboard circuit.

The outcome did rely on a certain degree of good fortune: the template for the Arduino Uno is not quite compatible with the Duemilanove, but this was rectified by clipping two superfluous pins from one of the headers I soldered onto the board; two of the connections belonging to the socket holding the flash memory chip touch the outside of the plastic “power jack” socket, but not enough to cause a real problem. But I would like to think that a lot of preparation dealt with problems that otherwise might have occurred.

Apart from liberating my breadboards and wires, this exercise has provided useful experience with PCB design. And of course, you can find the sources for all of this in my repository, as well as a project page for the board on the Fritzing projects site. I hope that this account of my experiences will encourage others to consider trying it out, too. It isn’t as scary as it would first appear, after all, although I won’t deny that it was quite a bit of work!

A while ago, I noted that I had moved on from attempting to contribute to Kolab and had started to explore other ways of providing groupware through existing infrastructure options. Once upon a time, I had hoped that I could contribute to Kolab on the basis of things I mostly knew about, whilst being able to rely on the solution itself (and those who made it) to take care of the things I didn’t really know very much about.

But that just didn’t work out: I ultimately had to confront issues of reliably configuring Postfix, Cyrus, 389 Directory Server, and a variety of other things. Of course, I would have preferred it if they had just worked so that I could have got on with doing more interesting things.

Now, I understand that in order to pitch a total solution for someone’s groupware needs, one has to integrate various things, and to simplify that integration and to lower the accompanying support burden, it can help to make various choices on behalf of potential users. After all, if they don’t have a strong opinion about what kind of mail storage solution they should be using, or how their user database should be managed, it can save them from having to think about such things.

One certainly doesn’t want to tell potential users or customers that they first have to go off, read some “how to” documents, get some things working, and then come back and try and figure out how to integrate everything. If they were comfortable with all that, maybe they would have done it all already.

And one can also argue about whether Kolab augments and re-uses or merely replaces existing infrastructure. If the recommendation is that upon adopting Kolab, one replaces an existing Postfix installation with one that Kolab provides in one form or another, then maybe it is possible to re-use the infrastructure that is already there.

It is harder to make that case if one is already using something else like Exim, however, because Kolab doesn’t support Exim. Then, there is the matter of how those components are used in a comprehensive groupware solution. Despite people’s existing experiences with those components, it can quickly become a case of replacing the known with the unknown: configuring them to identify users of the system in a different way, or to store messages in a different fashion, and so on.

Incremental Investments

I don’t have such prior infrastructure investments, of course. And setting up an environment to experiment with such things didn’t involve replacing anything. But it is still worthwhile considering what kind of incremental changes are required to provide groupware capabilities to an existing e-mail infrastructure. After all, many of the concerns involved are orthogonal:

• Where the mail gets stored has little to do with how recipients are identified
• How recipients are identified has little to do with how the mail is sent and received
• How recipients actually view their messages and calendars has little to do with any of the above

Where components must talk to one another, we have the benefit of standards and standardised protocols and interfaces. And we also have a choice amongst these things as well.

So, what if someone has a mail server delivering messages to local users with traditional Unix mailboxes? Does it make sense for them to schedule events and appointments via e-mail? Must they migrate to another mail storage solution? Do they have to start using LDAP to identify each other?

Ideally, such potential users should be able to retain most of their configuration investments, adding the minimum necessary to gain the new functionality, which in this case would merely be the ability to communicate and coordinate event information. Never mind the idea that potential users would be “better off” adopting LDAP to do so, or whichever other peripheral technology seems attractive for some kinds of users, because it is “good practice” or “good experience for the enterprise world” and that they “might as well do it now”.

The difference between an easily-approachable solution and one where people give you a long list of chores to do first (involving things that are supposedly good for you) is more or less equivalent to the difference between you trying out a groupware solution or just not bothering with groupware features at all. So, it really does make sense as someone providing a solution to make things as easy as possible for people to get started, instead of effectively turning them away at the door.

Some Distractions

But first, let us address some of the distractions that usually enter the picture. A while ago, I had the displeasure of being confronted with the notion of “integrated e-mail and calendar” solutions, and it turned out that such terminology is coined as a form of euphemism for adopting proprietary, vendor-controlled products that offer some kind of lifestyle validation for people with relatively limited imagination or experience: another aspirational possession to acquire, and with it the gradual corruption of organisations with products that shun interoperability and ultimately limit flexibility and choice.

When standards-based calendaring has always involved e-mail, such talk of “integrated calendars” can most charitably be regarded as a clumsy way of asking for something else, namely an e-mail client that also shows calendars, and in the above case, the one that various people already happened to be using that they wanted to impose on everyone else as well. But the historical reality of the integration of calendars and e-mail has always involved e-mails inviting people to events, and those people receiving and responding to those invitation e-mails. That is all there is to it!

But in recent times, the way in which people’s calendars are managed and the way in which notifications about events are produced has come to involve “a server”. Now, some people believe that using a calendar must involve “a server” and that organising events must also involve doing so “on the server”, and that if one is going to start inviting people to things then they must also be present “on the same server”, but it is clear from the standards documents that “a server” was never a prerequisite for anything: they define mechanisms for scheduling based on peer-to-peer interactions through some unspecified medium, with e-mail as a specific medium defined in a standards document of its own.

Having “a server” is, of course, a convenient way for the big proprietary software vendors to sell their “big server” software, particularly if it encourages the customer to corrupt various other organisations with which they do business, but let us ignore that particular area of misbehaviour and consider the technical and organisational justifications for “the server”. And here, “server” does not mean a mail server, with all the asynchronous exchanges of information that the mail system brings with it: it is the Web server, at least in the standards-adhering realm, that is usually the kind of server being proposed.

The Superfluous Server

Given that you can send and receive messages containing events and other calendar-related things, and given that you can organise attendance of events over e-mail, what would the benefit of another kind of server be, exactly? Well, given that you might store your messages on a server supporting POP or IMAP (in the standards-employing realm, that is), one argument is that you might need somewhere to store your calendar in a similar way.

But aside from the need for messages to be queued somewhere while they await delivery, there is no requirement for messages to stay on the server. Indeed, POP server usage typically involves downloading messages rather than leaving them on the server. Similarly, one could store and access calendar information locally rather than having to go and ask a server about event details all the time. Various programs have supported such things for a long time.

Another argument for a server involves it having the job of maintaining a central repository of calendar and event details, where the “global knowledge” it has about everybody’s schedules can be used for maximum effect. So, if someone is planning a meeting and wants to know when the potential participants are available, they can instantly look such availability information up and decide on a time that is likely to be acceptable to everyone.

Now, in principle, this latter idea of being able to get everybody’s availability information quickly is rather compelling. But although a central repository of calendars could provide such information, it does not necessarily mean that a centralised calendar server is a prerequisite for convenient access to availability data. Indeed, the exchange of such data – referred to as “free/busy” data in the various standards – was defined for e-mail (and in general) at the end of the last century, although e-mail clients that can handle calendar data typically neglect to support such data exchanges, perhaps because it can be argued that e-mail might not obtain availability information quickly enough for someone impatiently scheduling a meeting.

But then again, the routine sharing of such information over e-mail, plus the caching of such data once received, would eliminate most legitimate concerns about being able to get at it quickly enough. And at least this mechanism would facilitate the sharing of such data between organisations, whereas people relying on different servers for such services might not be able to ask each other’s servers about such things (unless they have first implemented exotic and demanding mechanisms to do so). Even if a quick-to-answer service provided by, say, a Web server is desirable, there is nothing to stop e-mail programs from publishing availability details directly to the server over the Web and downloading them over the Web. Indeed, this has been done in certain calendar-capable e-mail clients for some time, too, and we will return to this later.

And so, this brings us to perhaps the real reason why some people regard a server as attractive: to have all the data residing in one place where it can potentially be inspected by people in an organisation who feel that they need to know what everyone is doing. Of course, there might be other benefits: backing up the data would involve accessing one location in the system architecture instead of potentially many different places, and thus it might avoid the need for a more thorough backup regime (that organisations might actually have, anyway). But the temptation to look and even change people’s schedules directly – invite them all to a mandatory meeting without asking, for example – is too great for some kinds of leadership.

With few truly-compelling reasons for a centralised server approach, it is interesting to see that many Free Software calendar solutions do actually use the server-centric CalDAV standard. Maybe it is just the way of the world that Web-centric solutions proliferate, requiring additional standardisation to cover situations already standardised in general and for e-mail specifically. There are also solutions, Free Software and otherwise, that may or may not provide CalDAV support but which depend on calendar details being stored in IMAP-accessible mail stores: Kolab does this, but also provides a CalDAV front-end, largely for the benefit of mobile and third-party clients.

Decentralisation on Demand

Ignoring, then, the supposed requirement of a central all-knowing server, and just going along with the e-mail infrastructure we already have, we do actually have the basis for a usable calendar environment already, more or less:

• People can share events with each other (using iCalendar)
• People can schedule events amongst themselves (using iTIP, specifically iMIP)
• People can find out each other’s availability to make the scheduling more efficient (preferably using iTIP but also in other ways)

Doing it this way also allows users to opt out of any centralised mechanisms – even if only provided for convenience – that are coordinating calendar-related activities in any given organisation. If someone wants to manage their own calendar locally and not have anything in the infrastructure trying to help them, they should be able to take that route. Of course, this requires them to have capable-enough software to handle calendar data, which can be something of an issue.

That Availability Problem Mentioned Earlier

For instance, finding an e-mail program that knows how to send requests for free/busy information is a challenge, even though there are programs (possibly augmented with add-ons) that can send and understand events and other kinds of objects. In such circumstances, workarounds are required: one that I have implemented for the Lightning add-on for Thunderbird (or the Iceowl add-on for Icedove, if you use Debian) fetches free/busy details from a Web site, and it is also able to look up the necessary location of those details using LDAP. So, the resulting workflow looks like this:

1. Create or open an event.
2. Add someone to the list of people invited to that event.
3. View that person’s availability.
4. Lightning now uses the LDAP database to discover the free/busy URL.
5. It then visits the free/busy URL and obtains the data.
6. Finally, it presents the data in the availability panel.

Without LDAP, the free/busy URL could be obtained from a vCard property instead. In case you’re wondering, all of this is actually standardised or at least formalised to the level of a standard (for LDAP and for vCard).

If only I had the patience, I would add support for free/busy message exchange to Thunderbird, just as RFC 6047 would have had us do all along, and then the workflow would look like this:

1. Create or open an event.
2. Add someone to the list of people invited to an event.
3. View that person’s availability.
4. Lightning now uses the cached free/busy data already received via e-mail for the person, or it could send an e-mail to request it.
5. It now presents any cached data. If it had to send a request, maybe a response is returned while the dialogue is still open.

Some people might argue that this is simply inadequate for “real-world needs”, but they forget that various calendar clients are likely to ask for availability data from some nominated server in an asynchronous fashion, anyway. That’s how a lot of this software is designed these days – Thunderbird uses callbacks everywhere – and there is no guarantee that a response will be instant.

Moreover, a request over e-mail to a recipient within the same organisation, which is where one might expect to get someone’s free/busy details almost instantly, could be serviced relatively quickly by an automated mechanism providing such information for those who are comfortable with it doing so. We will look at such automated mechanisms in a moment.

So, there are plenty of acceptable solutions that use different grades of decentralisation without needing to resort to that “big server” approach, if only to help out clients which don’t have the features one would ideally want to use. And there are ways of making the mail infrastructure smarter as well, not just to provide workarounds for clients, but also to provide genuinely useful functionality.

Agents and Automation

Groupware solutions offer more than just a simple means for people to organise things with each other: they also offer the means to coordinate access to organisational resources. Traditionally, resources such as meeting rooms, but potentially anything that could be borrowed or allocated, would have access administered using sign-up sheets and other simple mechanisms, possibly overseen by someone in a secretarial role. Such work can now be done automatically, and if we’re going to do everything via e-mail, the natural point of integrating such work is also within the mail system.

This is, in fact, one of the things that got me interested in Kolab to begin with. Once upon a time, back at the end of my degree studies, my final project concerned mobile software agents: code that was transmitted by e-mail to run once received (in a safe fashion) in order to perform some task. Although we aren’t dealing with mobile code here, the notion still applies that an e-mail is sent to an address in order for a task to be performed by the recipient. Instead of some code sent in the message performing the task, it is the code already deployed and acting as the recipient that determines the effect of the transaction by using the event information given in the message and combining it with the schedule information it already has.

Such agents, acting in response to messages sent to particular e-mail addresses, need knowledge about schedules and policies, but once again it is interesting to consider how much information and how many infrastructure dependencies they really need within a particular environment:

• Agents can be recipients of e-mail, waiting for booking requests
• Agents will respond to requests over e-mail
• Agents can manage their own schedule and availability
• Other aspects of their operation might require some integration with systems having some organisational knowledge

In other words, we could implement such agents as message handlers operating within the mail infrastructure. Can this be done conveniently? Of course: things like mail filtering happen routinely these days, and many kinds of systems take advantage of such mechanisms so that they can be notified by incoming messages over e-mail. Can this be done in a fairly general way? Certainly: despite the existence of fancy mechanisms involving daemons and sockets, it appears that mail transport agents (MTAs) like Postfix and Exim tend to support the invocation of programs as the least demanding way of handling incoming mail.

The Missing Pieces

So what specifically is needed to provide calendaring features for e-mail users in an incremental and relatively non-invasive way? If everyone is using e-mail software that understands calendar information and can perform scheduling, the only remaining obstacles are the provision of free/busy data and, for those who need it, the provision of agents for automated scheduling of resources and other typically-inanimate things.

Since those agents are interesting (at least to me), and since they may be implemented as e-mail handler programs, let us first look at what they would do. A conversation with an agent listening to mail on an address representing a resource would work like this (ignoring sanity checks and the potential for mischief):

1. Someone sends a request to an address to book a resource, whereupon the agent provided by a handler program examines the incoming message.
2. The agent figures out which periods of time are involved.
3. The agent then checks its schedule to see if the requested periods are free for the resource.
4. Where the periods can be booked, the agent replies indicating the “attendance” of the resource (that it reserves the resource). Otherwise, the agent replies “declining” the invitation (and does not reserve the resource).

With the agent needing to maintain a schedule for a resource, it is relatively straightforward for that information to be published in another form as free/busy data. It could be done through the sending of e-mail messages, but it could also be done by putting the data in a location served by a Web server. And so, details of the resource’s availability could be retrieved over the Web by e-mail programs that elect to retrieve such information in that way.

But what about the people who are invited to events? If their mail software cannot prepare free/busy descriptions and send such information to other people, how might their availability be determined? Well, using similar techniques to those employed during the above conversation, we can seek to deduce the necessary information by providing a handler program that examines outgoing messages:

1. Someone sends a request to schedule an event.
2. The request is sent to its recipients. Meanwhile, it is inspected by a handler program that determines the periods of time involved and the sender’s involvement in those periods.
3. If the sender is attending the event, the program notes the sender’s “busy” status for the affected periods.

Similarly, when a person responds to a request, they will indicate their attendance and thus “busy” status for the affected periods. And by inspecting the outgoing response, the handler will get to know whether the person is going to be busy or not during those periods. And as a result, the handler is in a position to publish this data, either over e-mail or on the Web.

Mail handler programs can also be used to act upon messages received by individuals, too, just as is done for resources, and so a handler could automatically respond to e-mail requests for a person’s free/busy details (if that person chose to allow this). Such programs could even help support separate calendar management interfaces for those people whose mail software doesn’t understand anything at all about calendars and events.

Building on Top

So, by just adding a few handler programs to a mail system, we can support the following:

• Free/busy publishing and sharing for people whose software doesn’t support it already
• Autonomous agents managing resource availability
• Calendar interfaces for people without calendar-aware mail programs

Beyond some existing part of the mail system deciding who can receive mail and telling these programs about it, they do not need to care about how an organisation’s user information is managed. And through standardised interfaces, these programs can send messages off for storage without knowing what kind of system is involved in performing that task.

With such an approach, one can dip one’s toe into the ocean of possibilities and gradually paddle into deeper waters, instead of having to commit to the triathlon that multi-system configuration can often turn out to be. There will always be configuration issues, and help will inevitably be required to deal with them, but they will hopefully not be bound up in one big package that leads to the cry for help of “my groupware server doesn’t work any more, what has changed since I last configured it?” that one risks with solutions that try to solve every problem – real or imagined – all at the same time.

I don’t think things like calendaring should have to be introduced with a big fanfare, lots of change, a new “big box” product introducing different system components, and a stiff dose of upheaval for administrators and end-users. So, I intend to make my work available around this approach to groupware, not because I necessarily think it is superior to anything else, but because Free Software development should involve having a conversation about the different kinds of solutions that might meet user needs: a conversation that I suspect hasn’t really been had, or which ended in jeering about e-mail being a dead technology, replaced by more fashionable “social” or “responsive” technologies; a bizarre conclusion indeed when you aren’t even able to get an account with most fancy social networking services without an e-mail address.

It is possible that no-one but myself sees any merit in the strategy I have described, but maybe certain elements might prove useful or educational to others interested in similar things. And perhaps groupware will be less mysterious and more mundane as a result: not something that is exclusive to fancy cloud services, but something that is just another tiny part of the software available through each user’s and each organisation’s chosen operating system distribution, doing the hard work and not making a fuss about it. Which, like e-mail, is what it probably should have been all along.

I must admit that I haven’t been following Fairphone of late, so it was a surprise to see that vague details of the second Fairphone device have been published on the Fairphone Web site. One aspect that seems to be a substantial improvement is that of hardware modularity. Since the popularisation of the notion that such a device could be built by combining functional units as if they were simple building blocks, with a lot of concepts, renderings and position statements coming from a couple of advocacy initiatives, not much else has actually happened in terms of getting devices out for people to use and develop further. And there are people with experience of designing such end-user products who are sceptical about the robustness and economics of such open-ended modular solutions. To see illustrations of a solution that will presumably be manufactured takes the idea some way along the road to validation.

If it is possible to, say, switch out the general-purpose computing unit of the Fairphone with another one, then it can be said that even if the Fairphone initiative fails once again to deliver a software solution that is entirely Free Software, perhaps because the choice of hardware obliges the initiative to deliver opaque “binary-only” payloads, then the opportunity might be there for others to deliver a bottom-to-top free-and-open solution as a replacement component. But one might hope that it should not be necessary to “opt in” to getting a system whose sources can be obtained, rebuilt and redeployed: that the second Fairphone device might have such desirable characteristics out of the box.

Now, it does seem that Fairphone acknowledges the existence and the merits of Free Software, at least in very broad terms. Reading the support site provides us with an insight into the current situation with regard to software freedom and Fairphone:

Our goal is to take a more open source approach to be able to offer owners more choice and control over their phone’s OS. For example, we want to make the source code available to the developer community and we are also in discussions with other OS vendors to look at the possibility of offering alternative operating systems for the Fairphone 2. However, at the moment there are parts of the software that are owned or licensed by third parties, so we are still investigating the technical and legal requirements to accomplish our goals of open software.

First of all, ignoring vague terms like “open software” that are susceptible to “openwashing” (putting the label “open” on something that really isn’t), it should be noted that various parts of the deployed software will, through their licensing, oblige the Fairphone initiative to make the corresponding source code available. This is not a matter that can be waved away with excuses about people’s hands being tied, that it is difficult to coordinate, or whatever else the average GPL-violating vendor might make. If copyleft-licensed code ships, the sources must follow.

Now there may also be proprietary software on the device (or permissively-licensed software bearing no obligation for anyone to release the corresponding source, which virtually amounts to the same thing) and that would clearly be against software freedom and should be something Fairphone should strongly consider avoiding, because neither end-users nor anyone who may wish to help those users would have any control over such software, and they would be completely dependent on the vendor, who in turn would be completely dependent on their suppliers, who in turn might suddenly not care about the viability of that software or the devices on which it is deployed. So much for sustainability under such circumstances!

As I noted before, having control over the software is not a perk for those who wish to “geek out” over the internals of a product: it is a prerequisite for product viability, longevity and sustainability. Let us hope that Fairphone can not only learn and apply the lessons from their first device, which may indeed have occurred with the choice of a potentially supportable chipset this time around, but that the initiative can also understand and embrace their obligations to those who produced the bulk of their software (as well as to their customers) in a coherent and concrete fashion. It would be a shame if, once again, an unwillingness to focus on software led to another missed opportunity, and the need for another version of the device to be brought to market to remedy deficiencies in what is otherwise a well-considered enterprise.

Now, if only Fairphone could organise their Web site in a more coherent fashion, putting useful summaries of essential information in obvious places instead of being buried in some random forum post…

The Mailpile project has been soliciting feedback on the licensing of their product, but I couldn’t let one of the public responses go by without some remarks. Once upon a time, as many people may remember, a disinformation campaign was run by Microsoft to attempt to scare people away from copyleft licences, employing insensitive terms like “viral” and “cancer”. And so, over a decade later, here we have an article employing the term “viral” liberally to refer to copyleft licences.

Now, as many people already know, copyleft licences are applied to works by their authors so that those wishing to contribute to the further development of those works will do so in a way that preserves the “share-alike” nature of those works. In other words, the recipient of such works promises to extend to others the privileges they experienced themselves upon receiving the work, notably the abilities to see and to change how it functions, and the ability to pass on the work, modified or not, under the same conditions. Such “fair sharing” is intended to ensure that everyone receiving such works may be equal participants in experiencing and improving the work. The original author is asking people to join them in building something that is useful for everyone.

Unfortunately, all this altruism is frowned upon by some individuals and corporations who would prefer to be able to take works, to use, modify and deploy them as they see fit, and to refuse to participate in the social contract that copyleft encourages. Instead, those individuals and corporations would rather keep their own modifications to such works secret, or even go as far as to deny others the ability to understand and change any part of those works whatsoever. In other words, some people want a position of power over their own users or customers: they want the money that their users and customers may offer – the very basis of the viability of their precious business – and in return for that money they will deny their users or customers the opportunity to know even what goes into the product they are getting, never mind giving them the chance to participate in improving it or exercising control over what it does.

From the referenced public response to the licensing survey, I learned another term: “feedstock”. I will admit that I had never seen this term used before in the context of software, or I don’t recall its use in such a way, but it isn’t difficult to transfer the established meaning of the word to the context of software from the perspective of someone portraying copyleft licences as “viral”. I suppose that here we see another divide being erected between people who think they should have most of the power (and who are somehow special) and the grunts who merely provide the fuel for their success: “feedstock” apparently refers to all the software that enables the special people’s revenue-generating products with their “secret ingredients” (or “special sauce” as the author puts it) to exist in the first place.

It should be worrying for anyone spending any time or effort on writing software that by permissively licensing your work it will be treated as mere “feedstock” by people who only appreciate your work as far as they could use it without giving you a second thought. To be fair, the article’s author does encourage contributing back to projects as “good karma and community”, but then again this statement is made in the context of copyleft-licensed projects, and the author spends part of a paragraph bemoaning the chore of finding permissively-licensed projects so as to not have to contribute anything back at all. If you don’t mind working for companies for free and being told that you don’t deserve to see what they did to your own code that they nevertheless couldn’t get by without, maybe a permissive licence is a palatable choice for you, but remember that the permissive licensing will most likely be used to take privileges away from other recipients: those unfortunates who are paying good money won’t get to see how your code works with all the “secret stuff” bolted on, either.

Once upon a time, Bill Gates remarked, “A lot of customers in a sense don’t want — the notion that they would go in and tinker with the source code, that’s the opposite of what’s supposed to go on. We’re supposed to give that to them and it’s our problem to make sure that it works perfectly and they rely on us.” This, of course, comes from a man who enjoys substantial power through accumulation of wealth by various means, many of them involving the denial of choice, control and power to others. It is high time we all stopped listening to people who disempower us at every opportunity so that they can enrich themselves at our expense.

Having recently seen an article about the closure of a project featuring that project’s usage of proprietary tools from Atlassian – specifically JIRA and Confluence – I thought I would share my own experiences from the migration of another project’s wiki site that had been using Confluence as a collaborative publishing tool.

Quite some time ago now, a call for volunteers was posted to the FSF blog, asking for people familiar with Python to help out with a migration of the Mailman Wiki from Confluence to MoinMoin. Subsequently, Barry Warsaw followed up on the developers’ mailing list for Mailman with a similar message. Unlike the project at the start of this article, GNU Mailman was (and remains) a vibrant Free Software project, but a degree of dissatisfaction with Confluence, combined with the realisation that such a project should be using, benefiting from, and contributing to Free Software tools, meant that such a migration was seen as highly desirable, if not essential.

Up and Away

Initially, things started off rather energetically, and Bradley Dean initiated the process of fact-finding around Confluence and the Mailman project’s usage of it. But within a few months, things apparently became noticeably quieter. My own involvement probably came about through seeing the ConfluenceConverter page on the MoinMoin Wiki, looking at the development efforts, and seeing if I couldn’t nudge the project along by pitching in with notes about representing Confluence markup format features in the somewhat more conventional MoinMoin wiki syntax. Indeed, it appears that my first contribution to this work occurred as early as late May 2011, but I was more or less content to let the project participants get on with their efforts to understand how Confluence represents its data, how Confluence exposes resources on a wiki, and so on.

But after a while, it occurred to me that the volunteers probably had other things to do and that progress had largely stalled. Although there wasn’t very much code available to perform concrete migration-related tasks, Bradley had gained some solid experience with the XML format employed by exported Confluence data, and such experience when combined with my own experiences dealing with very large XML files in my day job suggested an approach that had worked rather well with such large files: performing an extraction of the essential information, including identifiers and references that communicate the actual structure of the information, as opposed to the hierarchical structure of the XML data itself. With the data available in a more concise and flexible form, it can then be processed in a more convenient fashion, and within a few weeks I had something ready to play with.

With a day job and other commitments, it isn’t usually possible to prioritise volunteer projects like this, and I soon discovered that some other factors were involved: technological progress, and the tendency for proprietary software and services to be upgraded. What had initially involved the conversion of textual content from one markup format to another now seemed to involve the conversion from two rather different markup formats. All the effort documenting the original Confluence format now seemed to be almost peripheral if not superfluous: any current content on the Mailman Wiki would now be in a completely different format. And volunteer energy seemed to have run out.

A Revival

Time passed. And then the Mailman developers noticed that the Confluence upgrade had made the wiki situation even less bearable (as indeed other Confluence users had noticed and complained about), and that the benefits of such a solution were being outweighed by the inconveniences of the platform. And it was at this point that I realised that it was worthwhile continuing the migration effort: it is bad enough that people feel constrained by a proprietary platform over which they have little control, but it is even worse when it appears that they will have to abandon their content and start over with little or no benefit from all the hard work they have invested in creating and maintaining that content in the first place.

And with that, I started the long process of trying to support not only both markup formats, but also all the features likely to have been used by the Mailman project and those using its wiki. Some might claim that Confluence is “powerful” by supporting a multitude of seemingly exotic features (page relationships, comments, “spaces”, blogs, as well as various kinds of extensions), but many of these features are rarely used or never actually used at all. Meanwhile, as many migration projects can attest, if one inadvertently omits some minor feature that someone regards as essential, one risks never hearing the end of it, especially if the affected users have been soaking up the propaganda from their favourite proprietary vendor (which was thankfully never a factor in this particular situation).

Despite the “long tail” of feature support, we were able to end 2012 with some kind of overview of the scope of the remaining work. And once again I was able to persuade the concerned parties that we should focus on MoinMoin 1.x not 2.x, which has proved to be the correct decision given the still-ongoing status of the latter even now in 2015. Of course, I didn’t at that point anticipate how much longer the project would take…

Iterating

Over the next few months, I found time to do more work and to keep the Mailman development community informed again and again, which is a seemingly minor aspect of such efforts but is essential to reassure people that things really are happening: the Mailman community had, in fact, forgotten about the separate mailing list for this project long before activity on it had subsided. One benefit of this was to get feedback on how things were looking as each iteration of the converted content was made available, and with something concrete to look at, people tend to remember things that matter to them that they wouldn’t otherwise think of in any abstract discussion about the content.

In such processes, other things tend to emerge that initially aren’t priorities but which have to be dealt with eventually. One of the stated objectives was to have a full history, meaning that all the edits made to the original content would need to be preserved, and for an authentic record, these edits would need to preserve both timestamp and author information. This introduced complications around the import of converted content – it being no longer sufficient to “replay” edits and have them assume the timestamp of the moment they were added to the new wiki – as well as the migration and management of user profiles. Particularly this latter area posed a problem: the exported data from Confluence only contained page (and related) content, not user profile information.

Now, one might not have expected user details to be exportable anyway due to potential security issues with people having sufficient privileges to request a data dump directly from Confluence and then to be able to obtain potentially sensitive information about other users, but this presented another challenge where the migration of an entire site is concerned. On this matter, a very pragmatic approach was taken: any user profile pages (of which there were thankfully very few) were retrieved directly over the Web from the existing site; the existence of user profiles was deduced from the author metadata present in the actual exported wiki content. Since we would be asking existing users to re-enable their accounts on the new wiki once it became active, and since we would be avoiding the migration of spammer accounts, this approach seemed to be a reasonable compromise between convenience and completeness.

Getting There

By November 2013, the end was in sight, with coverage of various “actions” supported by Confluence also supported in the migrated wiki. Such actions are a good example of how things that are on the edges of a migration can demand significant amounts of time. For instance, Confluence supports a PDF export action, and although one might suggest that people just print a page to file from their browser, choosing PDF as the output format, there are reasonable arguments to be made that a direct export might also be desirable. Thus, after a brief survey of existing options for MoinMoin, I decided it would be useful to provide one myself. The conversion of Confluence content had also necessitated the use of more expressive table syntax. Had I not been sufficiently interested in implementing improved table facilities in MoinMoin prior to this work, I would have needed to invest quite a bit of effort in this seemingly peripheral activity.

Again, time passed. Much of the progress occurred off-list at this point. In fact, a degree of confusion, miscommunication and elements of other factors conspired to delay the availability of the infrastructure on which the new wiki would be deployed. Already in October 2013 there had been agreement about hosting within the python.org infrastructure, but the matter seemed to stall despite Barry Warsaw trying to push it along in February and April 2014. Eventually, after complaining from me on the PSF members’ mailing list at the end of May, some motion occurred on the matter and in July the task of provisioning the necessary resources began.

After returning from a long vacation in August, the task of performing the final migration and actually deploying the content could finally begin. Here, I was able to rely on expert help from Mark Sapiro who not only checked the results of the migration thoroughly, but also configured various aspects of the mail system functionality (one of the benefits of participating in a mail-oriented project, I guess), and even enhanced my code to provide features that I had overlooked. By September, we were already playing with the migrated content and discussing how the site would be administered and protected from spam and vandalism. By October, Barry was already confident enough to pre-announce the migrated site!

At Long Last

Alas, things stalled again for a while, perhaps due to other commitments of some of the volunteers needed to make the final transition occur, but in January the new Mailman Wiki was finally announced. But things didn’t stop there. One long-standing volunteer, Jim Tittsler, decided that the visual theme of the new wiki would be improved if it were made to match the other Mailman Web resources, and so he went and figured out how to make a MoinMoin theme to do the job! The new wiki just wouldn’t look as good, despite all the migrated content and the familiarity of MoinMoin, if it weren’t for the special theme that Jim put together.

The all-new Mailman Wiki with Jim Tittsler's theme

There have been a few things to deal with after deploying the new wiki. Spam and vandalism have not been a problem because we have implemented a very strict editing policy where people have to request editing access. However, this does not prevent people from registering accounts, even if they never get to use them to do anything. To deal with this, we enabled textcha support for new account registrations, and we also enabled e-mail verification of new accounts. As a result, the considerable volume of new user profiles that were being created (potentially hundreds every hour) has been more or less eliminated.

It has to be said that throughout the process, once it got started in earnest, the Mailman development community has been fantastic, with constructive feedback and encouragement throughout. I have had disappointing things to say about the experience of being a volunteer with regard to certain projects and initiatives, but the Mailman project is not that kind of project. Within the limits of their powers, the Mailman custodians have done their best to enable this work and to see it through to the end.

Some Lessons

I am sure that offers of “for free” usage of certain proprietary tools and services are made in a genuinely generous way by companies like Atlassian who presumably feel that they are helping to make Free Software developers more productive. And I can only say that those interactions I experienced with Contegix, who were responsible for hosting the Confluence instance through which the old Mailman Wiki was deployed, were both constructive and polite. Nevertheless, proprietary solutions are ultimately disempowering: they take away the control over the working environment that users and developers need to have; they direct improvement efforts towards themselves and away from Free Software solutions; they also serve as a means of dissuading people from adopting competing Free Software products by giving an indication that only they can meet the rigorous demands of the activity concerned.

I saw a position in the Norwegian public sector not so long ago for someone who would manage and enhance a Confluence installation. While it is not for me to dictate the tools people choose to do their work, it seems to me that such effort would be better spent enhancing Free Software products and infrastructure instead of remedying the deficiencies of a tool over which the customer ultimately has no control, to which the customer is bound, and where the expertise being cultivated is relevant only to a single product for as long as that product is kept viable by its vendor. Such strategic mistakes occur all too frequently in the Norwegian public sector, with its infatuation with proprietary products and services, but those of us not constrained by such habits can make better choices when choosing tools for our own endeavours.

I encourage everyone to support Free Software tools when choosing solutions for your projects. It may be the case that such tools may not at first offer precisely the features you might be looking for, and you might be tempted to accept an offer of a “for free” product or to use a no-cost “cloud” service, and such things may appear to offer an easier path when you might otherwise be confronted with a choice of hosting solutions and deployment issues. But there are whole communities out there who can offer advice and will support their Free Software project, and there are Free Software organisations who will help you deploy your choice of tools, perhaps even having it ready to use as part of their existing infrastructure.

In the end, by embracing Free Software, you get the control you need over your content in order to manage it sustainably. Surely that is better than having some random company in charge, with the ever-present risk of them one day deciding to discontinue their service and/or, with barely enough notice, discard your data.

It didn’t all start with a poorly-considered April Fools’ joke about hosting a Python conference in Cuba, but the resulting private mailing list discussion managed to persuade me not to continue as a voting member of the Python Software Foundation (PSF). In recent years, upon returning from vacation, discovering tens if not hundreds of messages whipping up a frenzy about some topic supposedly pertinent to the activities of the PSF, and reading through such messages as if I should inform my own position on the matter, was undoubtedly one of the chores of being a member. This time, my vacation plans were slightly unusual, so I was at least spared the surprise of getting the bulk of people’s opinions in one big serving.

I was invited to participate in the PSF at a time when it was an invitation-only affair. My own modest contributions to the EuroPython conference were the motivating factor, and it would seem that I hadn’t alienated enough people for my nomination to be opposed. (This cannot be said for some other people who did eventually become members as well after their opponents presumably realised the unkindness of their ways.) Being asked to participate was an honour, although I remarked at the time that I wasn’t sure what contribution I might make to such an organisation. Becoming a Fellow of the FSFE was an active choice I made myself because I align myself closely with the agenda the FSFE chooses to pursue, but the PSF is somewhat more vague or more ambivalent about its own agenda: promoting Python is all very well, but should the organisation promote proprietary software that increases Python adoption, or would this undermine the foundations on which Python was built and is sustained? Being invited to participate in an organisation with often unclear objectives combines a degree of passivity with an awareness that some of the decisions being taken may well contradict some of the principles I have actively chosen to support in other organisations. Such as the FSFE, of course.

Don’t get me wrong: there are a lot of vital activities performed within the PSF. For instance, the organisation has a genuine need to enforce its trademarks and to stop other people from claiming the Python name as their own, and the membership can indeed assist in such matters, as can the wider community. But looking at my archives of the private membership mailing list, a lot of noise has been produced on other, more mundane matters. For a long time, it seemed as if the only business of the PSF membership – as opposed to the board who actually make the big decisions – was to nominate and vote on new members, thus giving the organisation the appearance of only really existing for its own sake. Fortunately, organisational reform has made the matter of recruiting members largely obsolete, and some initiatives have motivated other, more meaningful activities. However, I cannot be the only person who has noted that such activities could largely be pursued outside the PSF and within the broader community instead, as indeed these activities typically are.

PyCon

Some of the more divisive topics that have caused the most noise have had some connection with PyCon: the North American Python conference that mostly replaced the previous International Python Conference series (from back when people thought that conferences had to be professionally organised and run, in contrast to PyCon and most, if not all, other significant Python conferences today). Indeed, this lack of separation between the PSF and PyCon has been a significant concern of mine. I will probably never attend a PyCon, partly because it resides in North America as a physical event, partly because its size makes it completely uninteresting to me as an attendee, and largely because I increasingly find the programme uninteresting for a variety of other reasons. When the PSF members’ time is spent discussing or at least exposed to the discussion of PyCon business, it can just add to the burden of membership for those who wish to focus on the supposed core objectives of the organisation.

What may well be worse, however, is that PyCon exposes the PSF to substantial liability issues. As the conference headed along a trajectory of seemingly desirable and ambitious growth, it collided with the economic downturn caused by the global financial crisis of 2008, incurring a not insignificant loss. Fortunately, this outcome has not since been repeated, and the organisation had sufficient liquidity to avoid any serious consequences. Some have argued that it was precisely because profits from previous years’ conferences had been accumulated that the organisation was able to pay its bills, but such good fortune cannot explain away the fundamental liability and the risks it brings to the viability of the organisation, especially if fortune happens not to be on its side in future.

Volunteering

In recent times, I have been more sharply focused on the way volunteers are treated by organisations who rely on their services to fulfil their mission. Sadly, the PSF has exhibited a poor record in various respects on this matter. Once upon a time, the Python language Web site was redesigned under contract, but the burden of maintenance fell on community volunteers. Over time, discontentment forced the decision to change the technology and a specification was drawn up under a degree of consultation. Unfortunately, the priorities of certain stakeholders – that is, community volunteers doing a fair amount of hard work in their own time – were either ignored or belittled, leaving them confronted with either having to adapt to a suboptimal workflow not of their own choosing, as well as spending time and energy developing that workflow, or just quitting and leaving it to other people to tidy up the mess that those other people (and the hired contractors) had made.

Understandably, the volunteers quit, leaving a gap in the Web site functionality that took a year to reinstate. But what was most disappointing was the way those volunteers were branded as uncooperative and irresponsible in an act of revisionism by those who clearly failed to appreciate the magnitude of the efforts of those volunteers in the first place. Indeed, the views of the affected volunteers were even belittled when efforts were championed to finally restore the functionality, with it being stated by one motivated individual that the history of the problem was not of his concern. When people cannot themselves choose the basis of their own involvement in a volunteer-run organisation without being vilified for letting people down or for “holding the organisation to ransom”, the latter being a remarkable accusation given the professionalism that was actually shown in supporting a transition to other volunteers, one must question whether such an organisation deserves to attract any volunteers at all.

Politics

As discussion heated up over the PyCon Cuba affair, the usual clash of political views emerged, with each side accusing the other of ignorance and not understanding the political or cultural situation, apparently blinkered by their own cultural and political biases. I remember brazen (and ill-informed) political advocacy being a component in one of the Python community blogging “planets” before I found the other one, back when there was a confusing level of duplication between the two and when nobody knew which one was the “real” one (which now appears to consist of a lot of repetition and veiled commercial advertising), and I find it infuriating when people decide to use such matters as an excuse to lecture others and to promote their own political preferences.

I have become aware of a degree of hostility within the PSF towards the Free Software Foundation, with the latter being regarded as a “political” organisation, perhaps due to hard feelings experienced when the FSF had to educate the custodians of Python about software licensing (which really came about in the first place because of the way Python development had been moved around, causing various legal representatives to play around with the licensing, arguably to make their own mark and to stop others getting all the credit). And I detect a reluctance in some quarters to defend software freedom within the PSF, with a reluctance to align the PSF with other entities that support software and digital freedoms. At least the FSF can be said to have an honest political agenda, where those who support it more or less know where they stand.

In contrast, the PSF seems to cultivate all kinds of internal squabbling and agenda-setting: true politics in the worst sense of the word. On one occasion I was more or less told that my opinion was not welcome or, indeed, could ever be of interest on a topic related to diversity. Thankfully, diversity politics moved to a dedicated mailing list and I was thereafter mostly able to avoid being told by another Anglo-Saxon male that my own perspectives didn’t matter on that or on any other topic. How it is that someone I don’t actually know can presume to know in any detail what perspectives or experiences I might have to offer on any matter remains something of a mystery to me.

Looking through my archives, there appears to be a lot of noise, squabbling, quipping, and recrimination over the last five years or so. In the midst of the recent finger-wagging, someone dared to mention that maybe Cubans, wherever they are, might actually deserve to have a conference. Indeed, other places were mentioned where the people who live there, through no fault of their own, would also be the object of political grandstanding instead of being treated like normal people wanting to participate in a wider community.

I mostly regard April Fools’ jokes as part of a tedious tradition, part of the circus that distracts people away from genuine matters of concern, perhaps even an avenue of passive aggression in certain circles, a way to bully people and then insist – as cowards do – that it was “just a joke”. The lack of a separation of the PSF’s interests combined with the allure of the circus conspired to make fools out of the people involved in creating the joke and of many in the accompanying debate. I find myself uninterested in spending my own time indulging such distractions, especially when those distractions are products of flaws in the organisation that nobody wishes to fix, and when there are more immediate and necessary activities to pursue in the wider arena of Free Software that, as a movement in its own right, some in the PSF practically refuse to acknowledge.

Effects

Leaving the PSF won’t really change any of my commitments, but it will at least reduce the level of background noise I have to deal with. Such an underwhelming and unfortunate assessment is something the organisation will have to rectify in time if it wishes to remain relevant and to deserve the continued involvement of its members. I do have confidence in some of the reform and improvement processes being conducted by volunteers with too little time of their own to pursue them, and I hope that they make the organisation a substantially better and more effective one, as they continue to play to an audience of people with much to say but, more often than not, little to add.

I would have been tempted to remain in the PSF and to pursue various initiatives if the organisation were a multiplier of effect for any given input of effort. Instead, it currently acts as a divider of effect for all the effort one would apparently need to put in to achieve anything. That isn’t how any organisation, let alone one relying on volunteer time and commitment, should be functioning.

A Footnote

On political matters and accusations of ignorance being traded, my own patience is wearing thin indeed, and this probably nudged me into finally making this decision. It probably doesn’t help that I recently made a trip to Britain where election season has been in full swing, with unashamed displays of wilful idiocy openly paraded on a range of topics, indulged by the curated ignorance of the masses, with the continued destruction of British society, nature and the economy looking inevitable as the perpetrators insist they know best now and will undoubtedly in the future protest their innocence when challenged on the legacy of their ruinous rule, adopting the “it wasn’t me” manner of a petulant schoolchild so befitting of the basis of the nepotism that got most of them where they are today.