Paul Boddie's Free Software-related blog
Paul's activities and perspectives around Free Software
Packaging Kolab for Debian using pbuilder
My recent excursion into Debian packaging with Kolab has involved a tour of lots of different tools and services, but it started out with a brief attempt to build the existing packages with pbuilder: a tool that has become fairly familiar to me in the process of experimenting with Debian packages and even contributing one to “Debian proper”. After some initial frustrations that prevented me from building packages using my normal workflow, I decided to familiarise myself with the infrastructure that the Kolab project itself uses to make packages, if only to reassure myself that the packages really could be built and didn’t require any special magic to do so. I won’t go into this because Timotheus has already done so in sufficient depth.
After some playing around with osc and the Development project in the Kolab OBS (Open Build Service), building packages, installing them in Debian root filesystems and User Mode Linux instances (administered by some scripts I’ve developed over the years), I persuaded myself to have another go at feeding the packages to pbuilder via the pdebuild tool, determined to overcome any build issues and to demonstrate that it could be done. One fairly good reason for doing this is that even though pbuilder-based builds can be sluggish as pbuilder decides to unpack an existing filesystem, install build dependencies, and do other housekeeping before actually building the package, it seems to be more efficient and quicker than osc, which I found took as long as 9 minutes before it was ready to build a relatively straightforward Python-based package. Another reason for going with pbuilder is that it is what the Debian project itself will be using, more or less, if – hopefully when – the packages get accepted back into Debian, whereas the OBS infrastructure seems to be based on other technologies for the management of the build environment.
Packages of the Future
One challenge posed by Kolab is that some of its packages depend on some of the other Kolab-provided packages when being built – those other packages are so-called “build dependencies” – but tools like pdebuild rely on such build dependencies already being available as installable packages from the Debian archive. For things like make and gcc, this isn’t a problem at all: they were packaged a very long time ago (although I suppose you could get caught out with very recent versions of such packages). But when a yet-to-be-submitted package is required as a build dependency for another package, the issue of satisfying that dependency arises, and this was something I hadn’t encountered before.
A perusal of the Debian Wiki provided a solution: after building the packages that the other ones rely upon, put them in a directory, expose them in a repository, and then make pbuilder consult that repository when deciding how it can satisfy those build dependencies. This involves three things:
- A directory with newly built packages, obviously, together with the necessary package metadata.
- A hook script that will refresh the metadata and perform the necessary update action that lets the pbuilder environment know about the packages.
- A special configuration for pbuilder.
The Package Directory
At first, this will contain nothing at all, but as you add packages it will contain both .deb package files and the package metadata. We will call this directory deps.
The Hook Script
As the Debian Wiki page describes, a hook called D05deps can be placed in a directory called, say, hooks and populated with the following code:
#!/bin/sh (cd /path-to-kolab-packaging/deps; apt-ftparchive packages . > Packages) apt-get update
I did find that the permissions on the hook file were crucial, fixing them as follows:
chmod a+x hooks/D05deps
Otherwise, no mention of the script will be made in the copious output from pbuilder and it will simply be ignored. I also experienced some initial problems with the package metadata, but running the first line of actual commands from the script and manually producing the metadata was enough to get pbuilder on the right track:
cd deps; apt-ftparchive packages . > Packages
After that, it didn’t have any difficulties seeing the new packages as I added them to the deps directory.
The Configuration File
All this has to do is to point to the deps and hooks directories. You can more or less copy the contents of /etc/pbuilderrc and add the following to it (customising it for your own choices, of course):
OTHERMIRROR="deb [trusted=yes] file:///path-to-kolab-packaging/deps ./" BINDMOUNTS="/path-to-kolab-packaging/deps" HOOKDIR="/path-to-kolab-packaging/hooks" EXTRAPACKAGES="apt-utils"
Yes, there really isn’t anything different about this than the example on the Debian Wiki page. I put this configuration file alongside the deps and hooks directories and called it pbuilderrc.
Actually Building Packages
With the above extra stuff in place, the process of building packages is slightly different: you have to tell pbuilder to use this alternative configuration, and then you just hope that all the different aspects of it are consistent and that pbuilder is able to take notice of it. The command that will eventually be run inside a directory containing “Debianized” sources is the following:
pdebuild -- --distribution wheezy --override-config --configfile ../pbuilderrc
Obviously, the pbuilderrc file resides in the parent directory after you change into a package’s sources directory.
Build Order
Above, I mentioned that some packages depend on others in order to be built. Finding out which packages are affected involves consulting their build dependencies which are conveniently listed in their .dsc files. Doing the following permits a general overview to be obtained and the basis of a suitable build order to be worked out:
grep ^Build-Depends *.dsc
Obviously, it makes sense to start with packages that do not depend on others that are also being built for this exercise. Devising or discovering an automated approach for this is left as an exercise for the reader, but Kolab is relatively uncomplicated and I used the following build order:
python-icalendar pykolab libkolabxml libcalendaring libkolab kolab kolab-freebusy kolab-schema kolab-syncroton kolab-utils kolab-webadmin mozilla-ldap-sdk chwala irony pyasn1-modules php-http-request2 roundcubemail roundcubemail-plugin-contextmenu roundcubemail-plugin-dblog roundcubemail-plugin-threadingasdefault roundcubemail-plugins-kolab smarty3
The General Workflow
To obtain, unpack and build the packages I used the following workflow:
- Visit the package downloads page (found via the Development project’s repository overview) and obtain a list of package URLs.
- Get each package using dget from the devscripts package. This will probably give a warning about unsigned or unverifiable packages and not unpack the sources. (I suppose that importing the repository key using apt-key fixes this. One should obviously consider the risks and recommendations around downloading code from the Internet.)
- Unpack each package using dpkg-source. For example:
dpkg-source -x python-icalendar_3.4-1.dsc
- Change into the source directory.
- Run the pdebuild command given above.
- Copy or move the resulting package files from /var/cache/pbuilder/result into the deps directory.
- Optional but tidy: move the other artefacts of building from the parent directory into some other place for future reference.
There are probably much more efficient and cleaner ways of building lots of packages, but this allowed me to inspect them and to consider a few changes.
Making Changes
I took the opportunity to make some changes to the python-icalendar package because I saw that it wanted me to install python-setuptools before pdebuild would even launch pbuilder. I have little confidence in setuptools generally and would prefer not to have it on my system, and it is an unfortunate but recurring phenomenon that one finds the setup.py script commonly used for the preparation or installation of Python software packages using setuptools when its functionality only requires the older and less disruptive distutils library. Regardless of whether such changes are desired in the eventual Kolab packaging, I took the opportunity to investigate how changes should be made to the package.
Modern Debian packages prefer such “upstream patching” – where the code being changed originates with the actual developers of the software, as opposed to people packaging it for different distributions – to be done using a tool called quilt. I have some experience using quilt for my previous packaging work, but it’s easy to forget how to use it. Fortunately, the Debian Wiki came to the rescue once again. Here’s what I did in the sources directory for the package:
quilt new setup.py.patch # tell quilt about my patch quilt add setup.py # tell quilt that I will patch setup.py # Now, I edited setup.py to replace setuptools with distutils. quilt refresh # update the patch within quilt quilt pop -a # go back to the way everything was (but remember the patch for later)
When running pdebuild, the tool will notice such patches and apply them. Upon finishing they should be provided in a file containing all the necessary patches that allow the software to be built as a package. In this case, a file called python-icalendar_3.4-1.debian.tar.gz was produced.
One or two things are probably necessary to make this work:
- A suitable quilt configuration as described on the Debian Wiki page referenced above.
- A suitable debian/source/format file containing the 3.0 (quilt) value.
What Next?
Some of the previously encountered pitfalls had very little to do with the actual packaging of Kolab in terms of getting the software installed, but were more to do with the way it behaved once configured (and perhaps how the configuration gets done). I intend to look a bit more closely at the configuration process and to see if some of the awkward situations that may arise can’t be diagnosed and remedied by some helpful enhancements to the tools. On the way, I expect to find areas of improvement in the ways some things are done – that’s just the way things are with software – but with regard to the packaging itself most of the hard work has already been done and it seems to hold up rather well. So thanks are obviously due to Paul and Jeroen (and others) for allowing me to join in at this fairly late stage in the game.