Bobulate

I wrote this on Saturday afternoon, but didn’t hit “post”: After running fsck repeatedly until it finally stopped finding unreferenced files, I’m hopefully calling the disk array fixed on the EBN. I’m still going to reconfigure the server in some way, but for the time being I’ve restarted the VM for api.kde.org and the EBN. That means that api.kde.org is accessible again and the EBN is running. I’m hoping that the NIC and RAID will hold out. It will take a while for API regeneration to finish as well as a new round of Krazy checks to run.

And I would add this today: the RAID array didn’t survive the night, with new read timeouts on the disk followed by mirror corruption and end of story. So I rebooted, dropped that VM again and the machine is currently running only essential services. We’re in the process of moving things off of the machine now so that it is easier to reinstall, with fewer tasks. Then we’ll hopefully have something usable again.

I end up writing so much about OpenSolaris that people sometimes assume I work for Sun or Oracle. Not the case. The KDE4-OpenSolaris project is my development hobby, and isn’t part of any employment at all. I worked for the FSFE until recently, and nowadays I work in the garden while looking for something new and suitable to do. Besides the work and hobby bits, I have my responsibilities as a volunteer board member of both the NLUUG and KDE e.V.

This weekend, I’m in Berlin for a meeting of the board of KDE e.V. A board meeting is partly a team-building exercise, partly an opportunity to synchronize the various projects that KDE e.V. is working on, partly a legal, financial and administrative exercise and lastly (but perhaps most importantly) a chance to plan what we will be doing in the next six months or so. There’s a few things we’ll be talking about, like supporting our version control systems, growing the membership of the association, conference and sprint organization. Many of our activities are mandated or demanded by the community — like keeping our servers up-to-date and useful for software development done by the KDE community as a whole. One small topic might be financing a small spare server to speed up the recovery of api.kde.org (ok, that’s my hobby as well). The financial stability of the organization — for financing sprints and conferences and whatever the development community needs — is another one of those topics that always takes some time.

If you’re in Berlin, feel free to stop by for lunch (around 1pm) or a beer later at the KDE e.V. office in Linienstrasse — drop me a note.

After a few upgrades on various desktop and laptop machines (don’t get me started about the EBN machine, which is curiously unstable under FreeBSD 8.0) so that my main development workstation is running OSOL build nv134, I’ve kicked off a new round of OSOL builds. Of course, I ran into the 134 upgrade issue where /dev/ptmx ends up with the wrong permissions, but that’s documented along with a workaround in the release notes. The new set of builds is plain KDE SC 4.4.3, with only a few updates: Assuan is updated to 2.0.0. This disables parts of KDE PIM, at least until I get KMail recompiled so that I can get at the patch that Will Stephenson sent me two weeks ago to handle Assuan2. ICU4C is updated to 4.4.1. The FOSShier package makes a new appearance, so you can easily manage KDE SC separately from the dependencies that KDE4 has.

For the bits-that-I-haven’t-tried-myself, I think KDM now partly works on OSOL, but it doesn’t run all of the scripts that GDM does, so you don’t get a very good desktop experience out of it.

On the whole, it’s not really exciting. We (as in the KDE4-OpenSolaris folks) have decided to give the KDE SC 4.5 betas a miss for now — I know that trunk was having some interesting template issues a few weeks again, so I have serious doubts it will compile at all — and just try to get the current crop of packages in the best shape possible for OpenSolaris.

There’s an OpenSolaris 2010.03 upcoming, some day. Yes, that 03 was supposed to be March, but clearly corporate takeovers got in the way. There are some pretty big changes in packaging there (in particular, package naming) so in order to make the KDE4 packages integrate cleanly we’ll have to do some fixing there.

As always, the specfiles can be had from our specfile repo; however, packages are not up-to-date on the package server because of stability issues.

As Luca B. and some others have pointed out already, the EBN is down again. Read errors on one disk followed by a panic on one of the filesystems overnight. This means plenty of things offline again, and by now the situation is such that I’m going to have to take drastic measures. That means fetching the disk array and rebuilding most of the machine. In the meantime I will leave the two SAS disks in so that some of the essential services can be restored, but it means that the EBN and api.kde.org (which use up most of the disk space, processor power and bandwidth) will be down for as long as it takes.

Needless to say, anyone with a nice 1U or 2U storage unit they are willing to donate to KDE is really welcome to step up.

Well, it took ages, but the EBN and the different VMs it hosts are back. Add “sysadmin” to the list of occupations I probably shouldn’t attempt without (1) more training (2) a stricter schedule. The NLUUG spring conference on systems administration was quite educational — and fun, too, chatting with various companies and learning about NanoBSD and ZFS — but it didn’t give me any magical beans to fix what ailed the EBN.

So what was the problem? Well, the whole thing started (yay, placing the blame!) with Bertjan, who wanted a newer Qt version on the EBN for his software quality checking tools. The EBN ran 6.2-R, and the necessary Qt versions and stuff are not supported on that OS anymore. While the EOL for FreeBSD 6 is still six months away, the ports maintainers don’t necessarily want to support that. So we needed to update the OS to something newer.

There’s tools to do that now, but I’ve never used them, and anyway I don’t think they support FreeBSD 6. So that means lots of “make buildworld buildkernel installkernel installworld” kinds of steps. First off I found that doing the compilations took a lot longer than I expected (or hoped). So where I planned to go 6-6.4-7.3-8.0 in one day, the fact was that just compiling was going to take longer than that. I couldn’t pre compile everything either with the machine still up, because FreeBSD 8 doesn’t compile in a FreeBSD 6 environment. Hence the multiple steps. Note to self: update more frequently to avoid this kind of large upgrade.

Second problem was that the jails (virtual machines) on the server were poorly set up. They all had their own copies of the world. I hadn’t realized that a 6.2 jail wouldn’t work in a 7.3 host (for instance, ps fails and lots of other system tools don’t like it). If I had spent more time thinking, I would have realized that I could installworld to each jail again and things would be ok. Note to self: set up jails with an easily upgradeable world, as described in lots of best-practices documents on jails.

So I upgraded the host onwards to FreeBSD 8.0. Another long long compile, with no GNU screen to make it easier to deal with. Thank goodness for the ILOM and the system console redirection it provides.

Of course, then I went on to make delete-old-libs, which meant that the ports on the system — all of which were compiled against the 6.2 libraries — didn’t work anymore. Note to self: see that little note “in case no 3rd party program uses them anymore”? Keep it in mind next time.

So, after about two days, I had a base system updated to 8.0, no working jails at all, and all ports — both in the host and in the jails — broken. At this point, I started doing two things in parallel. Note to self: don’t. I started rebuilding the ports in the host system, and reconfiguring the jails to have a single base installation with just /home, /etc, /var and /usr/local local to each jail, using nullfs mounts; I also decided to drop the starting of jails in /etc/rc.local and to use the jail-launching support that is now built in (but which wasn’t, as far as I know, available in 6.0 which is when I first configured the machine). Note to self: that was actually a good idea, and thanks also to Sjors who reminded me of the jail_* variables.

So, rebuilding ports after a big step like that is complicated by the fact that perl, ruby, php and python all needed to be recompiled and portupgrade -apP sometimes doesn’t quite get it right. In any case I needed to rebuild the ruby stack first to get a working portupgrade. The other three languages were a mess, with some modules of the languages disappearing at inopportune points along the upgrade path. Basically I did portupgrade -apP ; pkgdb -F ; portinstall <something missing> an awful lot until things were working again. This morning I finally got rid of the last missing PHP 5.3 modules which brought the EBN parts back to life. Note to self: read UPDATING twice before doing this again.

Of course, all that would have been less problematic if the disk array hadn’t given out twice during the whole operation. Once the ridiculously heavy load on the machine caused a panic and once the power on one of the disks fluctuated enough to cause another panic. Running fsck on a 600GB filesystem with 14M inodes is not quick (especially if there’s a few directories with 1M files in each, as is the case with KDE SVN mirrors). Note to self: badger more people about a better disk array for KDE.

Combine all that with sickness and family time and that’s why it took a week. I’m blogging this for the notes to self for the next time I run an upgrade (resolution: when FreeBSD 8.1 comes out) and to notify folks that things should be back to normal. (If not, drop me a note in comments). One the positive side, the server is better organized now, disk usage is down a little bit, and future upgrades should be much easier.

It took some wrestling, but the EBN machine is back up, with some of the VMs up and running FreeBSD 8-STABLE. I’m still repairing the EBN itself, since that has many many more packages installed that need recompiling than the other VMs. There’s also Java to deal with — it may have become superfluous on the machine, but I need to sort that out or build by hand again (due to license restrictions, you still need to fetch Java distribution sources manually).

In any case, with the server up and running again the KDE4-OpenSolaris folks could push their updates to our Mercurial Repository with specfiles (and patches) for the KDE SC 4.4 series, including dependencies. As always, this is intended to build all of the KDE packages on a recent OpenSolaris (I use 130, others use 134, and I would recommend against anything pre-124). Mark and Ben also work on supporting Solaris 10 (the stable, enterprise OS release) so you can probably build the whole thing on there as well. Both SPARC and amd64 are supported, and we do try to support 64-bit builds for all of the dependencies. KDE itself builds only in 32 bit mode, though (simply because I don’t see much point in doubling build times and having an extra copy around).

Let’s take a look at what has happened recently in the repository:

• Updated to KDE SC 4.4.3: after it was released on May 5th, 2010, it took only a day to bump the repo to 4.4.3 as well. We now tag things in the Mercurial repo a little more consistently, so you can pick releases as needed — I do think this is a little more convenient than cloning a separate repo for each minor release as we did for the 4.3 series.
• GPG Updated: updated gpg to 1.3.0, which is the latest version. This also meant updating libassuan to 2.0.0 and some other minor tweaks, but it gives us something modern again.
• icu4c updated: The Unicode consortium’s library for Unicode manipulation — needed by Boost — has been bumped to the latest version. There’s a “competing” version from Sun itself, SUNWicu4c, but that’s built against the old Cstd library so we can’t use it.
• Various administrivia: updated libattica to 0.1.3 and changed the QImageBlitz library version to 0.0.4; this was 4.2.0, matching the KDE release that it originally came with, but the library itself thinks that it is 0.0.4. Looks a bit unmaintained otherwise, and I had to go (thankfully!) to the Debian source mirrors to get a sensible source tarball.
• GNU Telephony: The dependencies for the GNU Telephony project have been imported as well. I don’t know if this is needed on OpenSolaris — it hasn’t been built for any of my KDE needs yet.

So there’s plenty of updates going on, and we’re happy to have a single source where you can type “make” and get a working KDE4 desktop on an OpenSolaris machine. Packages might be available by now, too. I’m still working on transferring the package server to an OpenSolaris VirtualBox on FreeBSD.

By tracking things more closely each minor release (of anything) becomes easier to test and integrate — I suppose I should write something about using ZFS to do that — and we can stick closer to the actual release dates. I don’t think we’ll ever be able to keep up with the Linux distro’s or FreeBSD, but we’re close.

There’s a few things I learned today: One is that FreeBSD with UFS2 is a little slow when dealing with directories with over a million files in them. The KDE SVN — created way back in the SVN 1.4 or earlier days — is set up like that, with one flat directory structure. As a consequence, copying a SVN repo mirror from one place on the disk to another is rather slow. Moving it (within the same filesystem) would be a lot faster, but I wanted a copy. Second is that the EBN machine has grown SVN mirrors and experiments and KDE checkouts (of the whole thing) like mushrooms after rain. I’ll have to clean some of that up, not so much for the diskspace, but for tidyness. Third is that while copying three distinct million-file trees in parallel, your disk array will have a power hiccup, panicking the machine and leading to another two days of fsck. So more waiting for the EBN to come back — particularly annoying since I had the other virtual machines on the system back up and running, so that Sebas had his website back, the KDE4-Solaris packages were available again, and Claudia could share documents with the rest of the board.

Fourth is that Mystic Kriek is really quite tasty, in a pink-and-foamy-cherry-coke-with-alcohol kind of way.

Speaking of pink, I got word that my talk for Akademy has been accepted, with the condition that I must bring my pink whip. Paul Adams has nothing to do with that, I’m sure. However, I need to point out that I got a new whip in Kano last month, made of rolled up goat hide. White, plumed, a little bit more floppy than the nylon-core things we’re used to at Akademy. We’ll see how that turns out as a speaker motivational tool.

Overconfidence goeth before a fall, they say. The software upgrades on the EBN machine are taking a good deal longer than intended. Part of this is some unexpected trips I had to make, but mostly it’s just that 6.2-6.4-7.4-8.0 is a lot of buildworlds (which take surprisingly long!) and reboots, followed by the realization that the jails need upgrading too to work at all (although the ports may continue to work, the base system doesn’t).

All in all it’s just taking a lot longer than intended, but it’s coming back bit-by-bit, rest assured.

Also, I should say that Sun’s ILOM really rocks for remote management, since I needed a great deal of console access to get this done, and I don’t feel like sitting in a cold and drafty datacentre to do so — not with this ongoing hacking cough and headache, no sirree.

[[ OSOL-only entry here, with interest for KDE only from the perspective of “it’s a package server that will be serving up the KDE packages for OSOL”. ]] There’s some bits and pieces involved in setting up an OSOL appliance to be actually useful for serving up packages, so I thought I would document them here.

Adding a user and configuring ssh access: The adduser command in OSOL is a little arcane compared to the interactive versions available on FreeBSD and GNU/Linux, so may as well: useradd -s `which bash` -m username and remember to set up ssh access for that user. This user is also going to have the pkg repository available.

Making sure that the user can ssh login: On my OSOL appliance, /dev/ptmx is missing (which is a known problem, see this forum thread with the always-helpful Alan Coopersmith on it), so I followed the instructions there.

Setting up static IP: The appliance is configured to use DHCP from the local network. For my later use I need a static IP, so I picked 10.0.0.26 and solaris.bionicmutton.org as hostnames (there’s already a host with that name, elsewhere, which is the current FreeBSD-based package server for KDE’s OSOL packages). I added the hostname and IP in /etc/hosts, checked that /etc/nsswitch.conf was using files and dns, edited /etc/nwam/llp to set the interface to static IP (as described here), and added an /etc/hostname.e1000g0 matching one of the hostnames I had just set up. One reboot later (presumably svcadm restart would do, but it didn’t work for me in my haste), things were working and I could ssh in. While I was at it, I also modified /etc/nodename to use the new name of the machine (see, for instance, this thread on changing the hostname).

Setting up pkg.depotd space: As root, I ran pkg.depotd once to set up the repository space before setting it up to auto-start. /usr/lib/pkg.depotd -d /export/home/pkg -p 10000 --set-property publisher.prefix=solaris.bionicmutton.org starts the server. I checked that it was working at all by visiting 10.0.0.26:10000 from the host machine and then I stopped the server again.

Further configuration of pkg.depotd: Next, I use the Service Management Facility (SMF) to configure the package server further. svccfg -s application/pkg/server starts a command-line for configuration purposes. I set the port and root directory for the pkg.depot server here:
setprop pkg/port = 10000
setprop pkg/inst_root = /export/home/pkg
After that bit of configuration was done, I used svcadm refresh svc:/application/pkg/server ; svcadm restart svc:/application/pkg/server to start up the package server by hand, and checked once again that it was actually running. Then I rebooted, and checked that it was running still, just to be sure it will come back under a reboot of the host.

Minor remaining configuration issues: I switched off atime on the filesystem hosting the package respository, so that those are not updated on every package view (zfs set atime=off rpool/export, which reminds me I could have gone for a finer-grained setup by creating a filesystem for the packages first). I also set the read-only flag so that people can’t just publish into this repository — the KDE4-OpenSolaris people use tarballs + ssh for that. I also modified some settings in /export/home/pkg/cfg_cache to reflect the purpose of the repository better.

The OSOL appliance released by Jignesh Shah is 2009.6 — nearly a year old now. Since then the packaging tools for OpenSolaris have made great strides. IPS remains .. interesting. I’m not entirely convinced that it was worth the effort of not using some of the existing package formats. However, by now I understand the API it brings in as well as the various uses of the tool, (and it crashes no more) so I’m starting to like it.

Anyway, the point of this particular appliance is to host KDE4 packages for OpenSolaris in an IPS repo. To do that means running the IPS pkg.depotd. I have a port of an older version of pkg(5) to FreeBSD, but it all seems a little klunky and hard to maintain. So using an OpenSolaris VM seems the right approach.

Except that the OSOL VM from Jignesh is kind of old, isn’t it. So I ended up looking into upgrading the OSOL VM after installation, and ended up with the following steps:

1. Set the publisher to the dev branch: pkg set-publisher -O http://pkg.opensolaris.org/dev/ opensolaris.org . This means that future packages will come from the dev branch, not the released branch.
2. pkg refresh --full to fetch all the new package references.
3. pkg image-update -f (I have to force it, because something’s wrong with cherrypy and the image will not update otherwise) and wait a while while 138MB of package updates are downloaded.
4. Reboot into the new boot environment (VOSApp-1). This will fall into a root shell for fixit-purposes, because the filesystems are messed up.
5. Fix mountpoints for those messed-up filesystems and reboot again; I recall I needed to re-set a number of zfs mountpoints to get the correct organization for regular system operation. Bear in mind this isn’t needed with normal OSOL upgrades, just for this slightly peculiar application setup. I think I had to set the new rpool/ROOT/VOSAPP-1/opt/ (and var and root) to mount in the right places and move the older rpool/ROOT/VOSApp/ filesystems to somewhere in /mnt.
6. Reboot, possibly fix up remaining mountpoints, then zfs destroy the old ones.

There’s a useful HOWTO entry on updating OSOL to a specific build which I might otherwise have used, but the appliance image does not have an entire package installed which could be used to drive the upgrade. Hence this slightly klunkier approach.

Since Jignesh has left Sun, I’m not sure if there will be any future growth in these appliance builds. They were popular enough in Nigeria, where I distributed verbatim copies of the original (probably still in violation of the license terms though) to people interested in running OSOL at all.