Bobulate

The Sun Ray thin client is a nifty piece of hardware, especially when it’s painted with a KDE Oxygen logo. It’s a thin client with USB forwarding and sound and session management and smartcard support (but it does not support the FSFE’s Fellowship smartcard). You can put the roughly the same thing together with Free Software, starting with LTSP, but I’ve been a fan of this largely-hardware solution for years. Unfortunately, it has been a poor platform for KDE4 — I’ve blogged about this in the past.

Until now.

There’s new Sun Ray server 5 coming up, and one of the features it has is Xrender (finally!). You need to enable it after installation and there are warnings about performance, but with a small deployment things should be just fine. I deployed it on my desk, with a single DTU with a single 1280×1024 monitor attached, running on an amd64 X2 with 6GB of RAM. Crunchy, but then again I was also building KDEtoys and KDEpim in parallel on it alongside a local GNOME session and a KDE 4.3.1 session on the Sun Ray.

Wait, 4.3.1? Wasn’t it just yesterday that I posted about the availability of KDE 4.3.0 on OpenSolaris? Yes, true. Bumping KDE up one minor revision did not take all that long, although I had to bump Akonadi to 1.2.0 in the process.

We use the gstreamer backend for Phonon on OpenSolaris, because gstreamer is installed anyway — or maybe it’s the Sun native audio backend? I don’t know and I don’t even know how to check. Suffice to say that whatever the default build picked out, sound works. Now, that might not be all that impressive at first thought, but that means that the thin client is receiving whatever audio output KDE is generating. I have not tried Amarok yet (that port needs a little more work still) on OpenSolaris, but that offers new realms of network-transparent desktop use.

The performance work done in Plasma over the past release cycle(s) has been impressive, and it just feels much snappier; Plasmoid handles show up like they should. This is a world of difference with my old laptop and KDE 4.1 which I was not-exactly-showing-off in London last year. Kudos to the Plasma team (and I’m truly looking forward to the results of Tokamak 3). The screenshot here is KDE 4.3.1 running on a Sun Ray, with whatever theme and decoration I ended up with while fiddling around (although Fluffy Bunny is listed in the “Get themes” widget, it doesn’t seem to work). There is a rendering issue with the text in systemsettings, where white-on-white isn’t all that useful, but on the whole it seems to work well enough graphically. A quick test using qgears gets 20fps on the Sun Ray, versus 30fps on the local display of the same machine (Radeon X1200).

There are some functional bugs, though. Many applications do not start up from the K menu, although they do start from the command line (and they do start up if I log in on a local display instead of the Sun Ray). Konqueror takes forever and a day to start or respond to keypresses — this is not production ready, but it is debug-ready, for anyone with a DTrace hammer. There’s a good chance, actually, that the issues are all related to threading problems deep within the C++ libraries. I know there are some newer patches available for it, and I think the library has been fully integrated into Solaris Nevada as of this week (presumably with those patches), which will make our own packaging of those C++ libraries superfluous. Hunting these issues down and integrating some feature work for OpenSolaris will be the KDE4-OpenSolaris team’s focus for the coming release cycle. It’s coming closer.

This week I’ve spent a few late nights working on the OpenSolaris packaging of KDE 4.3.0 — since 4.3.1 is out, it’s just about time, don’t you think? FreeBSD has the good stuff already.

New development repositories: we have been using Mercurial for a while for maintaining the specfile repositories that drive the packaging of KDE4 on OpenSolaris, which — like any DVCS — makes branching and merging and opening up development a breeze. That also means some unrestricted growth in the number of repositories and sometimes feature creep on individual repo’s, so we’ve stopped for a moment to consider what’s what. There are now five repositories, of which one is legacy, two deprecated for being too confused (but kept for a bit just in case someone is using them), and two repositories that are in current use.

The KDE 4.3.0 repository has specfiles for KDE 4.3.0 (this should not be a surprise) and the dependencies that we have built. Those include Qt 4.5.2, BOOST, Apache stdcxx and lots more. This repository is considered stable and should not change any more. The 4.3.1 repo is not stable, and is under construction.

Things that will go into the 4.3.1 repo are an update of poppler to 0.10.7 (current version) and of course updating KDE to 4.3.1; this also means adding the new KDEpim-runtime module to the set of packages. We don’t package it for 4.3.0 and that doesn’t seem to hurt KMail. We hope to improve collaboration with Belenix by stealing all of their updates to the dependency tree, while we’re at it.

[[ And there was a Qt security announcement on September 1st — that one is in both repositories and applies to the packages that are available as well. ]]

New IPS package depot: in the past three months or so we have accumulated a crufty set of packages on various IPS servers; these have been knocked down and a fresh set of IPS packages is available from the publisher http://pkg.bionicmutton.org:10000/ (also supports port 8080, by popular request).

These packages are the ones built from the 4.3.0 repository using Sun Studio Express on OpenSolaris 2009.6 and are supposed to “just work.” As usual, people with the initials MS will find out that they don’t — honestly Michael, what do you do to those poor install images.

Tools usage: the KDE4 packages are still not built using Source Juicer. This is easy enough to explain: because it’s a family of 122 packages which share settings and common files, it’s difficult to merge into Source Juicer effectively. We have tried, and SJ does drive us to do some simplification and a reduction in the number of files used for the build, but we are not there yet.

Instead, packages are built on a stock amd64 (an X2 4850e w/ 6GB RAM under my desk) OpenSolaris 2009.6 install, then tarballed using the tools from the IPS best practices wiki, archivepkgs.py in particular. These are copied to the actual server which then needs to be restarted every time we import new packages — wanting to avoid the restart is what led to our wild growth of IPS servers.

One tool I’m missing (lazyweb wishlist warning) is one that cleans up the on-disk pkg depot; I’d like to be able to strip out all but the most recent version of every package from a depot. I have an inkling of how this would work (determine what’s “latest”, flag all the files it uses; later walk the filesystem tree and remove all non-flagged files), but I don’t feel like going the destructive route myself right now.

There’s a problem in pkg(1) with large file uploads, so publishing the packages directly to the server is infeasible; that’s something that we need to fix with a nice workflow and one appointed person to kick the depot server when needed. This will happen once we have KDE 4.3.1 packages built and somewhat tested.

Stability: There are some known crashes; Dolphin has some threading issues and KMail crashes on exit, for instance, but the overall impression I get from running KDE 4.3.0 as my desktop on OSOL is pretty good. Even with an underpowered X1200 onboard video. With a proper video card (e.g. the GF9600 512MB on my laptop) things are much nicer.

Feedback: Please use the kde-discuss forum/mailing list or file bugs in the regular KDE bug tracking system and set the Operating System to Solaris in your report. We appreciate bug reports (if you can, check that they are Solaris-specific by comparing with, say, FreeBSD behaviour) and appreciate patches even more. Remember, the specfiles are open and everything can be built from source by following the instructions on the KDE TechBase page on OpenSolaris (and that’s a wiki, so if the instructions are not clear, you can help improve them!).

Time for a collection of screenshots, as an illustration of Qt applications on OpenSolaris, both on a local display driven by a Radeon X1200 and on a Sun Ray thin client. Not from KDE applications (although we have KDE 4.3.0 packages for OpenSolaris now) but from qtconfig — possibly the first Qt app you will want to run in OpenSolaris to set up some of the fonts correctly. Before running this version of qtconfig, I removed ~/.config — the whole directory tree — so I would get the default settings. There are screenies of the same 300×100 section of the application on four setups: local display or Sun Ray thin client, and system fontconfig or one built from our own packages. I switched my set of package builds to use the system’s fontconfig a while back, but the specfile for fontconfig (useful if you care about Solaris10) is still there. Both are version 2.5.0; for freetype system is 2.3.7 and the specfiles build 2.3.6.

So you would expect very little difference between the two, possibly only a difference in the default fonts. You could use some kind of png diff tool to see the differences, but it’s not a whole heck of a lot.

A bigger change happens when you switch from a local display (even a lousy onboard X1200) to a Sun Ray thin client display. I know I’ve ragged on parts of Qt in the past and the way KDE deals with less-capable displays, so this time I’m not going anywhere near graphics views.

What strikes me immediately is that the style has changed, from one display to another; it now looks a good deal more Morif like. Eww! So what is Qt looking at here to change style like that? (That’s a lazyweb question).

And switching fontconfigs doesn’t do much again. I’m actually curious if either of the fontconfigs does anti-aliasing right now.

One thing you can’t see in the screenie is the window titlebar. Whatever the default GNOME window manager is — metacity? — produces really ugly title bars on a Sun Ray, as it looks like the AA falls over, producing hard-to-read letters with white-ish strips between them. Those are screenies for another day.

Flashing: I spent an intensely annoying morning flashing firmwares and BIOSes. I had purchased an MSI GX620 — a “gamer” laptop, but pretty nice from a developer point of view, too — and noticed that the keyboard would “stick” every now and then, repeating the last key typed ad inifinitum. Power button and trackpad would also cease responding. Either pull the battery and AC — hard power off — or plug in a USB keyboard and mouse and go for a regular shutdown, but then the machine would hang with power and fans still going after the OS says its final goodbyes. It is both reassuring and frightening that “MSI GX620 keyboard problem” turns up plenty of resources about exactly this. And a suggested solution is to flash the firmware and then the BIOS.

Easier said than done, of course, from a dual-boot nonGNU/Solaris and GNU/Linux machine. Heck, even with the Vista that was on the machine when it shipped I’m not sure you can flash. The MSI instructions all assume a DOS environment. I tried the boot disks you can create with Windows XP (which boots to “Windows Millennium”?) but no dice, abnormal program termination. I went looking in my box-of-crap in the attic and found my 3-disk set of MS-DOS 6.2 install floppies, but those no longer seem to work. Heck, it’s a wonder I even have a floppy drive to boot from — I collected a few USB floppy drives a few years back.

This kind of situation makes me long for my usual ASUS motherboards which support flashing from the BIOS itself, and can read CDs.

In any case, I needed to be able to boot an OS I don’t actually have in order to update the machine. Fortunately Daniel documented USB boot disks here, and that finally gave me enough to craft my own simple 10-step procedure: 1. boot ancient windows XP laptop strill used by the kids for Sesame Street games 2. install HP USB disk tool 3. boot modern Linux machine 3a. find USB floppy drive 4. download FreeDOS “ODIN” image 5. try to write the image to a floppy disk 6. download a smaller image, until you get one that fits before the first bad sector on the floppy (for me, fortunately, that was the 720k image) 7. move floppy to windows machine 8. format USB stick with HP tool, using system files from ODIN 9. boot defective laptop from USB stick 10. revel in C:\>

I suppose I could have gone via VirtualBox on Solaris or qemu, but I had neither installed already, and frustration seems to rapidly erode my capacity for lucid thought. In any case, I’ve now gone the day without the keyboard locking up, so the flash updates seem to have helped.

Photos: At Akademy I always try to have some plan on capturing the spirit of the event. In past years I’ve intended to do voice recordings and always forgotten, so this time I did something much more low key: just take pictures of some whiteboards left behind in the conference rooms. This little collage of strings found on boards does seem to capture the working week, though. What are we doing? Even though we’re die-hard computer technologists, we still end up smearing ink on light-coloured surfaces.

[[ You’d be surprised, incidentally, about how many of the photos in my phone are of whiteboards at the end of meetings. Best way to remember things. ]]

Finishing: Congratulations, Gökmen and Görkcem on Pardus 2009. It looks very nice. And I really appreciate a distro that comes with TeX.

KDE 4.3 is also looming terrifyingly close, which means that there’s a fistful of OpenSolaris patches we haven’t gotten around to pushing upstream yet. Some are really peculiar, and I appreciate it when people ask me about things like bool() casts — there was one bug report about operator ?: with a bool and a QBool, which seems very peculiar to me. Interesting how QBool doesn’t have an operator bool(), only an operator void *() — I think SS12 is being extra picky here.

Finally, one of the anonsvn mirrors was down again for a bit; this was due to r.997199 which has a badly-encoded log message. Since we have a wild mix of Subversion 1.4, 1.5 and 1.6 infrastructure — and also users with clients of differing versions as well — we do not get a “clean” history. In this case, subversion 1.6 is more picky about the encoding of log messages than older versions. Since I was running 1.6 on the EBN mirror, it refused to synchronize that particular commit and went down. I have since downgraded to 1.5.2 and the mirror has resumed operation, but it’s an illustration that our source history is “interesting” to say the least and the conversion to any other SCM will once again be rife with manual labour to get it all right.

We have the technology, we can fix it. The nice thing about showing people things that are broken and then sitting down to figure out what’s wrong is that the things get fixed. Holger helped out with understanding some of the rendering issues on the Sun Ray server, we tracked down some more suitable default themes (but there’s so many cool themes out there it’s hard to find a boring flat one), pushed graphicssystem RASTER into startkde, and ended up with a KDE4 that looks like this. Thin client heaven? I think not, but it’s a little better than when it started. Just a wee bit, though, even something totally flat like Heron (wish list: semantic tagging of plasma themes so I can search for “flat and boring”) totally borks the plasma panel.

Aaron was, probably rightly so, kind of annoyed with my cheap shot about ugliness on feature-limited X servers.

Still, it’s important to flag the problems, show them to people. The cheapness was more related to me complaining to the wrong people, but now at GCDS I could show them to the Sun Ray folks — and I’m very happy to have met Bob, Brian, Ken (?) and others from Sun — and collect information. It looks mostly like Qt has two problems: the fallback for non-porter-duff compositing is not very nice, and BGR colormaps are not supported. That’s the guesses from the Sun folk and they sound plausible enough — I’ll be trying them out when I get back home and can spend an evening re-compiling Qt all the time (instead of like here at GCDS where the evenings are for discussion and food).

.. and then we can fix it.

I wrote my previous bit on KDE4 OpenSolaris packages before all of them were done; I scheduled the post and left the packages building and went off to teach classes (networking and SSL technologies for high school kids). What’s the worst that could happen?

Well, for one thing a dependency on a non-released version of Soprano could show up — one that isn’t tested for in the cmake code either. I can understand this to some extent, after all there is a great deal of co-development going on between soprano and its clients. Fortunately, soprano has been tarballed since then and added to the KDE source download mirrors. I see on the KDE release management mailing list that there’s a couple more gotchas, but those will only show up once I get past KDEbase and pim.

The life-size stopper for KDE PIM on OpenSolaris right now is the behavior of nepomuk-rcgen, which just segfaults on startup. That effectively prevents KDE PIM from building, since it uses the latest nepomuk cmake magic. On the other hand, things like konsole do work.

One commenter was asking about sourceJuicer and p5i files. The p5i files are a new thing in pkg since OSOL 2009.6 — I don’t know if pkgtool sets them up correctly, or even who is responsible for making these things or where it is supposed to get the publisher information from anyway. Any hints on what should go in the specfiles to produce useful p5i files would be appreciated. As for jucr, these packages aren’t yet in any shape to push there — and there are various problems related to build-time dependencies as yet unresolved. For instance, you can’t depend on cmake as a build tool on the jucr build machines. Some of the KDE4 dependencies have been pushed there, but they are not building regularly. It’s easier to do so outside of jucr, where updates can be done faster (and of course, the spec file repo if you want to build everything yourself is on bionicmutton, as documented on techbase).

PS. Konqueror + certain proprietary plugins needed for popular culture (viz. YouTube) works, too. I’m impressed (by the Konqueror developers).

PPS. You can now install KDEgdm-integration normally with “pkg install”. Note that the fonts for Qt applications are butt-ugly; run Qt config and set up nicer ones, then run systemsettings to fix up KDE’s appearance. At some point I hope to figure out how to set better defaults for both on package install.

The KDE4 IPS packages that I’ve been working on have been rolled out at bionicmutton so if you are a brave soul, you could install them by adding that as a package publisher (pkg.bionicmutton.org points to the same). Something like:
pfexec pkg set-authority -O http://solaris.bionicmutton.org:10000/ kdeips
pfexec pkg install KDEgdm-integration
This will get you an /opt/foss and a /opt/kde-4.2 (the latter actually contains KDE 4.3-beta2). And you should be able to run bits and pieces from there, or even log in to KDE from the regular OpenSolaris login manager.

Getting the packages out there did have some issues. These are entirely networking related: I build them at home on one of my OSOL machines, then push them out into the IPS package server, which is running on a FreeBSD machine at the university. Since my DSL upload speed is low (about 50% faster than Jon) uploads of large files time out regularly. Pushing Qt or Boost is pretty much a no-go, as it will fall over after, say, 90MB or more. I suppose I could get on my bicycle and carry my laptop to a place where I could use the direct wired network to the server, but that would mean going outside.

Instead, I found — relatively well-hidden, which I find typical of Sun documentation — a wiki page that provides a nice-to-read HOWTO package which in turn points to a download page where you can get tools. The universal toolkit image is sufficient, and it contains pkg-toolkit/pkg/bin/archivepkgs.py, which will extract a package from a local repository and dump it to a tarball. In essence it defines an on-disk format for moving packages around, and you can move the tarball to another IPS repo, unpack it, and it can be served from there.

This means that I can build, publish to a repository on localhost (no network timeouts!), dump to a tarball, scp to the actual server (scp is a lot more robust in that sense than python scripts doing web-services over slow links), unpack, and there it is. It’s still a 40 minute upload of Qt, but knowing that it will succeed is important.

Thanks to Kohsuke Kawaguchi for writing the original tarpkgs.py script and documenting it on his blog.

Right. So here’s a foolish plan: I want to publish some OpenSolaris packages to the world. They are not ready for SourceJuicer (submissions there are slightly complicated, and the forest of KDE packages I’m producing is tedious and time-consuming to import there). Last time I tried that publishing thing, I did it on a package server on an OSOL machine at the end of my own DSL line, which pretty much immediately spanked my bandwidth completely. So I want to do it on a machine with better connections.

It may surprise some that all the real servers I have run FreeBSD. I use OSOL as my main desktop operating system. There must be historical reasons for this. In any case, that means building the OpenSolaris IPS package server pkg.depotd on FreeBSD. It turns out to be relatively straightforward, as most of the actual server is written in Python (yay!) and there’s only a few bits of C in there. So my approach was:

• Fetch the sources for pkg as documented on the pkg project page (thank you Alan Coopersmith),
• Install necessary development tools on my FreeBSD machine:
  portinstall py-openssl py-cherrypy intltool gnome-doc
portinstall py25-mako py25-simplejson
• Run gmake. At this point I realised that the makefile has some bugs leading to No rule to make target `help/C/package-manager.xml' — there is some semi-complicated implicit target stuff going on in there. I built the needed files by hand with
  cd gui
msgfmt -o help/C/C.mo help/C/C.po
xml2po -t help/C/C.mo -o help/C/package-manager.xml help/C/package-manager.xml.in
  and stripped out the locales that I didn’t need.
• Build extract_hostid by hand with
  cd util/misc
cc extract_hostid.c -o extract_hostid -L/usr/local/lib -R/usr/local/lib -lintl
  This is needed because gettext lives in /usr/local on FreeBSD and the Makefile (obviously) assumes the OSOL setup.
• Fix the top-level makefile to use python from /usr/local/bin and run gmake again.
• Patch setup.py to support FreeBSD — there looks to be support for sunos, linux, windows and darwin already (!?). This takes a patch at lines 102 and 194. Run gmake again.
• Run gmake install
• Finally,
  cd gate/proto/root_freebsd_amd64
PYTHONPATH=`pwd`/usr/lib/python2.4/vendor-packages:
sh usr/lib/pkg.depotd -d ~/pkg/ips -p 10000

And there you have it. FreeBSD machines serving up OpenSolaris packages. For the purposes of KDE4 on Solaris, this means that pkg.bionicmutton will soon be returning and you will be able to install IPS packages instead of having to compile everything from source yourself. YMMV, contains rabid weasels, etc.

[[ And, as an addendum, let me congratulate the pkg / IPS team on the enormous improvement in the packaging system between OSOL 2008.11 and 2009.6; it used to be “ugh” and has reached, for me at least, a level of “hey, that’s pretty neat.” ]]

I mentioned that I had one machine that wouldn’t run OSOL 2009.6, so I did a little hardware debugging on it. Step 1 is, of course, removing all the bits that aren’t essential. So I stripped the machine down to two sticks of RAM on the motherboard and a CD drive, and lo! It booted and behaved normally. Step 2 was adding back RAM until it was full again (at 6GB). The reason to strip RAM at all is that I’ve had bad sticks in the past and some OSsen are picky, so I figured stripping down was possibly useful. And, having repopulated the RAM banks, things were still OK. So it must have been one of the two extra NICs — an RTL8100 and an RTL8169 — in the PCI slots. Left them out for now, since the on-board NIC was now supported. Chalk up another problem worked-around and not really fixed.

I have a stupid early train to Germany tomorrow, so just a few short notes on KDE 4.3-beta on OSOL: all graphicsview things seem broken. Using the raster renderer doesn’t help. Icons are not found — at least not the oxygen ones that I thought would come with kdebase-runtime, but I can use the Tango ones installed on the system. Plasma crashes. Even konsole crashes a while after plasma and kwin do. KMail is busy pulling in 500MB of disconnected IMAP, nothing bad happened there yet.

While pstack(1) would have been an obvious tool to apply to any of the crashes, that’s something to look at some other day. For now, it’s enough to have finished round one of compiling. [[ At GCDS there will be lots more time for polishing, for sure. ]]

As I mentioned previously, I’ve updated some of my machines to OSOL 2009.6. Only some, though. It refuses to start on my AMD 760G based machine — gets through grub, shows the splash and then hangs. I haven’t sat down to debug that one. It does run quite nicely on my new laptop (some folks asked: it’s an MSI GX620, which is nominally a “gamer’s laptop”. Poor battery life, but I realised that I don’t use the lappy in planes and on the longer train trips I have there’s power. GF9600, P8600, 4GB, 320GB — it’s slightly more powerful than my desktop machines. The keyboard is OK, takes a little getting used to because some of the punctuation is slightly smaller than usual. It has a numeric keypad, which as far as I’m concerned could have been left out for some bigger keys. The machine is a little noisier than I might have wanted, too.). There’s also a really nice VirtualBox image for OSOL 2009.6. That one only gets you a text login, though.

On the KDE4 front on OSOL, we hit some issues similar to what happens on Windows (and to a lesser degree on FreeBSD). I’ll quote Christian Ehrlicher from his recent “stopping Windows development” blog entry:

Another problem I’ve is that I could not fix bugs in kdelibs just because the dependencies are moving fast and since we have to take care for all system libs (png, xml, openssl, pcre, …). Making sure that they’re up-to-date can take a lot of time. And when I finally managed to compile a KDE program I hit compiler errors. This is all fixable but not when you’ve only a little amount of time for KDE development.

Pavel has been hacking up a veritable storm in updating our KDE builds to 4.3-beta, and I’ve been following along trying to update dependencies (gpg and friends, lzo, akonadi, …) as well. And then we hit new dependencies (what is openslp?) all of a sudden, which means backtracking and packaging some other bit of Free Software first.

All this hacking happens in the publicly writable kde4-specs-42 Mercurial repository on solaris.bionicmutton.org. It’s publicly writable so that anyone can contribute, but that also means that it sometimes gets screwed up. By me, for instance, because I pushed something last week that broke all 64-bit builds in it. Gah. Anyway, that’s cleaned up now, and the current status is: KDE 4.3-beta builds, runs ok (oh joy of konsole compared to GNOME-terminal). In VirtualBox there are rendering issues. Those will go away in time, I imagine, or once someone fiddles around with default themes and such.

My intention is to release a OSOL 2009.6 KDE 4.3-beta VirtualBox appliance as soon as I have something that works “well enough.” That means that you need to be able to log in from a display manager (gdm or kdm, I don’t care — I haven’t gotten either of them to start up properly on the appliance yet) and get at least the KDE Plasma Desktop functionality, with Konqueror and Konsole as applications. That would make me happy, as a new milestone in the ever-shifting race to keep up with KDE development.