Bobulate
Home [ade] cookies
Installing the EBN on OpenSUSE 11.2
June 5th, 2010
The EBN in moving. Not physically, since it’s going to be on the same hardware, but conceptually, from FreeBSD 8-STABLE to OpenSUSE 11.2.
There’s a pretty simple reason why I’m trying to do this on OpenSUSE: KDE’s sysadmin team will support either OpenSUSE or Debian, and I happened to have a SUSE install CD lying around. The end goal is to turn over administration of the EBN machine to KDE’s sysadmin team. The recent stability issues suggest that there’s some subtle problem between FreeBSD 8-STABLE and the hardware. Combine that with my lack of time to actually fix stuff when the panic hits the screen, and you have a bad situation. Hence the move.
Before the move, though, I do need to set up enough of a machine to be able to test that it’s going to work.
I’m a total Linux n00b (in a way: the skills I had in RedHat Linux 4.2 administration no longer apply) so things like installing a database server are once again a challenge for me. On FreeBSD or OpenSolaris I know where the documentation is and which knobs to twiddle. It’s either in /etc/rc.conf or using svcadm(8), and I can deal with both. I’m taking notes while I get the same bits done on OpenSUSE.
The installation notes and prerequisites will end up in KDE SVN where the rest of the EBN code and material is hosted, as part of the instructions on setting up your own instance; I won’t bother blogging about which users to add and how to set up groups. That’s boring.
I had a bit of a rant (along the lines of “OpenSUSE is just terrible“) written up, too, about inconsistencies in the interface, difficulty of finding the applications I need, finicky web searches needed to weed out the useless results for ancient versions of the distro when trying to find out how to set up RAID1 — that kind of stuff. But it’s not terrible. It just takes a few days getting used to, like any tool. Gut-level and immediate responses shouldn’t color my use of a tool that is eminently successful. Heck, I tell my kids the same thing when they need to eat their vegetables. So no rant (except maybe that the Search bars in YaST are wildly inconsistent).
There was some good stuff, too: I liked the installer, and the disk setup tool, once you find the right documentation (er .. where did I put it?), is quite nice to use. So in the end I’ve got a test machine at home booting OpenSUSE off of one disk and storing data for the EBN on a RAID1 pair of 500GB drives; PostgreSQL, Apache, Mercurial, Git, Subversion is all up and running, and next up is getting the regular tools to go.
And then hopefully the EBN will be up and running again once the disks move to the server.
Done in Berlin
May 30th, 2010
We need a couch in the office, for sunday-afternoon naps after long, long meetings of the KDE e.V. board. Since we need some more furniture for the office anyway — shelving for storing schwag, some organizing cabinets and things to keep down clutter and a better monitor stand for Thorsten — we just might end up ordering a cheap one. It might improve efficiency in the long run. Some complicated discussions today about interacting with the KDE community and the membership of KDE e.V., since the board has a formal role in the association but is also composed of KDE contributors. Planned some bits and pieces for Akademy. I finally got around to booking a hotel — far away from Team Humongous, so bedtime stories are going to be an issue. And I was just finishing up an ecological fairytale for sharing, too.
A board meeting wobbles between abstract, long-term thinking (for instance) about financing developer sprints and procedures around those sprints and really concrete things like (for instance) ensuring that we have a stable document storage facility or brainstorming potential KDE SC release codenames. It’s a fun kind of meeting, but it really takes it out of you. I’ll be glad to take an extra nap on the train back home tomorrow. Given that the train leaves Berlin at 6:37am, that shouldn’t be all that difficult. I’d like to thank Claudia and Thorsten for taking time out of their weekend to facilitate the meeting, and applaud Celeste for doing crazy stuff.
EBN up^Wdown for the moment
May 30th, 2010
I wrote this on Saturday afternoon, but didn’t hit “post”: After running fsck repeatedly until it finally stopped finding unreferenced files, I’m hopefully calling the disk array fixed on the EBN. I’m still going to reconfigure the server in some way, but for the time being I’ve restarted the VM for api.kde.org and the EBN. That means that api.kde.org is accessible again and the EBN is running. I’m hoping that the NIC and RAID will hold out. It will take a while for API regeneration to finish as well as a new round of Krazy checks to run.
And I would add this today: the RAID array didn’t survive the night, with new read timeouts on the disk followed by mirror corruption and end of story. So I rebooted, dropped that VM again and the machine is currently running only essential services. We’re in the process of moving things off of the machine now so that it is easier to reinstall, with fewer tasks. Then we’ll hopefully have something usable again.
A Weekend in Berlin
May 29th, 2010
I end up writing so much about OpenSolaris that people sometimes assume I work for Sun or Oracle. Not the case. The KDE4-OpenSolaris project is my development hobby, and isn’t part of any employment at all. I worked for the FSFE until recently, and nowadays I work in the garden while looking for something new and suitable to do. Besides the work and hobby bits, I have my responsibilities as a volunteer board member of both the NLUUG and KDE e.V.
This weekend, I’m in Berlin for a meeting of the board of KDE e.V. A board meeting is partly a team-building exercise, partly an opportunity to synchronize the various projects that KDE e.V. is working on, partly a legal, financial and administrative exercise and lastly (but perhaps most importantly) a chance to plan what we will be doing in the next six months or so. There’s a few things we’ll be talking about, like supporting our version control systems, growing the membership of the association, conference and sprint organization. Many of our activities are mandated or demanded by the community — like keeping our servers up-to-date and useful for software development done by the KDE community as a whole. One small topic might be financing a small spare server to speed up the recovery of api.kde.org (ok, that’s my hobby as well). The financial stability of the organization — for financing sprints and conferences and whatever the development community needs — is another one of those topics that always takes some time.
If you’re in Berlin, feel free to stop by for lunch (around 1pm) or a beer later at the KDE e.V. office in Linienstrasse — drop me a note.
Another round of OSOL builds for KDE4
May 27th, 2010
After a few upgrades on various desktop and laptop machines (don’t get me started about the EBN machine, which is curiously unstable under FreeBSD 8.0) so that my main development workstation is running OSOL build nv134, I’ve kicked off a new round of OSOL builds. Of course, I ran into the 134 upgrade issue where /dev/ptmx ends up with the wrong permissions, but that’s documented along with a workaround in the release notes. The new set of builds is plain KDE SC 4.4.3, with only a few updates: Assuan is updated to 2.0.0. This disables parts of KDE PIM, at least until I get KMail recompiled so that I can get at the patch that Will Stephenson sent me two weeks ago to handle Assuan2. ICU4C is updated to 4.4.1. The FOSShier package makes a new appearance, so you can easily manage KDE SC separately from the dependencies that KDE4 has.
For the bits-that-I-haven’t-tried-myself, I think KDM now partly works on OSOL, but it doesn’t run all of the scripts that GDM does, so you don’t get a very good desktop experience out of it.
On the whole, it’s not really exciting. We (as in the KDE4-OpenSolaris folks) have decided to give the KDE SC 4.5 betas a miss for now — I know that trunk was having some interesting template issues a few weeks again, so I have serious doubts it will compile at all — and just try to get the current crop of packages in the best shape possible for OpenSolaris.
There’s an OpenSolaris 2010.03 upcoming, some day. Yes, that 03 was supposed to be March, but clearly corporate takeovers got in the way. There are some pretty big changes in packaging there (in particular, package naming) so in order to make the KDE4 packages integrate cleanly we’ll have to do some fixing there.
As always, the specfiles can be had from our specfile repo; however, packages are not up-to-date on the package server because of stability issues.
EBN down with hardware problems
May 20th, 2010
As Luca B. and some others have pointed out already, the EBN is down again. Read errors on one disk followed by a panic on one of the filesystems overnight. This means plenty of things offline again, and by now the situation is such that I’m going to have to take drastic measures. That means fetching the disk array and rebuilding most of the machine. In the meantime I will leave the two SAS disks in so that some of the essential services can be restored, but it means that the EBN and api.kde.org (which use up most of the disk space, processor power and bandwidth) will be down for as long as it takes.
Needless to say, anyone with a nice 1U or 2U storage unit they are willing to donate to KDE is really welcome to step up.
Hugin and patents
May 18th, 2010
It’s interesting to see Hugin show up twice in one day: Ken Wimer describes how he produced the UDS group photo with it — he focuses on the use of Hugin and the user interface — while Michael Kesper has made a panorama in Tromsø. But Michael points out an interesting issue with Hugin: a patent that may apply to the panoramic-stitching algorithm. Hugin prints a warning when using that tool, apparently.
This gives me an opportunity to talk about patents a little bit — a refreshing change from OpenSolaris packaging, even if I spend more time on the latter these days.
So, first off, the GPL (version 2, but version 3 has similar language) has an interesting clause about patents that many people forget about. It’s clause 8:
8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License.
So supposing you knew in advance that a patent applied to an algorithm used in your piece of GPLv2-licensed Free Software, you could exclude those areas where the patent applies. In other words, you could have something like “Hugin is Free Software, but may not be used in the United States as a consequence of US patent #N”. A surprising idea, no? That’s one special case where Free Software isn’t necessarily Open Source software (the Open Source definition disallows restrictions by country).
Source code as such (certainly in the case of compiled languages) does not implement an invention — it merely describes an algorithm. As such, I believe that the source code of a Free Software implementation of some patented algorithm (as if it makes sense to patent an idea, but hey, some patent offices hand these things out) cannot infringe the patent even if you accept the validity of algorithmic patents because it’s just describing something that has already been published — the patent text itself! On the other hand, compiled forms of the same do implement the algorithm in a machine and might be covered. I’m not sure if anyone has really dug into the implications of the division between source and object code in this area.
In cases like this, the Open Invention Network might be of use. It’s a patent pool organization for Linux. Since Hugin isn’t part of Linux (as in, the kernel bits) it’s unlikely to be helped out directly. The OIN folks are some of the most pragmatic and sensible people I’ve talked to about the effect (negative) software patents have on us all.
A brief patent search (ah, for people who think this might disqualify me forever from participating in development, I’ll point to Andrew Tridgell’s talk on reading patents — also on OSNews) didn’t turn up anything filed by the University of British Columbia that seems to apply. However, it might have been way too brief a search, as there’s darn little to go on based on the warning message from Hugin. To do it right the warning would have to be far clearer, possibly pointing to the actual patent number. Otherwise, calling the UBC patent licensing office (actually, you need to email Greg Lowe directly) for information is a little difficult.
One article on SIFT (Scale Invariant Feature Transform?) panorama stitching can be found at Springer, although the research page for SIFT from Greg Lowe is more interesting. It at least lists the exact patent number and stuff like that. Frankly I’d rather read the research papers than the patent text — at least the papers have as goal to share knowledge with the research community, as opposed to obfuscating the invention to make it broadly applicable in courts of law. However, one could apply the mechanisms Andrew Tridgell describes to the patent, and develop a stitching algorithm not covered by the patent simply by not doing something from the method described there. For instance (I’m not an image processing guy here) finding difference images in a different way.
Kudos to the patent writer, anyway, for claim 20 “A computer readable medium comprising codes for directing a processor circuit to execute the method of claim 1”, so that the patent covers the method, apparatus for implementing the method and computers doing the same. That’s surely a convolution caused by the way the patent system works (again, Tridgell: if the patent didn’t cover computers explicitly, then you could argue that implementing the method with a computer is not what is claimed, ergo you’re free).
Ugh. Too much time spent trying to understand the whole “this is my idea and you can’t have it” (as opposed to expressions or performances of an idea) culture.
While looking for the SIFT patent, I did find US patent numbers 7,639,897 and 7,711,262 which both cover guiding a user of a digital camera in making a panorama photo. They seem awfully similar to me, although obviously there’s a giant difference (sarcasm doesn’t work in writing unless Penny Arcade does it) between sweeping a scene and then re-photographing it and indicating already-photographed areas as the scene is swept. I guess there’s no patent yet on not helping at all.
EBN, api.kde.org, vizzion.org and others back at last
May 18th, 2010
Well, it took ages, but the EBN and the different VMs it hosts are back. Add “sysadmin” to the list of occupations I probably shouldn’t attempt without (1) more training (2) a stricter schedule. The NLUUG spring conference on systems administration was quite educational — and fun, too, chatting with various companies and learning about NanoBSD and ZFS — but it didn’t give me any magical beans to fix what ailed the EBN.
So what was the problem? Well, the whole thing started (yay, placing the blame!) with Bertjan, who wanted a newer Qt version on the EBN for his software quality checking tools. The EBN ran 6.2-R, and the necessary Qt versions and stuff are not supported on that OS anymore. While the EOL for FreeBSD 6 is still six months away, the ports maintainers don’t necessarily want to support that. So we needed to update the OS to something newer.
There’s tools to do that now, but I’ve never used them, and anyway I don’t think they support FreeBSD 6. So that means lots of “make buildworld buildkernel installkernel installworld” kinds of steps. First off I found that doing the compilations took a lot longer than I expected (or hoped). So where I planned to go 6-6.4-7.3-8.0 in one day, the fact was that just compiling was going to take longer than that. I couldn’t pre compile everything either with the machine still up, because FreeBSD 8 doesn’t compile in a FreeBSD 6 environment. Hence the multiple steps. Note to self: update more frequently to avoid this kind of large upgrade.
Second problem was that the jails (virtual machines) on the server were poorly set up. They all had their own copies of the world. I hadn’t realized that a 6.2 jail wouldn’t work in a 7.3 host (for instance, ps fails and lots of other system tools don’t like it). If I had spent more time thinking, I would have realized that I could installworld to each jail again and things would be ok. Note to self: set up jails with an easily upgradeable world, as described in lots of best-practices documents on jails.
So I upgraded the host onwards to FreeBSD 8.0. Another long long compile, with no GNU screen to make it easier to deal with. Thank goodness for the ILOM and the system console redirection it provides.
Of course, then I went on to make delete-old-libs, which meant that the ports on the system — all of which were compiled against the 6.2 libraries — didn’t work anymore. Note to self: see that little note “in case no 3rd party program uses them anymore”? Keep it in mind next time.
So, after about two days, I had a base system updated to 8.0, no working jails at all, and all ports — both in the host and in the jails — broken. At this point, I started doing two things in parallel. Note to self: don’t. I started rebuilding the ports in the host system, and reconfiguring the jails to have a single base installation with just /home, /etc, /var and /usr/local local to each jail, using nullfs mounts; I also decided to drop the starting of jails in /etc/rc.local and to use the jail-launching support that is now built in (but which wasn’t, as far as I know, available in 6.0 which is when I first configured the machine). Note to self: that was actually a good idea, and thanks also to Sjors who reminded me of the jail_* variables.
So, rebuilding ports after a big step like that is complicated by the fact that perl, ruby, php and python all needed to be recompiled and portupgrade -apP sometimes doesn’t quite get it right. In any case I needed to rebuild the ruby stack first to get a working portupgrade. The other three languages were a mess, with some modules of the languages disappearing at inopportune points along the upgrade path. Basically I did portupgrade -apP ; pkgdb -F ; portinstall <something missing> an awful lot until things were working again. This morning I finally got rid of the last missing PHP 5.3 modules which brought the EBN parts back to life. Note to self: read UPDATING twice before doing this again.
Of course, all that would have been less problematic if the disk array hadn’t given out twice during the whole operation. Once the ridiculously heavy load on the machine caused a panic and once the power on one of the disks fluctuated enough to cause another panic. Running fsck on a 600GB filesystem with 14M inodes is not quick (especially if there’s a few directories with 1M files in each, as is the case with KDE SVN mirrors). Note to self: badger more people about a better disk array for KDE.
Combine all that with sickness and family time and that’s why it took a week. I’m blogging this for the notes to self for the next time I run an upgrade (resolution: when FreeBSD 8.1 comes out) and to notify folks that things should be back to normal. (If not, drop me a note in comments). One the positive side, the server is better organized now, disk usage is down a little bit, and future upgrades should be much easier.
KDE SC 4.4.3 on OpenSolaris
May 17th, 2010
It took some wrestling, but the EBN machine is back up, with some of the VMs up and running FreeBSD 8-STABLE. I’m still repairing the EBN itself, since that has many many more packages installed that need recompiling than the other VMs. There’s also Java to deal with — it may have become superfluous on the machine, but I need to sort that out or build by hand again (due to license restrictions, you still need to fetch Java distribution sources manually).
In any case, with the server up and running again the KDE4-OpenSolaris folks could push their updates to our Mercurial Repository with specfiles (and patches) for the KDE SC 4.4 series, including dependencies. As always, this is intended to build all of the KDE packages on a recent OpenSolaris (I use 130, others use 134, and I would recommend against anything pre-124). Mark and Ben also work on supporting Solaris 10 (the stable, enterprise OS release) so you can probably build the whole thing on there as well. Both SPARC and amd64 are supported, and we do try to support 64-bit builds for all of the dependencies. KDE itself builds only in 32 bit mode, though (simply because I don’t see much point in doubling build times and having an extra copy around).
Let’s take a look at what has happened recently in the repository:
- Updated to KDE SC 4.4.3: after it was released on May 5th, 2010, it took only a day to bump the repo to 4.4.3 as well. We now tag things in the Mercurial repo a little more consistently, so you can pick releases as needed — I do think this is a little more convenient than cloning a separate repo for each minor release as we did for the 4.3 series.
- GPG Updated: updated gpg to 1.3.0, which is the latest version. This also meant updating libassuan to 2.0.0 and some other minor tweaks, but it gives us something modern again.
- icu4c updated: The Unicode consortium’s library for Unicode manipulation — needed by Boost — has been bumped to the latest version. There’s a “competing” version from Sun itself, SUNWicu4c, but that’s built against the old Cstd library so we can’t use it.
- Various administrivia: updated libattica to 0.1.3 and changed the QImageBlitz library version to 0.0.4; this was 4.2.0, matching the KDE release that it originally came with, but the library itself thinks that it is 0.0.4. Looks a bit unmaintained otherwise, and I had to go (thankfully!) to the Debian source mirrors to get a sensible source tarball.
- GNU Telephony: The dependencies for the GNU Telephony project have been imported as well. I don’t know if this is needed on OpenSolaris — it hasn’t been built for any of my KDE needs yet.
So there’s plenty of updates going on, and we’re happy to have a single source where you can type “make” and get a working KDE4 desktop on an OpenSolaris machine. Packages might be available by now, too. I’m still working on transferring the package server to an OpenSolaris VirtualBox on FreeBSD.
By tracking things more closely each minor release (of anything) becomes easier to test and integrate — I suppose I should write something about using ZFS to do that — and we can stick closer to the actual release dates. I don’t think we’ll ever be able to keep up with the Linux distro’s or FreeBSD, but we’re close.
Rains, pours, downtime
May 13th, 2010
There’s a few things I learned today: One is that FreeBSD with UFS2 is a little slow when dealing with directories with over a million files in them. The KDE SVN — created way back in the SVN 1.4 or earlier days — is set up like that, with one flat directory structure. As a consequence, copying a SVN repo mirror from one place on the disk to another is rather slow. Moving it (within the same filesystem) would be a lot faster, but I wanted a copy. Second is that the EBN machine has grown SVN mirrors and experiments and KDE checkouts (of the whole thing) like mushrooms after rain. I’ll have to clean some of that up, not so much for the diskspace, but for tidyness. Third is that while copying three distinct million-file trees in parallel, your disk array will have a power hiccup, panicking the machine and leading to another two days of fsck. So more waiting for the EBN to come back — particularly annoying since I had the other virtual machines on the system back up and running, so that Sebas had his website back, the KDE4-Solaris packages were available again, and Claudia could share documents with the rest of the board.
Fourth is that Mystic Kriek is really quite tasty, in a pink-and-foamy-cherry-coke-with-alcohol kind of way.
Speaking of pink, I got word that my talk for Akademy has been accepted, with the condition that I must bring my pink whip. Paul Adams has nothing to do with that, I’m sure. However, I need to point out that I got a new whip in Kano last month, made of rolled up goat hide. White, plumed, a little bit more floppy than the nylon-core things we’re used to at Akademy. We’ll see how that turns out as a speaker motivational tool.