Bobulate
Home [ade] cookies
New Beginnings
April 21st, 2010
In spring I tend to write sentimental bits, like this one. I’ve spent the past two weeks mostly in the vegetable garden, fostering new life — potatoes, carrots and beans — and tearing out unwanted plants — stinging nettles and crab grass. There must be a metaphor for software development there somewhere. It’s a matter of out with the old and in with the new.
This spring brings renewed determination to do something with the garden; that’s both on my part and on the part of the other people with whom I keep it. I think there’s a realization that the discipline of maintaining the vegetable plots does us (nerd, philosopher, teachers of dutch and mathematics) all good. It gets me away from the computer, anyway, and hoeing is one way of cleansing the spirit.
Related, I am happy to see that Henrik Sandklef has regained his footing: Restarting life sounds pretty drastic. Catching up on the todo list is sometimes good — and sometimes you have to throw out the list and start over. I’m glad FSCONS is still on Henrik’s list, because that was probably the event that made the biggest impression on me last year, for being eclectic and social and technical and drunken and excellent all at once.
So, stay balanced. Create Free Software.
VirtualBox on FreeBSD
April 17th, 2010
For some of the things I would like to do with the EBN code-quality checking machine — things outside the immediate realm of quality checking — I need some VMs beyond what FreeBSD’s jails give me.
In particular, I need Solaris running on the machine as well as FreeBSD, so I looked into VirtualBox. I’ve used it on Solaris for various purposes (including running FreeBSD) so it seemed appropriate. There’s no binary version of VirtualBox for FreeBSD, but you can compile the Open Source Edition from ports, so that’s what I did. The results — missing USB passthrough and no RDP — are things I can live with, as it just means I’ll do most of my work on the VMs through ssh.
Installation (of VirtualBox OSE 3.1.6) is pretty non-eventful, the remember-to-load-the-kernel-driver reminder at the end of installation useful, and it just works.
For testing purposes and to satisfy some curiosity on my part I decided to install the Maemo development environment provided by Nokia (hey, maybe I’ll write my first GTK+ program like that). This turns out to be a VMDK (VMWare) file, not a OVF file, so a little bit of fiddling about was necessary. There are installer scripts available with lots of disclaimers for VBox 2, but none for version 3. So I tweaked and twiddled a little, and ended up with this installer script that calls VBoxManage a bunch of times to set up the machine. This is largely cribbed from Nokia’s script, adapted for new syntax in version 3.
One thing to note here is the invocation with –usb on. That could be combined with other calls to VBoxManage, but since the OSE has no USB passthrough, this will fail. For those using the non-OSE version, the call will succeed. Run the script with either –add or –remove from the directory containing the .vmdk file and it’ll set things up or tear them down.
In the resulting VM I haven’t gotten any further than starting esbox and quitting it again — but it shows that the VM works, that the apps work. I can ssh in for whatever command-line work I need to do.
The upshot — after all, the Maemo dev environment is just an experiment — is that a Solaris VM will happen real soon now. I was hoping for something jeos-like, but I hear that the people responsible for that have left Oracle now (this looks like a common theme: all the nifty Open Source but non-revenue projects are bailing out). So I’ll have to find another means of installing a fairly-minimal OpenSolaris version into the VM (which, thanks to Jignesh, is a matter of VBoxManage import). For my purposes no X is needed, so that saves us at least one (if not two) desktop environments in the disk image.
Credits where credit is due: to Jignesh Shah for the OSOL image and to Miwi for not only porting KDE4 to FreeBSD but also acting as source of information for VirtualBox.
FreeBSD and Radeon 4350
April 16th, 2010
The revival of my FreeBSD system meant that I was once again confronted with Xorg driver issues. The on-board GeForce 7050 isn’t recognized by the nv(4x) driver, and the proprietary nvidia one is a no-go because I’m running FreeBSD amd64 (and the Linux driver only works on FreeBSD i386). So, time to shop around a little.
This is one of those cases where I wish the Internet would forget sometimes. Wading through the reports of video card compatibility from 2006 just isn’t useful. I had one (I thought) simple question: will an ATI Radeon 4350 work with Xorg 1.6.5 under FreeBSD 8-STABLE?
Perhaps it’s just my search-fu letting me down, but in the end I went and just bought one (as it’s the cheapest video card available in town across the river right now).
And the answer seems to be: yes, the Radeon 4350 is supported under FreeBSD 8-STABLE with Xorg 1.6.5_1,1 and the xf86-video-ati 6.12.4_1 driver. At least I can get twm up and running and exit that same twm and restart X multiple times. As usual it took longest for me to remember which ports to install to get a workable X locally (xorg-minimal + xorg-apps + dbus and hal seems to do the trick).
Excerpt lines from Xorg.0.log:
(--) PCI:*(0:2:0:0) 1002:954f:1043:02a8 ATI Technologies Inc RV710 [Radeon HD 4350] rev 0, Mem @ 0xc0000000/268435456, 0xdfff0000/65536, I/O @ 0x0000e800/256, BIOS @ 0x????????/65536
(II) RADEON(0): [dri] Found DRI library version 1.3.0 and kernel module version 1.31.0
(II) RADEON(0): Detected total video RAM=524288K, accessible=262144K (PCI BAR=262144K)
(II) RADEON(0): Output VGA-0 connected
(II) RADEON(0): Output HDMI-0 disconnected
(II) RADEON(0): Output DVI-0 disconnected
So, aside from the oddness of having 512MB present and only half of that accessible, it looks OK. KDE4 from ports is still compiling, so I can’t comment on any 3D or compositing features.
One little bit of coolness I hadn’t expected is this, from dmesg:
hdac1: mem 0xdffec000-0xdffeffff irq 19 at device 0.1 on pci2
So presumably the audio via HDMI will work as well. No way to test, as I don’t have anything to plug into that.
I should note, too, that running twm and xterm and Qt4 VirtualBox and then running Ubuntu 8.10 with GNOME inside that looks .. decidedly strange. Time traveling desktops.
Compliance Engineering
April 15th, 2010
Compliance engineering as a topic covers those activities that make it possible to ship a (consumer electronics) product that complies with the license(s) of the software contained in that product. That includes things like: figuring out what software actually is in the product (you’d be surprised how often vendors don’t even know); ensuring that you know what configurations and versions were chosen to put in the product; finding out what the licenses on those versions of the software are; finding out out what the obligations under those licenses are; and finally actually doing what those obligations demand. Hence, comply.
Comply or explain (to one of the organizations that look into enforcing software license obligations, like the BSA or gpl-violations.org).
The FSFE has long had a brief article on how to report and fix violations and Armijn Hemel at Loohuis Consulting has written a fairly lengthy compliance engineering guide (also some articles on LWN).
One popular license for software that tends to end up in consumer electronics products is the GPL. Either version 2 or version 3. It has some specific obligations that make compliance both important and sensitive. Those are the clauses requiring the complete corresponding source code, which means you need to know what the code is and how to provide it. It also means that for every binary release you need to provide the sources that can be used to create exactly that binary release. Not every company does that consistently.
Heck, I’ll name names: Conceptronic, a Dutch consumer electronics company, tries hard to comply. It delivers source code for the firmware shipped with the original release of devices, and it sometimes updates the available source tarballs. But not always. Dennis, the guy responsible, knows this is a problem. He tries, but time pressure and the upstream don’t always make it possible to do the right thing.
So there’s a company technically in the wrong where I’m willing to believe that they could be in the right if there was a little less effort involved, or a little better support in the compliance engineering process.
Enter, once more, Armijn and Shane, in their business guises of Loohuis Consulting and Opendawn Consulting. They work, shall we say, both sides of the fence: both in helping people improve their compliance processes and in tracking down violators later. For both sides, knowing which sources should have been supplied with a given binary release is of paramount importance.
So Shane and Armijn — supported by the Linux Foundation and Stichting NLnet — have produced a tool that helps in identifying what software has gone into a binary firmware image. It’s still in its infancy, but it can usefully detect Linux kernel versions, Busybox versions and configurations. That means it can be used — for products containing those pieces of software — to answer questions like “what sources and configuration files and scripts should be delivered with this product?” And that’s important because of the requirement in the GPL to provide (when necessary as defined by the other license obligations) the complete corresponding source code. Not just a bunch of tarballs and a “figure it out” notice; not just the upstream code, but whatever patches went into the device as well; and preferably not a whole bunch of extraneous cruft, either.
The tool makes it easier to do compliance checking from the outside, and easier and cheaper (as in Free beer) to do basic checking on the inside. It’s no replacement for a dedicated compliance engineer, but it does help a lot in answering questions about “what’s in here?” before firmware goes out the door.
I should add that the tool understands some common firmware packaging styles, so it will find and unpack and check things in a squashfs image. Upcoming features will add more filesystems, like concatenated squashfs filesystems, which will save a lot of time compared to running od -c, grepping for magic numbers, dd-ing things apart and then loopback mounting parts individually — that will become automatic.
You can find the tool (which is Free Software under the Apache license) at BinaryAnalysis.org. BA to the rescue. Man, I love it when a plan comes together.
Moving and Updating a FreeBSD Boot Disk
April 14th, 2010
Part of my FreeBSD update effort consists of tackling a peculiar problem. I haven’t found the problem described online anywhere else, so I’m going to detail the steps taken. But first, an effort to pin down the problem itself.
I have a (remote) server. It has no DVD drive. It should remain up as much as possible. It’s currently running 6.1-STABLE, which is horribly old. It does have a spare drive in the machine, available for whatever. So what I want to do is make that spare disk a bootable 8-STABLE disk remotely, set up the 8-STABLE system as much as possible (still remotely), then reboot into the new system in one swell foop.
Let me rephrase: how do I add a new boot disk to a FreeBSD system and create a full bootable system on it without using the CD installer?
Or with different emphasis: I want to upgrade FreeBSD to a new major release and want to keep a complete bootable old version around just in case.
Preliminaries: let’s assume you have a full /usr/src for the system you want to end up running (for me, that’s 8-STABLE, and it actually lives in /mnt/sys/src-8); also a full ports tree; also that the system is currently running and that the new disk is /dev/ad6. The desired end situation is that /dev/ad6 is bootable and contains the whole new system.
Note, though, that 6-STABLE can’t even compile 8-STABLE from a source checkout, because of libelf header file problems. You need to go through two stages here: go from 6- to 7-, then 7- to 8-. However, one might hope that the second step is less invasive (in my case, no need to update ports into 7-, so I can boot 7-STABLE just once to update to 8-STABLE).
Make backups now. Really. This is all about messing around with the fundaments of the system with the intention of not touching the installed system and keeping a safe “way back”, but it’s still hazardous. Make backups now. Make sure they’re on physically removable media and remove them. Make another copy. Take it to a remote location. Store it in a dragon-proof safe.
You may also want to take a look at this upgrade tutorial for some other preliminaries.
Setting up the disk: we have the (new, presumed empty) disk attached as /dev/ad6. We’ll slice and partition it so that it can be used. We will use the whole disk, but not in “dangerously dedicated” mode. So we’ll put a single slice on it (partition in Linux and just about everybody else’s parlance).
fdisk -BI /dev/ad6 # Single slice on whole disk
fdisk -a1 /dev/ad6 # Make that slice active
Now that we’ve got a disk with a FreeBSD slice on it — and the rather simple FreeBSD boot manager in the MBR — we can set up partitions in the slice so that we can allocate filesystems. This requires thinking about the disk layout and sizing filesystems (we wouldn’t have to do that if we used ZFS, but I’m sticking to ZFS on OpenSolaris only for now, even if it is no longer considered experimental in FreeBSD). This means editing the label using $EDITOR, so I’ll show the end result as well.
bsdlabel -w /dev/ad6s1 # Create standard label
bsdlabel -e /dev/ad6s1 # Edit the label
When using the -e option to bsdlabel, you get a text editor to futz around with the partition layout and you can screw it up pretty badly if you try. I used this setup, which makes use of the modern size deisgnators and auto-offset so you can read it as “4G for this, then three of 8G, then all the rest”. Partition c is historically the whole disk.
# /dev/ad6s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 4G 16 unused
b: 8G * swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 8G * unused
e: 8G * unused
f: * * unused
I’ve left all the fstypes as unused except for swap, since they will get updated by newfs(8) later and it saves typing. Not to mention that the parameters are all pretty uninteresting or not worth tuning at this point.
In FreeBSD you can refer to a filesystem through its device (e.g. /dev/ad6s1a) or through its label — at least, if you have GEOM labels enabled in your kernel or loaded as a module. The label allows you to assign a human-readable name to a disk partition or filesystem so you can later refer to it by name. The name is independent of the physical location of the partition, so you can label something “myroot” and later refer to /dev/label/myroot regardless of where the disk has gotten shuffled off to. That’s really quite handy when you swap drives or cables around or add another SATA controller that potentially bumps device names around. So we’ll label everything, including swap:
glabel label myswap /dev/ad6s1b
newfs -U -L myroot /dev/ad6s1a
newfs -U -L mytmp /dev/ad6s1d
newfs -U -L myvar /dev/ad6s1e
newfs -U -L myusr /dev/ad6s1f
Now that all the filesystems are created and named, we can move on to filling them up with an installed base system. Do note that we’re not done with making the disks bootable — but for that, we need the right bits from the still-to-be-populated filesystems.
Populating filesystems: the newly-created filesystems are available in the running system, so we’re going to mount them and then put the updated system in them. Let’s assume we have a mountpoint /mnt/newsys in the running system to begin with. So we will start with mounting them all to re-create the future filesystem hierarchy under that mountpoint. We’ll throw in devfs for good measure.
mount /dev/ufs/myroot /mnt/newsys
mkdir /mnt/newsys/{tmp,var,usr,dev}
mount /dev/ufs/mytmp /mnt/newsys/tmp
# Similar for var and usr
mount -t devfs devfs /mnt/newsys/dev
If we were to chroot to the (still empty) /mnt/newsys we’d see the filesystem layout we want, including devices and everything. But we still need to populate them, so it’s time to build a new world. We assumed that /usr/src contains the sources for the system we want to end up with (e.g. it’s been csup’ped to RELENG_8). Plan another activity for an hour or so while the next steps complete (depending on the compile speed of your running machine).
cd /usr/src
make world DESTDIR=/mnt/newsys
make buildkernel installkernel DESTDIR=/mnt/newsys
make distribution DESTDIR=/mnt/newsys
You’ll note that some of those commands show up in the jail(8) manpage, which is where I cribbed them from. Because setting up a new bootable system is a lot like setting up a jail, just with disk, filesystem, kernel and boot blocks thrown in. Speaking of which, let’s update all the boot bits with the newly-generated files.
boot0cfg -B -b /mnt/newsys/boot/boot0 /dev/ad6
bsdlabel -B -b /mnt/newsys/boot/boot /dev/ad6s1
The last step — bsdlabel — might not work just like that, as there’s issues with (re-)labeling mounted disks. You may have to copy the new boot file to the running system, umount the whole newsys tree and then label. I don’t remember exactly what I did. Regardless, make sure that the filesystems are mounted again afterwards. You may find this blog post which mentions bsdlabel helpful, although I can’t figure out gpart(8) myself and it doesn’t seem to work under a running 8-STABLE system either. Some futzing required if the dreaded bsdlabel(8) “Class not found” error pops up.
But carrying on, once the filesystems are mounted again, you’ll need to add several files to the newly-populated system for it to boot and be useful. These are: /boot/loader.conf (kernel modules) and /etc/fstab (otherwise it won’t mount / and get confusing during boot; feel free to use the labels of the partitions if you add glabel_load=”YES” to loader.conf) and /etc/resolver.conf and /etc/rc.conf. Generally you could copy them over from the running system.
Testing: after all this, we have a filesystem filled with an updated system, created pretty much as if we were building a jail. Make sure devfs is mounted in there, and you can actually use it. Let’s assume you have an interface configured with IP 192.168.43.70 for the jail to run with. You could run a shell in there:
jail /mnt/newsys newsys 192.168.43.70 /bin/sh
You could even use the traditional /etc/rc instead of /bin/sh to bring the whole jail up, but this is fraught with peril. As in “Danger, Will Robinson!” This is particularly so when the jail and the running system do not share a kernel version. I experimented with a 7-STABLE system and an 8-STABLE jail, and noted the following (these are not bugs):
- ls works, but ls -la fails with “unsupported system call”.
- uname reports the kernel version of the running system, so pkg_add -r will use the running system, not the new system, as a source for packages. Fetch them manually.
- Some shell constructs just hang. I tried to build the libtool22 port and it hung with /bin/sh spinning at 100%. Again, probably a syscall problem.
On the other hand, being able to check that the new system is functional enough to compile something is useful.
Deployment: the last step is to reboot the machine into the new system. If you have a console (yay ILOM! or otherwise yay physical access!) it’s easy to babysit the system. In my testing I just swapped disks around (yay hot-swap SATA bays in my desktop machine!) but on the remote system, I’m going to want it to boot the boot manager from the first disk — which is now the old system disk attached as /dev/ad0 — then chain to the new bootloader on the new system disk /dev/ad4 and then boot from there.
Frankly, that’s something I have not tested or tried yet. I expect that the following will work: boot0cfg -s 5 -o noupdate -t 40 /dev/ad0 ; bootcfg -s 1 -o noupdate -t 40 to chain from one to the next with 2-second timeouts on both, but again: not tested. Yay ILOM.
Upcoming Conferences
April 14th, 2010
With the FSFE’s Amsterdam Legal Workshop 2010 behind us and Akademy-BR evidently a great (if rainy) success, it’s time to look forward again. Spring, new life, birds a-cheepin’, etc.
Let’s look at the beginning of may: Linux Audio Conference in Utrecht, for sound junkies of all shapes. Phonon? Nope. But Reinhold Kainhofer — once of KPilot and KDE PIM — is speaking on music notation standards. I should drop by — I still owe him 20 EUR for domain registrations. Lots of other things that make me think “gosh, people do that on computers too?”
Right after the LAC you could move to Ede (about 24 minutes by train) for the NLUUG Spring Conference on System Administration with the LHC on tap. Yes, it needs system administration as well if it’s ever going to blow up the world.
There’s a gap then — fill me in, folks — and end of May will see the Ubuntu 10.04 release party on the 29th. That should keep everyone busy and I’ve got some Ubuntu Thinking Putty to inflict on various people there.
A Return to FreeBSD
April 14th, 2010
After a great deal of procrastinating, I’ve rearranged my home office again and restored my FreeBSD machine to its rightful place under the desk. It must have been switched off for several months now, as the update (buildworld for 7- and 8- as well as portupgrade -aPP) is taking forever. There’s a reason for running through this routine: there are a few things I want to test with systems upgrades and jails. In addition, I’d like to document how the EBN is set up, to the level of detail of including package names. This is part of an effort to clean up the EBN code and separate the tools from the website a little; that in turn is in advance of adding some new features to Krazy / EBN in general.
It’s kind of nice to be back on FreeBSD again. I’ll have to take note and compare the FreeBSD KDE packages with the OpenSolaris ones I produce.
PS. There’s nothing like cleaning your desk with a shovel and just dumping everything in a box never to be looked at again. I did retrieve several Bluetooth dongles and SD cards that I’d thought lost.
EBN up again
April 12th, 2010
The EBN machine is up again. Will Stephenson pointed out how the API documentation can be generated locally which takes away one reason for the machine (because the machine serves up api.kde.org). The value of the on-line version comes from multiple versions, indexing, searchability (although the PHP stuff that drives the search is rather poor, so that’s been on the stuff-to-fix-sometime for a long time), and saving you time in generating the whole shebang. The scripts and styling get occasional tweaks, too, to improve the API documentation. As always comments are welcome, patches are welcomer.
The machine does some other things, such as serving up my personal website (not updated in a gazillion years since I blog on FSFE’s Fellowship blogging platform) and that of Sebastian Kügler.
The main CPU load on the machine is running Krazy, the KDE code quality checker. It still produces lots of warnings and small items to fix in the KDE codebase. Sometimes I see people committing whole bunches of Krazy fixes. I recently saw it referred to as “KDE’s level grind”, which is a pretty good description in the sense that it was originally intended to find and explain the kind of low-hanging fruit you can fix on a Friday afternoon.
One last thing the machine does is provide the OpenSolaris packages for KDE4 as well. For this I ported pkg.depotd(8) to FreeBSD some time ago, but it’s starting to show its age and I think I’ll have to start running an actual OpenSolaris on the machine at some point. That means fiddling around with the available virtualization options and possibly updating and rebooting the machine repeatedly. If VirtualBox gives any measure of performance under FreeBSD, that’ll be it (because that simplifies updating from home where I also use it).
So, you may ask: what was the cause of this extended downtime? How can we prevent this from happening again? Well, one thing to fix it would be donating a 1U 4×3.5″ SATA disk server chassis with redundant power. Right now — thanks to being hosted in the “random crap” rack — there’s a little mini-tower thing doing the job, and it turned out to have a dodgy power cord. Some shifting and pushing-around of cables in the rack caused a momentary disconnect, which panicked the server and left it at a fsck(8) prompt for a while. This means, perhaps, that I should try to configure the system with “noauto” on the affected file systems, so that it comes up with reduced functionality even if the disk array is toast. Two other really important points: configure serial console support so that the ILOM can get at it and remember the ILOM password. Cue jumpering the server to reset the password and all the Fun that entails.
Anyway, it comes down to a few decisions made three years ago caused this downtime to be longer than expected. It’s up again, decisions have been amended, and we’re good to go for another three, I hope.
Amsterdam Legal Workshop
April 12th, 2010
The third edition of the FTF’s Amsterdam Legal Workshop is behind us. Two days of excellent weather (for Amsterdam in April, anyway), good food and in-depth legal wrangling on a variety of to pics. Like patents and how to work to defuse them for businesses and Free Software projects. Canonical kindly provided the nicest lawyer in Europe and a speaker who could rush 168 slides in 20 minutes, so their input is greatly appreciated as well. It was good to see friends from HP as well for the engineers-and-lawyers perspective.
Each year the event brings in new faces and new topics, and we’ve grown to the point where we have traditions and can afford a little bit of silliness amongst the serious talks and the networking and the hashing-out-of-issues-left-hanging-elsewhere. So near the end of the conference I got to hand out custard cream cakes of merit and one attendee asked me “so where’s your pink whip?” That, though, would mean spilling too much KDE over into the FSFE, which is something that’s not going to happen.
That picture over there is the group photo — at least, the one I can publish, because I haven’t cleared the publishing rights to pictures of the conference participants nor the (potentially copyrighted) images on their T-shirts. And of course, even a room full of happy lawyers should not be provoked.
Planning is getting underway for next year already — that was my big failure this year, to get started early enough, so this is overcompensating a little — and we’re aiming for the same week of April, 2011 (say the 7th and 8th). Next year Easter is much later, so the workshop and the holiday clash less. Location to be decided — if we’re ever going to break free of Amsterdam, it needs to be done now.
Amsterdam Legal Workshop
April 9th, 2010
Today is the first day of the Amsterdam Legal Workshop — in full I suppose that’s called the Free Software Foundation Europe’s Freedom Task Force European Legal Network yearly workshop in Amsterdam. As in 2008 and 2009, we have a room full of the top lawyers and technologists in the Free Software legal field. Thanks to the organizational efforts of Shane, Karsten, Hugo and Rainer we’ve got a full two days of talks and demonstrations. As in past years, new relationships develop as we bring different parties to a neutral, private conference. We also take stock of where we are on a global scale with respect to Free Software licensing and legal issues. Glyn Moody was kind enough to open up the conference with a talk on the (singular) conversion from analogue to digital which — as is Glyn’s wont — ties together the past and future and fields of law, biology and computer science. And from there, we’ve gone off into deep legal territory which I won’t write about, but it’s an education.