Bobulate

If something weird is happening with a server, never think “It’ll just be an hour or two.” Never think “If I’m going to be in the server room anyway, I might as well do foo as well to another box.” Since I thought both of these foolish things, it shows off that there’s definitely areas of Linux system administration that I’m no good at and that are needlessly complicated, and that I’m an inveterate optimist when it comes to these things.

The CodeYard server — a five year old IBM x306 with hard drives showing over 30000 hours of continuous operation and which has had uptimes over 500 days — slowed to a crawl, then rebooted yesterday. Sjors pinged me by phone, so I biked to the University to take a look with him. While en-route, the box did another kernel panic while running fsck(8). Ugh.

Now, working on a server that has two partially-mirrored 250GB SATA-150 hard drives and only 1GB of RAM (seriously, when we got this machine it was a sensible box for supporting medium sized workgroups, now my phone has more oomph) just takes forever. It never takes just an hour or two to wait for GEOM mirror to complete and then the fsck(8) to wind up and then .. bam, another kernel panic. By the end of the day we hadn’t really pinned down what was causing the problem, but memcheck seems in order.

All the data — students SVN and git repositories — on the machine seems safe, but we’ve pretty much turned off all the services offered by the box by various service jails until we get things sorted out.

So one failure doesn’t a Murphy’s day make. The second is that my laptop — which worked in the morning and didn’t when I got to the server room — has suddenly forgotten that it has a display panel attached to it, so I don’t see a thing. Not even BIOS POST messages. It still seems to boot into Fedora OK and I can even log in to my wonderful pink desktop (now there’s a blessing in disguise). Can’t see a thing. This particularly puts a crimp in the plan to use the laptop as a KDE demonstration machine during the NLUUG fall conference. I might end up lugging a desktop machine along instead.

In parallel with all this I did some upgrades on the EBN machine, which was foolish of me. That server had been running off of a spare laptop drive for some time now — a situation that was bound to come crashing down at some point. So the plan was simple: add a 500GB data disk, put back the Sun 10kRPM SAS disk that came out of the machine some time ago, copy boot stuff to SAS disk, reboot, done.

Yeah, right.

Three things I’d forgotten: dump + restore no longer works, making disks bootable is non-trivial and initrd is some brain-dead invention intended to prevent you from moving things around effectively. Give me FreeBSD, which at least will boot (quickly) and then complain and you can type in the root directory for single-user mode in a human-friendly fashion.

In the end I dd’ed the old disk onto the new disk, then did a chroot and mkinitrd. It just doesn’t seem right. Maybe I’ve missed a really obvious manpage somewhere explaining how the boot process works nowadays and how to migrate an installation to a different disk (lazyweb!). Tracking down the remaining references to the old disk took a bit longer, but the machine is up-and-running again. Now my next challenge is to convince the disk subsystem that I hot-attached a new drive (which would be /dev/sdf) which is physically identical to /dev/sde, and then dd everything over again so there’s a spare boot disk.

Plenty of things to go wrong. In retrospect, the old Nethack adage serves best (e.g. when going down stairs while burdened with a cockatrice corpse) “just don’t do that.”

The revival of my FreeBSD system meant that I was once again confronted with Xorg driver issues. The on-board GeForce 7050 isn’t recognized by the nv(4x) driver, and the proprietary nvidia one is a no-go because I’m running FreeBSD amd64 (and the Linux driver only works on FreeBSD i386). So, time to shop around a little.

This is one of those cases where I wish the Internet would forget sometimes. Wading through the reports of video card compatibility from 2006 just isn’t useful. I had one (I thought) simple question: will an ATI Radeon 4350 work with Xorg 1.6.5 under FreeBSD 8-STABLE?

Perhaps it’s just my search-fu letting me down, but in the end I went and just bought one (as it’s the cheapest video card available in town across the river right now).

And the answer seems to be: yes, the Radeon 4350 is supported under FreeBSD 8-STABLE with Xorg 1.6.5_1,1 and the xf86-video-ati 6.12.4_1 driver. At least I can get twm up and running and exit that same twm and restart X multiple times. As usual it took longest for me to remember which ports to install to get a workable X locally (xorg-minimal + xorg-apps + dbus and hal seems to do the trick).

Excerpt lines from Xorg.0.log:

(--) PCI:*(0:2:0:0) 1002:954f:1043:02a8 ATI Technologies Inc RV710 [Radeon HD 4350] rev 0, Mem @ 0xc0000000/268435456, 0xdfff0000/65536, I/O @ 0x0000e800/256, BIOS @ 0x????????/65536
(II) RADEON(0): [dri] Found DRI library version 1.3.0 and kernel module version 1.31.0
(II) RADEON(0): Detected total video RAM=524288K, accessible=262144K (PCI BAR=262144K)
(II) RADEON(0): Output VGA-0 connected
(II) RADEON(0): Output HDMI-0 disconnected
(II) RADEON(0): Output DVI-0 disconnected


So, aside from the oddness of having 512MB present and only half of that accessible, it looks OK. KDE4 from ports is still compiling, so I can’t comment on any 3D or compositing features.

One little bit of coolness I hadn’t expected is this, from dmesg:

hdac1: mem 0xdffec000-0xdffeffff irq 19 at device 0.1 on pci2

So presumably the audio via HDMI will work as well. No way to test, as I don’t have anything to plug into that.

I should note, too, that running twm and xterm and Qt4 VirtualBox and then running Ubuntu 8.10 with GNOME inside that looks .. decidedly strange. Time traveling desktops.

I wrote recently about a little machine logbook I found in a disused storage cabinet in the old science building. The machine it refers to might be one produced by Floating Point Systems, Inc., since I found a processor manual (part #860-7259-003) in the same cabinet, and some of the instructions seem similar. The manual was published in 1979 (heck, I don’t think I had even seen a computer at the time) by FPS Inc. of Beaverton, Oregon. The manual starts out promising:

Historically, array transform processors have been largely integer-arithmetic devices, since the slower processing rate of floating-point arithmetic was undesirable when working with large arrays of data. However, integer methods have problems which make programming awkward due to the limited dynamic range of integer arithmetic. Array scaling and block floating-point techniques either allowed human and other errors to creep into the results or were costly and time consuming. Further, as processing became more sophisticated, even 16-bit integer data words were insufficiently precise for preserving the accuracy of simple 8-bit analog-to-digital converted input data. This is because the many multiplications and additions in typical cascaded array processing can cause the propagation of truncation errors.

You know, I think the exact same paragraph (well, ok, let’s increase the numbers from 16 to 64, for instance) could apply today. Paging forward, we find that the floating-point representation used in this machine is “..a true 10-bit binary exponent, which has more dynamic range than the standard 7-bit hexadecimal or 8-bit binary exponent. FPS then uses a 28-bit mantissa, plus three guard bits in the adder and a double mantissa at the multiplier output, …” Count ’em, a 38-bit floating point representation. Bear in mind that IEEE 754 was created in 1985, with a 32-bit format.

Moving on to the functional units of the processor, we find both a main memory (MD) of up to 64k words (each 38 bits) plus a faster memory (167ns cycle time) called the table memory (TM) of 512 words, each 38 bits. The idea is that you store constants in table memory and use an index into the table memory instead of direct mode. There’s two banks of accumulators, 32 in each bank, so one might speak of having 64 floating point registers. There’s some data path fiddly bits here, and anyone who has read Tanenbaum’s computer design books will understand the trade-off between bus width, control line count and ease of (micro-)programming. The adders and floating-point multiplier are pipelined, with two and three stages respectively.

The processor has DMA capability — not within its own memory, but to the memory of a host machine. In other words, it hooks up to the memory bus of some other processor, can copy blocks of data to the local memory, do computations, and then move them back. So it’s a co-processor.

And the whole thing fits in 16U of 19″ rack space and draws 1200W.

This goes to show that we (as in computer science and the computer industry) have come a long way since 1979. It also shows that there’s very little truly new stuff — stream processors in your graphics card are similar in many ways to this old beast. Also, in the best style of wikis everywhere, the manual ends up with a “help us improve this manual, send in comments on the following postage-paid form.” See, everything old is new again.

At the university of Nijmegen I was an inveterate collector of all-things-strange. The old sciences building was cnducive to that behavior, since it had lots and lots of storerooms with 40 years of accumulated junk, most of which were poorly secured or just left unlocked. So wandering through the basements looking for neat stuff was a good student-lunchtime activity. The building was knocked down a few years back, replaced by a shiny green glass-and-steel contraption that is much nicer to work in, but which has a great deal less charm. I still vaguely regret not nicking the elevator maintainence notebook the week before the old building was chained shut. One thing I did pick up was a notebook labeled “ARGS-Logbook” from the computer graphics department. Kept in meticulous handwriting from july 1981 to july 1985, it is full of notes that are both typically WTF and reminiscent of the problems anyone might have when dealing with machine-language programming.

July 1981 It seems impossible to pass parameters as subroutine arguments. Some “arbitrary” value is taken when referencing to such and argument. Strange is however that RET with a non-zero offset does work. (Solved middle of dec ’81). July 1981 The instructions to turn the scanner on/off works the wother way round, i.e. X is 0 for on and 1 for off!! (Doc adapted). September 1981 The conditional branche does not work properly. CMP followed by BRALT results in a branch if greater than! So the neg/pos bit seems to be set wrong (OK after middle of dec ’81). December 1981 Z Status Register instruction can only be used in a specific way. (after partitioning pictures across planes a new partitioning has to be preceded by an INI instruction = reset).

It seems that this was a machine with hardware for image manipulation (with instructions like ARF, area fill), which got a hardware upgrade at the end of 1981. I’ve left out all the complaints about the monitor which also mostlye went away at the end of that year. New issues show up, though:

August 1982 TST instruction does not work correctly (ok now, november 1982, error in cross-assembler). November 1983 Area fill problem. When pixel value at xset, yset = boundary value, the ARFO instruction hangs. February 1985 Recording screen on video impossible because of 60Hz frequency sync. (For changing video frequency, set switches E11, F11 and G11 in graphics store slot 4, need 33 msec refresh at 50Hz, set to 0011 0011)

The last entry in the logbook is the nicest: 18 July 1985 Key of monitor missing. Solution? (Lock turned 90 degrees so key is not needed).

I regularly visit fairs or attend development sprints, and in the course of a few years I’ve gathered together something I call a “mini booth-box”. It’s a plastic crate full of odds and ends that are invariably useful at an event; I can either take the whole box or cherry-pick from it for specific events. Like at FOSDEM this year, where I packed 2 19″ flatscreens, a Sun Ray, two keyboards and mice and a laptop into my backpack. Forgetting, of course, the required switch and stuff. Well, the best-laid plans …

Anyway, here’s a list of what’s in that crate, in the hope that it might be helpful for someone else.

Networking: A 24-port hub and its power lead; a 5-port switch and its wall wart; a WRT54GL and its wall wart; a crossover cable; 25m, 10m, 2x5m and a 2m straight UTP cables. Video: DVD-D dual link cable; VGA D-sub cable. DVI-I to D-sub adapter. Input: PS/2 mini-size keyboard; USB keyboard; USB mouse. Misc: 1GB USB stick; bluetooth dongle; 1m eSATA cable; 1.5m USB A-B cable; 1.5m USB A-mini cable; Nokia phone carger (mini-connector). 4-socket power strip. UK-Euro plug adapter.

I know some of that is redundant, overly specialized or just crap, but it serves its purpose of giving me a one-stop shop in house for picking up equipment when going to a fair. I just hope I didn’t forget anything. The Sun Ray thin client doesn’t travel in the box, but it does good service at stands if I want to use my laptop while at the same time leaving the laptop at the booth for display / demonstration purposes.

It’s hardly rocket science, but I have derived satisfaction this evening from changing the tire on my bicycle. It no longer goes wobble and bump — the sidewalls were tearing on the old tire — and my bike feels a good deal faster again. The previous tire lasted two years and several thousand km of touring and commuting.

I broke the work down into little steps: remove wheel from bike (quick release axle); remove old tire; put new tire on; reinsert inner tube; mount in frame. Between those steps I installed three different Free Software operating systems on a new laptop. My old Thinkpad is giving up the ghost — it overheats regularly when it’s warmer than 24 degrees or when I run any kind of compile on it — so it is time for a new one. Especially with the Gran Canaria Desktop Summit upcoming, and I want to be able to work while somewhere warm.

Somewhat to my surprise, OpenSolaris works best on the new machine; Kubuntu 8.10 (admittedly that’s an older version) is ok and FreeBSD 7.1 just doesn’t go. I gather that’s related to SATA DVD drives, and FreeBSD 8 snapshots don’t like the drive either. Unfortunately, this whole thing leads to a dilemma: given a nicely working new-ish bicycle and a nicely working new laptop, which do you choose? It’s not safe to code and cycle, kids (unless you’re riding something like this).