Posts Tagged ‘upgrade’

Murphy’s Day

Wednesday, November 10th, 2010

If something weird is happening with a server, never think “It’ll just be an hour or two.” Never think “If I’m going to be in the server room anyway, I might as well do foo as well to another box.” Since I thought both of these foolish things, it shows off that there’s definitely areas of Linux system administration that I’m no good at and that are needlessly complicated, and that I’m an inveterate optimist when it comes to these things.

The CodeYard server — a five year old IBM x306 with hard drives showing over 30000 hours of continuous operation and which has had uptimes over 500 days — slowed to a crawl, then rebooted yesterday. Sjors pinged me by phone, so I biked to the University to take a look with him. While en-route, the box did another kernel panic while running fsck(8). Ugh.

Now, working on a server that has two partially-mirrored 250GB SATA-150 hard drives and only 1GB of RAM (seriously, when we got this machine it was a sensible box for supporting medium sized workgroups, now my phone has more oomph) just takes forever. It never takes just an hour or two to wait for GEOM mirror to complete and then the fsck(8) to wind up and then .. bam, another kernel panic. By the end of the day we hadn’t really pinned down what was causing the problem, but memcheck seems in order.

All the data — students SVN and git repositories — on the machine seems safe, but we’ve pretty much turned off all the services offered by the box by various service jails until we get things sorted out.

So one failure doesn’t a Murphy’s day make. The second is that my laptop — which worked in the morning and didn’t when I got to the server room — has suddenly forgotten that it has a display panel attached to it, so I don’t see a thing. Not even BIOS POST messages. It still seems to boot into Fedora OK and I can even log in to my wonderful pink desktop (now there’s a blessing in disguise). Can’t see a thing. This particularly puts a crimp in the plan to use the laptop as a KDE demonstration machine during the NLUUG fall conference. I might end up lugging a desktop machine along instead.

In parallel with all this I did some upgrades on the EBN machine, which was foolish of me. That server had been running off of a spare laptop drive for some time now — a situation that was bound to come crashing down at some point. So the plan was simple: add a 500GB data disk, put back the Sun 10kRPM SAS disk that came out of the machine some time ago, copy boot stuff to SAS disk, reboot, done.

Yeah, right.

Three things I’d forgotten: dump + restore no longer works, making disks bootable is non-trivial and initrd is some brain-dead invention intended to prevent you from moving things around effectively. Give me FreeBSD, which at least will boot (quickly) and then complain and you can type in the root directory for single-user mode in a human-friendly fashion.

In the end I dd’ed the old disk onto the new disk, then did a chroot and mkinitrd. It just doesn’t seem right. Maybe I’ve missed a really obvious manpage somewhere explaining how the boot process works nowadays and how to migrate an installation to a different disk (lazyweb!). Tracking down the remaining references to the old disk took a bit longer, but the machine is up-and-running again. Now my next challenge is to convince the disk subsystem that I hot-attached a new drive (which would be /dev/sdf) which is physically identical to /dev/sde, and then dd everything over again so there’s a spare boot disk.

Plenty of things to go wrong. In retrospect, the old Nethack adage serves best (e.g. when going down stairs while burdened with a cockatrice corpse) “just don’t do that.”

EBN,, and others back at last

Tuesday, May 18th, 2010

Well, it took ages, but the EBN and the different VMs it hosts are back. Add “sysadmin” to the list of occupations I probably shouldn’t attempt without (1) more training (2) a stricter schedule. The NLUUG spring conference on systems administration was quite educational — and fun, too, chatting with various companies and learning about NanoBSD and ZFS — but it didn’t give me any magical beans to fix what ailed the EBN.

So what was the problem? Well, the whole thing started (yay, placing the blame!) with Bertjan, who wanted a newer Qt version on the EBN for his software quality checking tools. The EBN ran 6.2-R, and the necessary Qt versions and stuff are not supported on that OS anymore. While the EOL for FreeBSD 6 is still six months away, the ports maintainers don’t necessarily want to support that. So we needed to update the OS to something newer.

There’s tools to do that now, but I’ve never used them, and anyway I don’t think they support FreeBSD 6. So that means lots of “make buildworld buildkernel installkernel installworld” kinds of steps. First off I found that doing the compilations took a lot longer than I expected (or hoped). So where I planned to go 6-6.4-7.3-8.0 in one day, the fact was that just compiling was going to take longer than that. I couldn’t pre compile everything either with the machine still up, because FreeBSD 8 doesn’t compile in a FreeBSD 6 environment. Hence the multiple steps. Note to self: update more frequently to avoid this kind of large upgrade.

Second problem was that the jails (virtual machines) on the server were poorly set up. They all had their own copies of the world. I hadn’t realized that a 6.2 jail wouldn’t work in a 7.3 host (for instance, ps fails and lots of other system tools don’t like it). If I had spent more time thinking, I would have realized that I could installworld to each jail again and things would be ok. Note to self: set up jails with an easily upgradeable world, as described in lots of best-practices documents on jails.

So I upgraded the host onwards to FreeBSD 8.0. Another long long compile, with no GNU screen to make it easier to deal with. Thank goodness for the ILOM and the system console redirection it provides.

Of course, then I went on to make delete-old-libs, which meant that the ports on the system — all of which were compiled against the 6.2 libraries — didn’t work anymore. Note to self: see that little note “in case no 3rd party program uses them anymore”? Keep it in mind next time.

So, after about two days, I had a base system updated to 8.0, no working jails at all, and all ports — both in the host and in the jails — broken. At this point, I started doing two things in parallel. Note to self: don’t. I started rebuilding the ports in the host system, and reconfiguring the jails to have a single base installation with just /home, /etc, /var and /usr/local local to each jail, using nullfs mounts; I also decided to drop the starting of jails in /etc/rc.local and to use the jail-launching support that is now built in (but which wasn’t, as far as I know, available in 6.0 which is when I first configured the machine). Note to self: that was actually a good idea, and thanks also to Sjors who reminded me of the jail_* variables.

So, rebuilding ports after a big step like that is complicated by the fact that perl, ruby, php and python all needed to be recompiled and portupgrade -apP sometimes doesn’t quite get it right. In any case I needed to rebuild the ruby stack first to get a working portupgrade. The other three languages were a mess, with some modules of the languages disappearing at inopportune points along the upgrade path. Basically I did portupgrade -apP ; pkgdb -F ; portinstall <something missing> an awful lot until things were working again. This morning I finally got rid of the last missing PHP 5.3 modules which brought the EBN parts back to life. Note to self: read UPDATING twice before doing this again.

Of course, all that would have been less problematic if the disk array hadn’t given out twice during the whole operation. Once the ridiculously heavy load on the machine caused a panic and once the power on one of the disks fluctuated enough to cause another panic. Running fsck on a 600GB filesystem with 14M inodes is not quick (especially if there’s a few directories with 1M files in each, as is the case with KDE SVN mirrors). Note to self: badger more people about a better disk array for KDE.

Combine all that with sickness and family time and that’s why it took a week. I’m blogging this for the notes to self for the next time I run an upgrade (resolution: when FreeBSD 8.1 comes out) and to notify folks that things should be back to normal. (If not, drop me a note in comments). One the positive side, the server is better organized now, disk usage is down a little bit, and future upgrades should be much easier.

Moving and Updating a FreeBSD Boot Disk

Wednesday, April 14th, 2010

Part of my FreeBSD update effort consists of tackling a peculiar problem. I haven’t found the problem described online anywhere else, so I’m going to detail the steps taken. But first, an effort to pin down the problem itself.

I have a (remote) server. It has no DVD drive. It should remain up as much as possible. It’s currently running 6.1-STABLE, which is horribly old. It does have a spare drive in the machine, available for whatever. So what I want to do is make that spare disk a bootable 8-STABLE disk remotely, set up the 8-STABLE system as much as possible (still remotely), then reboot into the new system in one swell foop.

Let me rephrase: how do I add a new boot disk to a FreeBSD system and create a full bootable system on it without using the CD installer?

Or with different emphasis: I want to upgrade FreeBSD to a new major release and want to keep a complete bootable old version around just in case.

Preliminaries: let’s assume you have a full /usr/src for the system you want to end up running (for me, that’s 8-STABLE, and it actually lives in /mnt/sys/src-8); also a full ports tree; also that the system is currently running and that the new disk is /dev/ad6. The desired end situation is that /dev/ad6 is bootable and contains the whole new system.

Note, though, that 6-STABLE can’t even compile 8-STABLE from a source checkout, because of libelf header file problems. You need to go through two stages here: go from 6- to 7-, then 7- to 8-. However, one might hope that the second step is less invasive (in my case, no need to update ports into 7-, so I can boot 7-STABLE just once to update to 8-STABLE).

Make backups now. Really. This is all about messing around with the fundaments of the system with the intention of not touching the installed system and keeping a safe “way back”, but it’s still hazardous. Make backups now. Make sure they’re on physically removable media and remove them. Make another copy. Take it to a remote location. Store it in a dragon-proof safe.

You may also want to take a look at this upgrade tutorial for some other preliminaries.

Setting up the disk: we have the (new, presumed empty) disk attached as /dev/ad6. We’ll slice and partition it so that it can be used. We will use the whole disk, but not in “dangerously dedicated” mode. So we’ll put a single slice on it (partition in Linux and just about everybody else’s parlance).

fdisk -BI /dev/ad6 # Single slice on whole disk
fdisk -a1 /dev/ad6 # Make that slice active

Now that we’ve got a disk with a FreeBSD slice on it — and the rather simple FreeBSD boot manager in the MBR — we can set up partitions in the slice so that we can allocate filesystems. This requires thinking about the disk layout and sizing filesystems (we wouldn’t have to do that if we used ZFS, but I’m sticking to ZFS on OpenSolaris only for now, even if it is no longer considered experimental in FreeBSD). This means editing the label using $EDITOR, so I’ll show the end result as well.

bsdlabel -w /dev/ad6s1 # Create standard label
bsdlabel -e /dev/ad6s1 # Edit the label

When using the -e option to bsdlabel, you get a text editor to futz around with the partition layout and you can screw it up pretty badly if you try. I used this setup, which makes use of the modern size deisgnators and auto-offset so you can read it as “4G for this, then three of 8G, then all the rest”. Partition c is historically the whole disk.

# /dev/ad6s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 4G 16 unused
b: 8G * swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 8G * unused
e: 8G * unused
f: * * unused

I’ve left all the fstypes as unused except for swap, since they will get updated by newfs(8) later and it saves typing. Not to mention that the parameters are all pretty uninteresting or not worth tuning at this point.

In FreeBSD you can refer to a filesystem through its device (e.g. /dev/ad6s1a) or through its label — at least, if you have GEOM labels enabled in your kernel or loaded as a module. The label allows you to assign a human-readable name to a disk partition or filesystem so you can later refer to it by name. The name is independent of the physical location of the partition, so you can label something “myroot” and later refer to /dev/label/myroot regardless of where the disk has gotten shuffled off to. That’s really quite handy when you swap drives or cables around or add another SATA controller that potentially bumps device names around. So we’ll label everything, including swap:

glabel label myswap /dev/ad6s1b
newfs -U -L myroot /dev/ad6s1a
newfs -U -L mytmp /dev/ad6s1d
newfs -U -L myvar /dev/ad6s1e
newfs -U -L myusr /dev/ad6s1f

Now that all the filesystems are created and named, we can move on to filling them up with an installed base system. Do note that we’re not done with making the disks bootable — but for that, we need the right bits from the still-to-be-populated filesystems.

Populating filesystems: the newly-created filesystems are available in the running system, so we’re going to mount them and then put the updated system in them. Let’s assume we have a mountpoint /mnt/newsys in the running system to begin with. So we will start with mounting them all to re-create the future filesystem hierarchy under that mountpoint. We’ll throw in devfs for good measure.

mount /dev/ufs/myroot /mnt/newsys
mkdir /mnt/newsys/{tmp,var,usr,dev}
mount /dev/ufs/mytmp /mnt/newsys/tmp
# Similar for var and usr
mount -t devfs devfs /mnt/newsys/dev

If we were to chroot to the (still empty) /mnt/newsys we’d see the filesystem layout we want, including devices and everything. But we still need to populate them, so it’s time to build a new world. We assumed that /usr/src contains the sources for the system we want to end up with (e.g. it’s been csup’ped to RELENG_8). Plan another activity for an hour or so while the next steps complete (depending on the compile speed of your running machine).

cd /usr/src
make world DESTDIR=/mnt/newsys
make buildkernel installkernel DESTDIR=/mnt/newsys
make distribution DESTDIR=/mnt/newsys

You’ll note that some of those commands show up in the jail(8) manpage, which is where I cribbed them from. Because setting up a new bootable system is a lot like setting up a jail, just with disk, filesystem, kernel and boot blocks thrown in. Speaking of which, let’s update all the boot bits with the newly-generated files.

boot0cfg -B -b /mnt/newsys/boot/boot0 /dev/ad6
bsdlabel -B -b /mnt/newsys/boot/boot /dev/ad6s1

The last step — bsdlabel — might not work just like that, as there’s issues with (re-)labeling mounted disks. You may have to copy the new boot file to the running system, umount the whole newsys tree and then label. I don’t remember exactly what I did. Regardless, make sure that the filesystems are mounted again afterwards. You may find this blog post which mentions bsdlabel helpful, although I can’t figure out gpart(8) myself and it doesn’t seem to work under a running 8-STABLE system either. Some futzing required if the dreaded bsdlabel(8) “Class not found” error pops up.

But carrying on, once the filesystems are mounted again, you’ll need to add several files to the newly-populated system for it to boot and be useful. These are: /boot/loader.conf (kernel modules) and /etc/fstab (otherwise it won’t mount / and get confusing during boot; feel free to use the labels of the partitions if you add glabel_load=”YES” to loader.conf) and /etc/resolver.conf and /etc/rc.conf. Generally you could copy them over from the running system.

Testing: after all this, we have a filesystem filled with an updated system, created pretty much as if we were building a jail. Make sure devfs is mounted in there, and you can actually use it. Let’s assume you have an interface configured with IP for the jail to run with. You could run a shell in there:

jail /mnt/newsys newsys /bin/sh

You could even use the traditional /etc/rc instead of /bin/sh to bring the whole jail up, but this is fraught with peril. As in “Danger, Will Robinson!” This is particularly so when the jail and the running system do not share a kernel version. I experimented with a 7-STABLE system and an 8-STABLE jail, and noted the following (these are not bugs):

  • ls works, but ls -la fails with “unsupported system call”.
  • uname reports the kernel version of the running system, so pkg_add -r will use the running system, not the new system, as a source for packages. Fetch them manually.
  • Some shell constructs just hang. I tried to build the libtool22 port and it hung with /bin/sh spinning at 100%. Again, probably a syscall problem.

On the other hand, being able to check that the new system is functional enough to compile something is useful.

Deployment: the last step is to reboot the machine into the new system. If you have a console (yay ILOM! or otherwise yay physical access!) it’s easy to babysit the system. In my testing I just swapped disks around (yay hot-swap SATA bays in my desktop machine!) but on the remote system, I’m going to want it to boot the boot manager from the first disk — which is now the old system disk attached as /dev/ad0 — then chain to the new bootloader on the new system disk /dev/ad4 and then boot from there.

Frankly, that’s something I have not tested or tried yet. I expect that the following will work: boot0cfg -s 5 -o noupdate -t 40 /dev/ad0 ; bootcfg -s 1 -o noupdate -t 40 to chain from one to the next with 2-second timeouts on both, but again: not tested. Yay ILOM.