Part of my FreeBSD update effort consists of tackling a peculiar problem. I haven’t found the problem described online anywhere else, so I’m going to detail the steps taken. But first, an effort to pin down the problem itself.
I have a (remote) server. It has no DVD drive. It should remain up as much as possible. It’s currently running 6.1-STABLE, which is horribly old. It does have a spare drive in the machine, available for whatever. So what I want to do is make that spare disk a bootable 8-STABLE disk remotely, set up the 8-STABLE system as much as possible (still remotely), then reboot into the new system in one swell foop.
Let me rephrase: how do I add a new boot disk to a FreeBSD system and create a full bootable system on it without using the CD installer?
Or with different emphasis: I want to upgrade FreeBSD to a new major release and want to keep a complete bootable old version around just in case.
Preliminaries: let’s assume you have a full /usr/src for the system you want to end up running (for me, that’s 8-STABLE, and it actually lives in /mnt/sys/src-8); also a full ports tree; also that the system is currently running and that the new disk is /dev/ad6. The desired end situation is that /dev/ad6 is bootable and contains the whole new system.
Note, though, that 6-STABLE can’t even compile 8-STABLE from a source checkout, because of libelf header file problems. You need to go through two stages here: go from 6- to 7-, then 7- to 8-. However, one might hope that the second step is less invasive (in my case, no need to update ports into 7-, so I can boot 7-STABLE just once to update to 8-STABLE).
Make backups now. Really. This is all about messing around with the fundaments of the system with the intention of not touching the installed system and keeping a safe “way back”, but it’s still hazardous. Make backups now. Make sure they’re on physically removable media and remove them. Make another copy. Take it to a remote location. Store it in a dragon-proof safe.
You may also want to take a look at this upgrade tutorial for some other preliminaries.
Setting up the disk: we have the (new, presumed empty) disk attached as /dev/ad6. We’ll slice and partition it so that it can be used. We will use the whole disk, but not in “dangerously dedicated” mode. So we’ll put a single slice on it (partition in Linux and just about everybody else’s parlance).
fdisk -BI /dev/ad6 # Single slice on whole disk
fdisk -a1 /dev/ad6 # Make that slice active
Now that we’ve got a disk with a FreeBSD slice on it — and the rather simple FreeBSD boot manager in the MBR — we can set up partitions in the slice so that we can allocate filesystems. This requires thinking about the disk layout and sizing filesystems (we wouldn’t have to do that if we used ZFS, but I’m sticking to ZFS on OpenSolaris only for now, even if it is no longer considered experimental in FreeBSD). This means editing the label using $EDITOR, so I’ll show the end result as well.
bsdlabel -w /dev/ad6s1 # Create standard label
bsdlabel -e /dev/ad6s1 # Edit the label
When using the -e option to bsdlabel, you get a text editor to futz around with the partition layout and you can screw it up pretty badly if you try. I used this setup, which makes use of the modern size deisgnators and auto-offset so you can read it as “4G for this, then three of 8G, then all the rest”. Partition c is historically the whole disk.
# /dev/ad6s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 4G 16 unused
b: 8G * swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 8G * unused
e: 8G * unused
f: * * unused
I’ve left all the fstypes as unused except for swap, since they will get updated by newfs(8) later and it saves typing. Not to mention that the parameters are all pretty uninteresting or not worth tuning at this point.
In FreeBSD you can refer to a filesystem through its device (e.g. /dev/ad6s1a) or through its label — at least, if you have GEOM labels enabled in your kernel or loaded as a module. The label allows you to assign a human-readable name to a disk partition or filesystem so you can later refer to it by name. The name is independent of the physical location of the partition, so you can label something “myroot” and later refer to /dev/label/myroot regardless of where the disk has gotten shuffled off to. That’s really quite handy when you swap drives or cables around or add another SATA controller that potentially bumps device names around. So we’ll label everything, including swap:
glabel label myswap /dev/ad6s1b
newfs -U -L myroot /dev/ad6s1a
newfs -U -L mytmp /dev/ad6s1d
newfs -U -L myvar /dev/ad6s1e
newfs -U -L myusr /dev/ad6s1f
Now that all the filesystems are created and named, we can move on to filling them up with an installed base system. Do note that we’re not done with making the disks bootable — but for that, we need the right bits from the still-to-be-populated filesystems.
Populating filesystems: the newly-created filesystems are available in the running system, so we’re going to mount them and then put the updated system in them. Let’s assume we have a mountpoint /mnt/newsys in the running system to begin with. So we will start with mounting them all to re-create the future filesystem hierarchy under that mountpoint. We’ll throw in devfs for good measure.
mount /dev/ufs/myroot /mnt/newsys
mkdir /mnt/newsys/{tmp,var,usr,dev}
mount /dev/ufs/mytmp /mnt/newsys/tmp
# Similar for var and usr
mount -t devfs devfs /mnt/newsys/dev
If we were to chroot to the (still empty) /mnt/newsys we’d see the filesystem layout we want, including devices and everything. But we still need to populate them, so it’s time to build a new world. We assumed that /usr/src contains the sources for the system we want to end up with (e.g. it’s been csup’ped to RELENG_8). Plan another activity for an hour or so while the next steps complete (depending on the compile speed of your running machine).
cd /usr/src
make world DESTDIR=/mnt/newsys
make buildkernel installkernel DESTDIR=/mnt/newsys
make distribution DESTDIR=/mnt/newsys
You’ll note that some of those commands show up in the jail(8) manpage, which is where I cribbed them from. Because setting up a new bootable system is a lot like setting up a jail, just with disk, filesystem, kernel and boot blocks thrown in. Speaking of which, let’s update all the boot bits with the newly-generated files.
boot0cfg -B -b /mnt/newsys/boot/boot0 /dev/ad6
bsdlabel -B -b /mnt/newsys/boot/boot /dev/ad6s1
The last step — bsdlabel — might not work just like that, as there’s issues with (re-)labeling mounted disks. You may have to copy the new boot file to the running system, umount the whole newsys tree and then label. I don’t remember exactly what I did. Regardless, make sure that the filesystems are mounted again afterwards. You may find this blog post which mentions bsdlabel helpful, although I can’t figure out gpart(8) myself and it doesn’t seem to work under a running 8-STABLE system either. Some futzing required if the dreaded bsdlabel(8) “Class not found” error pops up.
But carrying on, once the filesystems are mounted again, you’ll need to add several files to the newly-populated system for it to boot and be useful. These are: /boot/loader.conf (kernel modules) and /etc/fstab (otherwise it won’t mount / and get confusing during boot; feel free to use the labels of the partitions if you add glabel_load=”YES” to loader.conf) and /etc/resolver.conf and /etc/rc.conf. Generally you could copy them over from the running system.
Testing: after all this, we have a filesystem filled with an updated system, created pretty much as if we were building a jail. Make sure devfs is mounted in there, and you can actually use it. Let’s assume you have an interface configured with IP 192.168.43.70 for the jail to run with. You could run a shell in there:
jail /mnt/newsys newsys 192.168.43.70 /bin/sh
You could even use the traditional /etc/rc instead of /bin/sh to bring the whole jail up, but this is fraught with peril. As in “Danger, Will Robinson!” This is particularly so when the jail and the running system do not share a kernel version. I experimented with a 7-STABLE system and an 8-STABLE jail, and noted the following (these are not bugs):
- ls works, but ls -la fails with “unsupported system call”.
- uname reports the kernel version of the running system, so pkg_add -r will use the running system, not the new system, as a source for packages. Fetch them manually.
- Some shell constructs just hang. I tried to build the libtool22 port and it hung with /bin/sh spinning at 100%. Again, probably a syscall problem.
On the other hand, being able to check that the new system is functional enough to compile something is useful.
Deployment: the last step is to reboot the machine into the new system. If you have a console (yay ILOM! or otherwise yay physical access!) it’s easy to babysit the system. In my testing I just swapped disks around (yay hot-swap SATA bays in my desktop machine!) but on the remote system, I’m going to want it to boot the boot manager from the first disk — which is now the old system disk attached as /dev/ad0 — then chain to the new bootloader on the new system disk /dev/ad4 and then boot from there.
Frankly, that’s something I have not tested or tried yet. I expect that the following will work: boot0cfg -s 5 -o noupdate -t 40 /dev/ad0 ; bootcfg -s 1 -o noupdate -t 40 to chain from one to the next with 2-second timeouts on both, but again: not tested. Yay ILOM.