Solaris 8: Bad Superblock at block 16

This weekend I got the joy of rebuilding a Solaris 8 server from scratch with backups. I had another server of the same hardware available to use for the restore, except it had Solaris 9 mirrored across 2 hard drives. Someday, I’ll go through and document the whole process, but basically I broke the mirror, repartitioned the second drive, mounted the new partitions inside of the Solaris 9 environment, restored the data and rebooted off the drive with the restored data.

The interesting thing that I found that I wanted to post about was the corruption of the Solaris 8 superblock that was caused by mounting the partitions within Solaris 9. There were changes made in fsck on Solaris 9 that made the superblock incompatible with Solaris 8. When you boot off of a Solaris 8 partition that has been mounted by Solaris 9, you’ll get the following errors:

The / file system (/dev/rdsk/c0t2d0s0) is being checked.
/dev/rdsk/c0t2d0s0: BAD SUPERBLOCK AT BLOCK 16: BAD VALUES IN SUPER BLOCK
/dev/rdsk/c0t2d0s0: USE AN ALTERNATE SUPERBLOCK TO SUPPLY NEEDED INFORMATION;
/dev/rdsk/c0t2d0s0: e.g. fsck [-F ufs] -o b=# [special …]
/dev/rdsk/c0t2d0s0: where # is the alternate super block. SEE fsck_ufs(1M).

/dev/rdsk/c0t2d0s0: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

WARNING – Unable to repair the / filesystem. Run fsck
manually (fsck -F ufs /dev/rdsk/c0t2d0s0). Exit the shell when
done to continue the boot process.

Of course, the drive path may be different, etc. The solution to this is simple, run fsck and restore the superblock from one of it’s backups – in this case I restored from block 32:

fsck -F ufs -y -o b=32 /dev/rdsk/c0t2d0s0

Run this on each of the affected partitions (all the partitions that you mounted) and you are good to go.

Postfix fails to shutdown on Linux VM

So I had a problem with a RHEL5 clone (OEL5) Linux VM that would not shutdown postfix correctly on either a powerdown or on a reboot. Here’s the error I saw:

Dec 15 16:43:01 testvm postfix[3160]: fatal: could not find any active network interfaces
Dec 15 16:44:04 testvm postfix/postfix-script: starting the Postfix mail system
Dec 15 16:44:04 testvm postfix/master[1995]: daemon started — version 2.3.3, configuration /etc/postfix

The first trick was realizing that the error was being generated during the shutdown and not during startup. I had checked through my main.cf and verified my /etc/hosts several times thinking that I had some configuration problem within Postfix itself. A virtual machine reboots so fast it’s very easy to miss a reboot within the timestamps.

This VM had the VMware Tools installed and apparently when you install the tools, it sets the priority on shutdown of the tools to be 08. The VMware Tools shutdown the network interfaces and a priority of 08 is way too early to be shutting down the network interfaces. Postfix is set to shutdown at 30 and normal Linux network interfaces are shutdown at 90. So, to fix this issue I simply changed the priority of the VMware Tools shutdown to 89:

mv /etc/rc0.d/K08vmware-tools /etc/rc0.d/K89vmware-tools
mv /etc/rc6.d/K08vmware-tools /etc/rc6.d/K89vmware-tools

I’m sure there is probably some way to do this through chkconfig or something, but sometimes it’s easier to do it the old fashioned way. Here’s my guess on how you would do it through chkconfig:

chkconfig vmware-tools off
vi /etc/init.d/vmware-tools # change chkconfig line from 08 to 89
chkconfig vmware-tools on

Of course, if you reinstall VMware Tools, it will probably replace the symlinks again with the default. Maybe it will even add a second symlink and then you’ll get more errors. I’ll let you know next time I upgrade the tools.

There are several things wrong with the VMware Tools, including it’s automatic installation of hgfs for folder sharing on a server, which I’ll post something about later. I’m still trying to decide if there is any advantage to installing the tools when there is no GUI, especially with it’s poor configuration setups.