Month: September 2008

So in my last post, I replaced a failed hard drive. What if the drive isn’t bad? Maybe the mirror just got out of sync for some other reason. As I mentioned previously, sometimes metastat, iostat and cfgadm will still show the failed drive. In this case, it is very possible that the hard drive is still functional.

So, here’s how you can analyze the hard drive to verify that it is still good and then resync your mirror that was built through Solaris Volume Manager on Solaris 9. First we check metastat and find the failed mirrors and the bad drive is c1t0d0. The line showing the failed drive at the bottom (in bold) is often missing when the drive is dead. On this failure the drive is still visible and may be reusable.

# metastat
d7: Mirror
Submirror 0: d17
State: Needs maintenance
Submirror 1: d27
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 48049848 blocks (22 GB)

d17: Submirror of d7
State: Needs maintenance
Invoke: metareplace d7 c1t0d0s7 <new device>
Size: 48049848 blocks (22 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s7 0 No Maintenance Yes

d27: Submirror of d7
State: Okay
Size: 48049848 blocks (22 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s7 0 No Okay Yes

d3: Mirror
Submirror 0: d13
State: Needs maintenance
Submirror 1: d23
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 8386767 blocks (4.0 GB)

d13: Submirror of d3
State: Needs maintenance
Invoke: metareplace d3 c1t0d0s3 <new device>
Size: 8386767 blocks (4.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s3 0 No Maintenance Yes

d23: Submirror of d3
State: Okay
Size: 8386767 blocks (4.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s3 0 No Okay Yes

d1: Mirror
Submirror 0: d11
State: Needs maintenance
Submirror 1: d21
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 6292242 blocks (3.0 GB)

d11: Submirror of d1
State: Needs maintenance
Invoke: metareplace d1 c1t0d0s1 <new device>
Size: 6292242 blocks (3.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s1 0 No Maintenance Yes

d21: Submirror of d1
State: Okay
Size: 6292242 blocks (3.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s1 0 No Okay Yes

d0: Mirror
Submirror 0: d10
State: Needs maintenance
Submirror 1: d20
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 8386767 blocks (4.0 GB)

d10: Submirror of d0
State: Needs maintenance
Invoke: metareplace d0 c1t0d0s0 <new device>
Size: 8386767 blocks (4.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s0 0 No Maintenance Yes

d20: Submirror of d0
State: Okay
Size: 8386767 blocks (4.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes

Device Relocation Information:
Device Reloc Device ID
c1t1d0 Yes id1,sd@SSEAGATE_ST336607LSUN36G_3JA1WZCX00007348J389
c1t0d0 Yes id1,sd@SSEAGATE_ST336607LSUN36G_3JA1B1CW00007337MZU9

And then we check with iostat. We can also see the failed drive in this output and that it has no errors. Often failed drives will contain invalid or incomplete information:

# iostat -En
c1t0d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST336607LSUN36G Revision: 0207 Serial No: 0312A1B1CW
Size: 36.42GB <36418595328 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c1t1d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST336607LSUN36G Revision: 0207 Serial No: 0322A1WZCX
Size: 36.42GB <36418595328 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0

Next we see if it’s available through format and then start testing the drive with a surface analysis:

# format
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@1c,600000/scsi@2/sd@0,0
1. c1t1d0 <SUN36G cyl 24620 alt 2 hd 27 sec 107>
/pci@1c,600000/scsi@2/sd@1,0
Specify disk (enter its number): 0
selecting c1t0d0
format> analyze
analyze> read
Ready to analyze (won’t harm SunOS). This takes a long time,
but is interruptable with CTRL-C. Continue? y

pass 0
24619/26/53

pass 1
24619/26/53

Total of 0 defective blocks repaired.
analyze> refresh
Ready to analyze (won’t harm data). This takes a long time,
but is interruptable with CTRL-C. Continue? y

pass 0
24619/26/53

pass 1
24619/26/53

Total of 0 defective blocks repaired.
analyze> test
Ready to analyze (won’t harm data). This takes a long time,
but is interruptable with CTRL-C. Continue? y

pass 0 – pattern = 0xc6dec6de
24619/26/53

pass 1 – pattern = 0x6db6db6d
24619/26/53

Total of 0 defective blocks repaired.

Since everything went well on the analysis, let’s go ahead and reuse this drive and resync the mirrors. You’ll notice that I am using the metareplace command, but it’s different then recommended in metastat. You need to use the -e parameter to have it resync the original drive (this can also be used when you have physically replaced the drive as well).

# metareplace -e d0 c1t0d0s0
d0: device c1t0d0s0 is enabled
# metareplace -e d1 c1t0d0s1
d1: device c1t0d0s1 is enabled
# metareplace -e d3 c1t0d0s3
d3: device c1t0d0s3 is enabled
# metareplace -e d7 c1t0d0s7
d7: device c1t0d0s7 is enabled

You can now see that all the mirrors are re-syncing. Of course, you’ll want to keep an eye on this server to see if it fails again.

# metastat | grep -i resync
State: Resyncing
Resync in progress: 0 % done
State: Resyncing
c1t0d0s7 0 No Resyncing Yes
State: Resyncing
Resync in progress: 2 % done
State: Resyncing
c1t0d0s3 0 No Resyncing Yes
State: Resyncing
Resync in progress: 7 % done
State: Resyncing
c1t0d0s1 0 No Resyncing Yes
State: Resyncing
Resync in progress: 21 % done
State: Resyncing
c1t0d0s0 0 No Resyncing Yes

Don’t forget to run installboot if you rebuilt your s0 partition as well as re-adding metadevice database replicas if you removed them due to a reboot. For more info, check my previous post.

# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t0d0s0
# metadb -i
flags first blk block count
a m p luo 16 4096 /dev/dsk/c1t1d0s4
a p luo 4112 4096 /dev/dsk/c1t1d0s4
# metadb -a -c 2 c1t0d0s4
# metadb -i
flags first blk block count
a m p luo 16 4096 /dev/dsk/c1t1d0s4
a p luo 4112 4096 /dev/dsk/c1t1d0s4
a u 16 4096 /dev/dsk/c1t0d0s4
a u 4112 4096 /dev/dsk/c1t0d0s4

Last time I posted about my experience replacing a drive in an array created with Veritas Volume Manager. That was a RAID 0 array that had lost its data immediately when the drive died, so I didn’t worry about saving the data when rebuilding it. This time I was rebuilding a RAID 1 array that was built with Solaris Volume Manager. When I lost this hard drive the data was still intact on the existing functional drive and I wanted to keep it that way. So as far as disclaimers on this post, I kept my data with no problems, but don’t just copy and paste commands without understanding how it will effect your system. Every server is different, so don’t assume yours is setup the same as mine! So this is a Solaris 9 server, with two IDE drives setup into 4 separate mirrors: d0, d1, d3, & d7. Since there are two hard drives, there are two components or submirrors under each mirror. The first drive is always setup as 1x where x is the number from the parent mirror. The second drive is always 2x where x is the number from the parent mirror. So d0 is the parent mirror, d10 is the drive one submirror, and d20 is the drive two submirror. Make sense? Let’s go!

The first problem that occurs when you have two drives and you don’t notice that you lost one, is when you reboot or the system restarts for some reason. The problem is with the metadevice database replicas. Sun recommends that you store at least one metadevice database replica on each of your hard drives. So if you only have two drives, it is very likely you have an even number of replicas. Solaris 9 uses a majority consensus algorithm to determine if there are stale databases and will not boot without one more than half of the total replicas online. Quite obviously, this will cause your system to not boot if you have two hard drives and one dies. Here’s the console output for this situation:

metainit: hostname: stale databases

Insufficient metadevice database replicas located.

Use metadb to delete databases which are broken.
Ignore any “Read-only file system” error messages.
Reboot the system when finished to reload the metadevice database.
After reboot, repair any broken database replicas which were deleted.

Type control-d to proceed with normal startup,
(or give root password for system maintenance):

The solution is to simply remove the bad replicas and reboot. First check to see how the replicas are defined:

# metadb -i
flags first blk block count
M p 16 unknown /dev/dsk/c0t0d0s4
M p 4112 unknown /dev/dsk/c0t0d0s4
a m p lu 16 4096 /dev/dsk/c0t2d0s4
a p l 4112 4096 /dev/dsk/c0t2d0s4

Remove the replicas from the dead drive:

# metadb -d c0t0d0s4
metadb: rembrandt: c0t0d0s4: no metadevice database replica on device

Recheck that they’re gone:

# metadb -i
flags first blk block count
a m p lu 16 4096 /dev/dsk/c0t2d0s4
a p l 4112 4096 /dev/dsk/c0t2d0s4

Once logged out, the system will continue to boot and then reset again before fully booting.

# exit
logout
Resuming system initialization. Metadevice database will remain stale.

Once the system is back up and running, we can go about replacing the bad drive. The first thing you want to do is backup your lvm configs:

# cp -r /etc/lvm /root/lvm.backup

Now take a look at what mirrors/drives you have down. To keep the post shorter, I am just showing one mirror here. The others look the same. You’ll also notice at the end that in my case, the second hard drive isn’t showing up at all, a pretty good sign that the drive is dead. If it’s still listed you might have a chance that it’s still good. (I’ll go over that scenario in my next post.)

# metastat

… <SNIP> …

d0: Mirror
Submirror 0: d10
State: Needs maintenance
Submirror 1: d20
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 6295440 blocks (3.0 GB)

d10: Submirror of d0
State: Needs maintenance
Invoke: metareplace d0 c0t0d0s0
Size: 6295440 blocks (3.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t0d0s0 0 No Maintenance Yes

d20: Submirror of d0
State: Okay
Size: 6295440 blocks (3.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t2d0s0 0 No Okay Yes

Device Relocation Information:
Device Reloc Device ID
c0t2d0 Yes id1,dad@AWDC_WD1200BB-00CAA1=WD-WMA8C2168885

This can also be verified through iostat. If the drive is still in here, it has some useful information like model numbers and serial numbers. If it’s gone, the info is still useful too. You know which drive(s) are good and through the process of elimination, you know which one is bad.

# iostat -En
c0t2d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Model: WDC WD1200BB-00C Revision: 17.07W17 Serial No: WD-WMA8C2168885
Size: 120.03GB <120031641600 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0

This can also be verified through format and cfgadm:

# format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c0t2d0
/pci@1f,0/ide@d/dad@2,0
Specify disk (enter its number):

# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t2d0 disk connected configured unknown
c0::dsk/c0t3d0 CD-ROM connected configured unknown
usb0/1 unknown empty unconfigured ok
usb0/2 unknown empty unconfigured ok

Since, I had all the information I needed, I shutdown the server (since they were IDE drives and not hot swappable) and replaced the faulty drive with another one of the same model, size, etc. Once booted, I went into format, it saw the drive, so I went ahead and cleared the partitions since I had used this drive previously for something else. I’m not going to go through all the format screens, they’re self explanatory. So I went to format > 0 > part > zero out partitions (all but partition 2) > label > quit > quit. Next, I needed to partition the drive to be the same as the drive it was going to mirror. The prtvtoc command makes this easy. Just make sure you type the right drives:

# prtvtoc -h /dev/rdsk/c0t2d0s2 | fmthard -s – /dev/rdsk/c0t0d0s2
fmthard: New volume table of contents now in place.

You can verify that both partition tables match now if you like, compare the output of the two commands:

# prtvtoc -h /dev/rdsk/c0t0d0s2
# prtvtoc -h /dev/rdsk/c0t2d0s2

Now go into your lvm backup and cat out md.cf, copy it to another screen somewhere since you will be referring back to it several times. I have highlighted the lines I used:

# cd /root/lvm.backup/
# cat md.cf
d3 -m d13 d23 1
d13 1 1 c0t0d0s3
d23 1 1 c0t2d0s3
d7 -m d17 d27 1
d17 1 1 c0t0d0s7
d27 1 1 c0t2d0s7
d1 -m d11 d21 1
d11 1 1 c0t0d0s1
d21 1 1 c0t2d0s1
d0 -m d10 d20 1
d10 1 1 c0t0d0s0
d20 1 1 c0t2d0s0

Now we will be going through the process of detaching the failed submirror, clearing , rebuilding, and then reattaching it. Go through this for every submirror that has failed. I will show you the output from the first round then just the commands for the additional rounds. The third command (metainit) just uses the information from the md.cf file:

# metadetach -f d3 d13
d3: submirror d13 is detached
# metaclear d13
d13: Concat/Stripe is cleared
# metainit d13 1 1 c0t0d0s3
d13: Concat/Stripe is setup
# metattach d3 d13
d3: submirror d13 is attached

Now you can check the status of the resync with metastat:

# metastat d3
d3: Mirror
Submirror 0: d13
State: Resyncing
Submirror 1: d23
State: Okay
Resync in progress: 12 % done
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 4198320 blocks (2.0 GB)

d13: Submirror of d3
State: Resyncing
Size: 4198320 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t0d0s3 0 No Okay Yes

d23: Submirror of d3
State: Okay
Size: 4198320 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0t2d0s3 0 No Okay Yes

Device Relocation Information:
Device Reloc Device ID
c0t0d0 Yes id1,dad@AWDC_WD1200BB-00CAA1=WD-WMA8C2114374
c0t2d0 Yes id1,dad@AWDC_WD1200BB-00CAA1=WD-WMA8C2168885

Now run through the rest of the commands:

# metadetach -f d7 d17
# metaclear d17
# metainit d17 1 1 c0t0d0s7
# metattach d7 d17

# metadetach -f d1 d11
# metaclear d11
# metainit d11 1 1 c0t0d0s1
# metattach d1 d11

# metadetach -f d0 d10
# metaclear d10
# metainit d10 1 1 c0t0d0s0
# metattach d0 d10

Of course, you can always check at anytime any of the mirrors or submirrors to see what you’re doing. If you want to check the status on multiple rebuilds, just run:

# metastat | grep Resync
State: Resyncing
Resync in progress: 2 % done
State: Resyncing
State: Resyncing
Resync in progress: 99 % done
State: Resyncing
State: Resyncing
Resync in progress: 1 % done
State: Resyncing

Once the drives have finished syncing, we need to make sure that we can still boot off this drive in case the other drive fails:

# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t0d0s0

And, of course, we need to re-add the metadevice database replicas for the same reason:

# metadb -a -c 2 c0t0d0s4
# metadb -i
flags first blk block count
a u 16 4096 /dev/dsk/c0t0d0s4
a u 4112 4096 /dev/dsk/c0t0d0s4
a m p luo 16 4096 /dev/dsk/c0t2d0s4
a p luo 4112 4096 /dev/dsk/c0t2d0s4

On one of my Solaris 9 servers, I lost a hard drive on a large array that had been built with no redundancy. RAID 0 is a tempting option for some admins trying to get the most space that they can out of the capacity they have, but it is almost always the wrong choice. Anyway, a drive died and had to be replaced. This array had been built using Veritas Volume Manager, so here are the steps I used. This is all from the command line. There is also a Veritas Enterprise Administrator GUI (vea) that you can use, but it has problems occasionally. Anyway, this is what I did – keep in mind that my array was already destroyed, so I didn’t worry about any data loss. If you are trying to keep your data, don’t follow my steps! You’ve been warned.

Legend: gendg is the disk group, genlv is the logical volume, gengd04 is the bad drive, gengd07 is the good drive, genlv-01 is the failed plex. You can find most of this information with the vxprint command.

Umount the effected file system

/sbin/umount /u0

Add Disk 13 (gendg07) to gendg

/usr/sbin/vxdg -g gendg adddisk gendg07=Disk_13

Replace gendg04 with gendg07

/usr/sbin/vxdg -g gendg repldisk gendg04=gendg07

Dis-associate the failed plex

/usr/sbin/vxplex -g gendg dis genlv-01

Re-associate the plex, it will be rebuilt

/usr/sbin/vxplex -g gendg att genlv genlv-01

Recover the disk group gendg and the logical volume genlv

/usr/sbin/vxrecover -b -g gendg -sE genlv

Since this was RAID 0, recreate the file system

/usr/sbin/mkfs -F vxfs /dev/vx/rdsk/gendg/genlv

Check the file system

/usr/sbin/fsck -F vxfs /dev/vx/rdsk/gendg/genlv

Mount the file system again

/sbin/mount /u0

I will try to post a similar experience using Sun’s Solaris Volume Manager next.

If you want to extract the files from an rpm, it’s really easy. I needed to use this recently when I was going to install a printer driver in UNIX. The manufacturer had a linux rpm available for download, but I didn’t need the whole thing, just the PPD file. So, I ran this command and copied out the PPD file that I needed.

rpm2cpio package.rpm | cpio -dimv