Adding a new LUN on Solaris 9 with VXFS and PowerPath

There’s not a lot of documentation out there on working with LUNs on servers that use both Veritas Storage Foundation and EMC PowerPath. I have spent the last few weeks perfecting the process of Dynamic LUN Expansion and will be adding more posts about that process in the near future. Most recently I have been working on Solaris 9 servers connected to an EMC Clariion CX500 SAN that have LUNs mounted using the VXFS (Veritas) file system (version 4.1,REV=4.1B18_sol_GA_s10b74L2a) using EMC PowerPath (version 4.5.0_b169). The servers have Qlogic HBAs so I use their utilities to scan the Fibre Channel paths. If you have a different HBA, use that vendor’s tools.

This setup, although not unusual, does make it more complicated. When a new LUN is connected to the host, by default, Veritas will see it first and claim the drive. This is a bad thing, because we want PowerPath to find the drive, setup the pseudo drive and encapsulate the multipathing.

Previous administrators with this setup were unable to get PowerPath to recognize the drive before Veritas without rebooting the server. They would connect the LUN and reboot the server. If Veritas detected the drive first, they would remove the disk from Veritas and reboot until PowerPath claimed the drive. This, of course, to me was unacceptable, so I have decided to document the procedure I follow for properly connecting a new LUN to a host with this configuration. (Note: If you have already got yourself in this situation, you will still need to reboot to cleanup Veritas’ hold on the drive. The alternative to rebooting would be to allocate another LUN for the host and worry about cleaning up the previous one later.)

So how do you get Veritas to not claim the drive when you connect the LUN to the host? Simple, turn off the service that discovers the drive! If you dig into the init scripts for Veritas (/etc/init.d/vxvm-startup2), you’ll see starting on line 193 the following lines:

# Start Event Source Daemon, to enable dynamic device discovery
# Empty the lock file before starting the daemon
> /etc/vx/.vxesd.lock

vxddladm start eventsource

This dynamic device discovery service must be turned off by executing the following command:

# vxddladm stop eventsource

Of course, if you don’t want dynamic discovery service running, you should also comment out line 197 so that it is never started in the first place. I have found that it is necessary to run the stop command multiple times to make sure it is off. Even if I have stopped it on a particular server before, I run it again before connecting any LUNs, just to make sure.

So here are the steps with sample output if any was generated:

Turn off Veritas dynamic discovery (IMPORTANT!!):

# vxddladm stop eventsource

Rescan your fibre connections, optionally verify that you can see the new LUN:

# /opt/QLogic_Corporation/SANsurferCLI/scli -do all rescan
Driver rescan completed on HBA instance 0.
Driver rescan completed on HBA instance 1.

Tell Solaris you’ve made some changes and let it make the proper devices:

# devfsadm -v -C

Configure PowerPath to work with the new LUN:

# /etc/powercf -q
# /etc/powermt config
# /etc/powermt save

Check that your new LUN is properly configured in PowerPath:

# /etc/powermt display dev=all
— cut —
Pseudo name=emcpower6a
CLARiiON ID=APM00012345678 [Test Storage Group]
Logical device ID=60060160A42611000A2B03D123456789 [LUN – TEMP LUN]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A
===============================================
—————- Host ————— – Stor – — I/O Path – — Stats —
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
===============================================
1280 pci@9/fibre-channel@2 c4t1d6s0 SP B0 active alive 0 0
1280 pci@9/fibre-channel@2 c4t2d6s0 SP A0 active alive 0 0
1281 pci@9/fibre-channel@2 c5t1d6s0 SP B1 active alive 0 0
1281 pci@9/fibre-channel@2 c5t2d6s0 SP A1 active alive 0 0
— cut —

Now we need to create a label on the drive, use the PowerPath pseudo name:

# format
Searching for disks…done

c4t1d6: configured with capacity of 349.99GB
c4t2d6: configured with capacity of 349.99GB
c5t1d6: configured with capacity of 349.99GB
c5t2d6: configured with capacity of 349.99GB
emcpower6a: configured with capacity of 349.99GB

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w210000008710a5b2,0
— cut —
40. emcpower6a
/pseudo/emcp@6
Specify disk (enter its number): 40
selecting emcpower6a
[disk formatted]
Disk not labeled. Label it now? yes
format> q

Get rid of the default partitions created by the label command:

# fmthard -d 0:00:00:0:0 /dev/rdsk/emcpower6a
# fmthard -d 1:00:00:0:0 /dev/rdsk/emcpower6a
# fmthard -d 6:00:00:0:0 /dev/rdsk/emcpower6a

Verify that everything looks the way you want it:

# format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w210000008710a5b2,0
— cut —
40. emcpower6a
/pseudo/emcp@6
Specify disk (enter its number): 40
selecting emcpower6a
[disk formatted]
format> verify

Primary label contents:

Volume name =
ascii name =
pcyl = 57344
ncyl = 57342
acyl = 2
nhead = 256
nsect = 50
Part Tag Flag Cylinders Size Blocks
0 unassigned wm 0 0 (0/0/0) 0
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 – 57341 349.99GB (57342/0/0) 733977600
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0

format> q

If you’ve done everything right, Veritas should not see your new drive yet. Verify:

# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
emcpower0s2 auto:sliced devdg-erp02 devdg-erp online
emcpower1s2 auto:cdsdisk testdg-erp02 testdg-erp online
emcpower2s2 auto:sliced devdg-dw02 devdg-dw online
emcpower3s2 auto:sliced devdg-tle01 devdg-tle online
emcpower4s2 auto:sliced dbadg-erp01 dbadg-erp online

Now that everything is setup right, go ahead and tell Veritas to look for new drives. Important Note: If it is not setup right at this point, Veritas will claim the raw device and it’s a pain to cleanup!

# vxdctl enable

Now, you will see the pseudo drive in the list. If it’s a raw device name, you messed up:

# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
emcpower0s2 auto:sliced devdg-erp02 devdg-erp online
emcpower1s2 auto:cdsdisk testdg-erp02 testdg-erp online
emcpower2s2 auto:sliced devdg-dw02 devdg-dw online
emcpower3s2 auto:sliced devdg-tle01 devdg-tle online
emcpower4s2 auto:sliced dbadg-erp01 dbadg-erp online
emcpower6s2 auto:none – – online invalid

You should now see new devices in /dev/vx/dmp and /dev/vx/rdmp. What’s next? Well, that’s entirely up to you. Now that you’ve got the right drive in Veritas, you can set it up anyway you want – add it to an existing disk group, mirror a disk group for some added redundancy, create a new disk group, etc. In upcoming posts, I’ll detail how to create a new disk group with this LUN and then start digging into dynamically resizing file systems while everything is still online and available to the users.





Please VOTE for this page at: ADD TO DEL.ICIO.US | ADD TO DIGG | ADD TO FURL | ADD TO NEWSVINE | ADD TO NETSCAPE | ADD TO REDDIT | ADD TO STUMBLEUPON | ADD TO TECHNORATI FAVORITES | ADD TO SQUIDOO | ADD TO WINDOWS LIVE | ADD TO YAHOO MYWEB | ADD TO ASK | ADD TO GOOGLE


3 Comments


  1. […] of a Utah System Administrator HomeAbout « Adding a new LUN on Solaris 9 with VXFS and PowerPathCreate new Veritas Disk […]

    Posted May 24, 2009, 10:48 pm

  2. Hi kevin,

    Thanks a lot for this invaluable piece of information.
    I’m such a bad administrator who let Veritas to discover the new LUN before PowerPath.
    Have you already posted how to cleanup this situation?
    My and your environment are almost identical.
    Any help would be greatly appreciated.

    Regards

    Posted June 23, 2009, 12:26 pm

  3. I haven’t posted about that yet. I want to go through the process again before posting. Here are my notes on the topic though:

    ---
    vxdisk -e list
    vxdisk rm device-name
    ?vi /etc/vx/disk.info? - keeps track of what os native name is using
    rm -f /dev/vx/dmp/device*
    rm -f /dev/vx/rdmp/device*
    reboot
    ---

    Let me know if that works for you or if you had to do something else.

    Personally, to make sure that it is all cleaned up, I would remove everything from the OS following the above notes and then before rebooting, disconnect the LUN from the host (from the SAN administration side.) After the reboot make sure everything is clean – when you run “vxdctl enable” with the new LUN disconnected, it shouldn’t find anything. If that’s clean, then you can just start over from the beginning.

    Posted June 23, 2009, 1:49 pm

Leave a reply