rageek

A place for Unix Thoughts and Ideas

Category Archives: Veritas

Configuring ODM devices in a Solaris 11 Zone

I was happy to see that Symantec supplied a IPS repository for Storage Foundation 6.0pr1.

I was disappointed to see that the documentation for installed and enable Storage foundation for Solaris 11 zones was incomplete and didn’t work.

After digging through the documentation and performing a little troubleshooting, here is the procedure for installing and enable ODM support for Solaris 11 Zones.

1. The first step is to add the IPS repository as a publisher &  install the packages, then unset the publisher

root@testzone: # ls
VRTSpkgs.p5p  info
pkg set-publisher -P -g `pwd`/VRTSpkgs.p5p Symantec
pkg install --accept   VRTSvlic VRTSodm VRTSperl

If you are using the Zone with VCS, you can also install the 3 VCS packages specified in the install docs

pkg install --accept   VRTSvcs VRTSvcsag VRTSvcsea

Unset the publisher

pkg unset-publisher Symantec

2. Now we will update the zone configuration to add the lofs mount for the veritas license files, the odm device mapping and then the adding permission to the zone to make a odm mount. You will want to reboot the zone after this step.
Read more of this post

Advertisements

Mapping OS Devices to Veritas Device names in Veritas 5.1

With Veritas 5.1, if you are using native DMP, enclosure based naming is mandatory.

Now this is good as 5.1 has enhancements where the device name now includes the LDEV number for the array and it makes it easy for storage grows.

But this can kind of be a PITA if you are building fresh or you don’t have the LDEV number and are just trying to find the veritas device name for a set of luns.

This is a expansion of a 1 liner I posted awhile back, it has been expanded to include the Veritas Device name in the output.

Here it is and you can copy/paste it into a bash session. You can change the order and padding by modifying the awk statement

for c in $(iostat -En | grep Soft | awk '{print $1}' | cut -d t -f1 | sort | uniq); do 
for i in `iostat -En | grep Soft | awk '{print $1}' | grep "$c"`;do 
vxdisk list $i &>/dev/null || continue
DEV=`vxdisk list $i | grep Device | awk '{print $2}'`
SZ=$(iostat -En $i | grep Size | cut -d'<' -f2)
echo "$i ${SZ%% *} $DEV" | awk '{printf ( "%s\t%s %4d GB (%d MB)\n", $1, $3, $2/1024/1024/1024+.05, $2/1024/1024+.05) }'
done | sort -t d +1 -n; done

This is the output of vxdisk list

Read more of this post

Adding Storage Foundation CFS mount points via the Command line.

For the longest time I used vea for my CFS operations because it saved me time with updating the main.cf etc…

Then i figured out that VEA has command line utilities that it calls to do all of its dirty work (check out /var/vx/isis/command.log) and when it comes to adding Cluster Filesystems, it is cfsmntadm.

Here is quick instructions on how to use it..

In this example I’m adding a new shared diskgroup with a single mount point.

Here are its command line options.

root@testnode1 # cfsmntadm
  Error: V-35-1: cfsmntadm: Incorrect usage
  Usage:
       cfsmntadm add
               [service_group_name]  ...
       cfsmntadm add
              [service_group_name] all=[mount_options] [node_name ...]
       cfsmntadm add ckpt
                all=[mount_options] [node_name ...]
       cfsmntadm add snapshot

       cfsmntadm add snapshot dev=
               =[mount_options]
       cfsmntadm delete [-f]
       cfsmntadm modify  =[mount_options]
       cfsmntadm modify  +=
       cfsmntadm modify  -=
       cfsmntadm modify  all=[mount_options]
       cfsmntadm modify  all+=
       cfsmntadm modify  all-=
       cfsmntadm modify  add  ...
       cfsmntadm modify  delete  ...
       cfsmntadm modify  vol
       cfsmntadm display [-v] { mount_point | node_name }
       cfsmntadm setpolicy  [node_name ...]

Read more of this post

Managing VCS zone dependencies in Veritas 5.1

I have been provisioning all my new servers with VCS 5.1sp1 and somewhere between 5.0mp4 and 5.1sp1 they changed the Zones Agent in some fundamental ways. Previously, zones were defined as a resource in the group and could be linked to other resources such as proxy/group/mount resources.

In 5.1, there is a zone resource, but the definition is handled via the ContainerInfo property on the Service group:

5.0 Config:

group ems_zones (
        SystemList = { testnode01 = 0, testnode02 = 1 }
        Parallel = 1
        AutoStartList = { testnode01, testnode02 }

        Zone ems_zones_01_02 (
                Critical = 0
                ZoneName @testnode02 = testzn_ems-02
                ZoneName @testnode01 = testzn_ems-01
                )

        requires group cvm_mounts online local firm

5.1 Config:

group ems_zones (
        SystemList = { testnode02 = 1, testnode01 = 0 }
        ContainerInfo @testnode02 = { Name = testzn_ems-02, Type = Zone, Enabled = 1 }
        ContainerInfo @testnode01 = { Name = testzn_ems-01, Type = Zone, Enabled = 1 }
        Parallel = 1
        AutoStartList = { testnode02, testnode01 }
        )

        Zone ems_zones (
                Critical = 0
                )

        requires group cvm_mounts online local firm

Despite there still being a zone resource and there being support for resource dependencies, resource dependencies only work for dependencies that require the zone resources.

Read more of this post

Growing a Veritas Filesystem via the command line.

In my career I have gone from building volumes from the bottom up approach, to using vxassist, to VEA (with CFS cluster) and back to command line. My Symantec Reps have been raving about a new management console to replace VEA, but I’m leery of new software that comes with warnings about triggering kernel panics on existing older CFS clusters.

In the last year I have switched back to using the command line almost exclusively and I’m now going to illustrate the easiest and quickest way to add luns and grow filesystems.

In this example I’m going to be using lun 7 and 8 to grow my filesystem by 300GB. Although this resize can be done in one step, I’m splitting it out into two commands for illustration purposes.

First thing I’m going to check is to make sure that there is space left over in the filesystem for the new inodes.

If the filesystem is 100% full with no space left, do not proceed as if you attempt the resize, the operation will likely freeze and you’ll need to reboot to be able to complete the grow. If you are close to 100%, but have some space left over, you can try growing slowly, in chunks of MBs until you comfortably have enough free space to grow the volume.

I had read somewhere that this behavior should be gone by now, but members of my team still encounter it on occasion.

root@testserver # df -h /zones/.zonemounts/testzone-01/niqa3
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/blddbdg/niqa3
                       200G   5.0G   183G     3%    /zones/.zonemounts/testzone-01/niqa3

In this case I have plenty of space so I’m going to proceed.

Read more of this post

Seeing a summary of I/O Throughput on a Solaris server

I recently migrated 16TB of storage between 2 systems & arrays using parallel copies with star.

As part of this, I wanted to know my total I/O bandwidth so I could tell when I hit the optimal number of parallel copy jobs and to estimate a completion time (btw the optimal number was 10 jobs).

Here is a simple way of seeing the IO throughput of the fibre/sas/scsi controllers on Solaris.

iostat -xCnM 5 | egrep '%|c.?$'
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    1.0    1.0    0.0    0.0  0.0  0.0    0.1    7.1   0   1 c0
  410.8  178.3   62.5    8.8 13.6  4.5   23.2    7.7   0 125 c1
  410.8  178.5   62.5    8.8 13.6  4.5   23.2    7.7   0 125 c3
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.4    0.0    0.0  0.0  0.0    0.0    5.7   0   0 c0
  692.9  227.1  151.0   20.4  0.0  8.1    0.0    8.8   0 395 c1
  678.1  253.9  151.9   21.3  0.0  8.1    0.0    8.7   0 377 c3
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0
  816.3  274.4  164.3   24.0  0.0  7.9    0.0    7.2   0 364 c1
  830.7  280.0  169.9   23.8  0.0  8.5    0.0    7.6   0 378 c3
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    6.6    0.0    0.1    0.0  0.0  0.0    0.0    4.2   0   2 c0
  785.8  273.8  153.1   24.4  0.2  7.6    0.2    7.2   0 355 c1
  832.0  260.8  168.4   21.8  0.1  8.1    0.1    7.5   0 377 c3

If all you storage is configured in Veritas, you can use vxstat to get a complete summary of reads and writes.

You can modify the vxstat commands to have it only show throughput for a specific disk group.

INT=5; while /bin/true; do CNT=$((`vxstat -o alldgs |wc -l ` + 2)); vxstat -o alldgs -i$INT -c2 | tail +$CNT | nawk -v secs=$INT ‘BEGIN{ writes=0; reads=0}{writes+=$6;reads +=$5} END {printf (“%.2f MB/s Reads %.2f MB/s Writes\n”, reads/2/1024/secs, writes/2/1024/secs) }’; done
46.58 MB/s Reads 334.42 MB/s Writes
47.25 MB/s Reads 320.77 MB/s Writes
47.55 MB/s Reads 340.67 MB/s Writes
45.85 MB/s Reads 498.19 MB/s Writes
52.51 MB/s Reads 478.23 MB/s Writes
42.32 MB/s Reads 465.49 MB/s Writes
31.30 MB/s Reads 439.65 MB/s Writes

every once in a while it will get a blip like
15937437.54 MB/S Reads 9646673.26 MB/S Writes

That is obviously wrong, not sure why I get it, but it can be ignored or filtered out by piping the output through

perl -ne ‘split; print if (!(@_[0] > 100000) || !(@_[3] > 100000))’

Putting it together:

INT=5; while /bin/true; do CNT=$((`vxstat -o alldgs |wc -l ` + 2)); vxstat -o alldgs -i$INT -c2 | tail +$CNT | nawk -v secs=$INT ‘BEGIN{ writes=0; reads=0}{writes+=$6;reads +=$5} END {printf (“%.2f MB/s Reads %.2f MB/s Writes\n”, reads/2/1024/secs, writes/2/1024/secs) }’; done | perl -ne ‘split; print if (!(@_[0] > 100000) || !(@_[3] > 100000))’

You could also tweak this a little bit for use with SNMPD for graphing.

Updated 2/9/11: Reads and Writes were swapped in text output.
Updated 2/6/12: Didn’t realize the iostat one-liner pasted was flat-out wrong and didn’t work.

Configuring ZFS using Native DMP on Veritas Storage Foundation 5.1

Veritas Storage Foundation 5.1 supported a new feature called Native DMP which provides native OS devices that are multipathed via the DMP driver. This is very useful if you have a need for a raw device for ASM or for a ZFS pool. Previously, you could only run ZFS or ASM on top of DMP if you used a veritas volume for its storage.

Alternatively you could simply just enable MPXIO and have it handle all multipathing. However, I believe that DMP does a better job with multipathing on my database servers.

Unfortunately, Native DMP is not supported on SF Basic edition. Which is shame as I originally wanted to use this in my Tibco and OBIEE environments where I needed san connectivity for my zones, but I didn’t have the huge IO requirements of the databases.

Here is a example of of how to enable it

root@testserver # vxdisk -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR        
disk_0       auto:ZFS       -            -           ZFS                  c0t3d0s2         -            
disk_1       auto:ZFS       -            -           ZFS                  c0t1d0s2         -            
hitachi_usp-v0_04b0 auto:none      -            -           online invalid       c2t50060E8005477215d0s2 hdprclm fc   
hitachi_usp-v0_04d4 auto:none      -            -           online invalid       c2t50060E8005477215d1s2 hdprclm fc   
hitachi_usp-v0_04d5 auto:none      -            -           online invalid       c2t50060E8005477215d2s2 hdprclm fc   
hitachi_usp-v0_04d6 auto:none      -            -           online invalid       c2t50060E8005477215d3s2 hdprclm fc   

root@testserver # vxdmpadm settune dmp_native_support=on

root@testserver # zpool create zonepool hitachi_usp-v0_04b0

root@testserver # zpool status zonepool
  pool: zonepool
 state: ONLINE
 scan: none requested
config:

        NAME                     STATE     READ WRITE CKSUM
        zonepool                 ONLINE       0     0     0
          hitachi_usp-v0_04b0s0  ONLINE       0     0     0

errors: No known data errors

root@testserver # vxdisk scandisks
root@testserver # vxdisk -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR        
disk_0       auto:ZFS       -            -           ZFS                  c0t3d0s2         -            
disk_1       auto:ZFS       -            -           ZFS                  c0t1d0s2         -            
hitachi_usp-v0_04b0 auto:ZFS       -            -           ZFS                  c2t50060E8005477215d0 hdprclm fc   
hitachi_usp-v0_04d4 auto:none      -            -           online invalid       c2t50060E8005477215d1s2 hdprclm fc   
hitachi_usp-v0_04d5 auto:none      -            -           online invalid       c2t50060E8005477215d2s2 hdprclm fc   
hitachi_usp-v0_04d6 auto:none      -            -           online invalid       c2t50060E8005477215d3s2 hdprclm fc

If you are planning to reuse disks/Luns that were previously in Veritas and are not shown in vxdisk list as invalid or for ZFS, the dmp devices will not have been created for the disks yet and the create will fail. In this case you can dd the drive label and relabel and then run a vxdisk scandisks and then the zpool creation will succeed.


			

Storage Foundation 5.1 Sp1 issues with Local zones

When Storage Foundation 5.1 Sp1 was released last year, I finally decided that it was time to start upgrading my servers to the new version.

I immediately found that the software was having issues with ODM.

The issues were different depending if it was a upgrade or a fresh install.

For fresh installs of 5.1, the vxodm service will fail to start:

root@testzone-01 # svcs -xv
svc:/system/vxodm:default (VERITAS Oracle Disk Manager)
State: offline since Fri Jan 27 13:27:52 2012
Reason: Dependency svc:/system/vxfs/vxfsldlic is absent.
See: http://sun.com/msg/SMF-8000-E2
See: man -M /opt/VRTS/man -s 1 mount_odm
Impact: This service is not running.

This is due to vxfsldlic missing in the local zone.

For upgrades, if you do a upgrade on attach, you will see the following error in the update_log for the installation of VRTSodm

===== VRTSodm ====
/var/tmp//installcgaGlC/checkinstallegaGlC: /tmp/sh144290: cannot create
pkgadd: ERROR: checkinstall script did not complete successfully

Installation of on zone failed.
No changes were made to the system.

To Fix this you have to remove and reinstall VRTSodm and the apply its patch from the Global zone. This is inaddtion to the fixes for vxfsldic.

For Fresh installs, here is a script I wrote for fixing the zone configuration. The script will copy the vxfsldlic manifest to the zone, import and then enable it.
Run this with your zones attached and booted. You will probably need to run this for any zones you create in the future.

Read more of this post

Updating HBA Firmware post patch 145098-02

I recently noticed the following Messages appearing in my message log.

Jan 22 11:07:19 test01.testdomain.com emlxs: [ID 349649 kern.info] [ 1.0340]emlxs0:WARNING:1540: Firmware update required. (A manual HBA reset or link reset (using luxadm or fcadm) is required.)
Jan 22 11:07:21 test01.testdomain.com emlxs: [ID 349649 kern.info] [ 1.0340]emlxs1:WARNING:1540: Firmware update required. (A manual HBA reset or link reset (using luxadm or fcadm) is required.)

After some quick research it appears that the after patch 145098-02, the emlxs driver switched from automatically upgrading the firmware of the hba’s to making it a manual process.

It can be easily upgraded through the following process.

1. Determine device paths
root@test01 # luxadm -e port
/devices/pci@0,600000/pci@0/pci@8/SUNW,emlxs@0/fp@0,0:devctl CONNECTED
/devices/pci@1,700000/pci@0/pci@0/SUNW,emlxs@0/fp@0,0:devctl CONNECTED

2. Disable path in DMP (if applicable)

Determine the controller number
root@test01 # vxdmpadm getctlr
LNAME PNAME VENDOR CTLR-ID
=============================================================================================
c2 /pci@1,700000/pci@0/pci@0/SUNW,emlxs@0/fp@0,0 Emulex 10:00:00:00:c9:8c:0f:a1
c1 /pci@0,600000/pci@0/pci@8/SUNW,emlxs@0/fp@0,0 Emulex 10:00:00:00:c9:8c:5c:74

Disable Path
root@test01 # vxdmpadm disable ctlr=c2

this can be seen in messages file as:
Jan 23 13:37:09 test01.testdomain.com vxdmp: [ID 575547 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 disabled controller /pci@1,700000/pci@0/pci@0/SUNW,emlxs@0/fp@0,0 connected to disk array 18290

3. Reset hba to upgrade firmware
root@test01 # luxadm -e forcelip /devices/pci@1,700000/pci@0/pci@0/SUNW,emlxs@0/fp@0,0:devctl

Jan 23 13:39:34 test01.testdomain.com emlxs: [ID 349649 kern.info] [ 5.0334]emlxs1: NOTICE: 710: Link down.
Jan 23 13:39:38 test01.testdomain.com emlxs: [ID 349649 kern.info] [13.02C0]emlxs1: NOTICE: 200: Adapter initialization. (Firmware update needed. Updating. id=24 fw=4)
Jan 23 13:39:38 test01.testdomain.com emlxs: [ID 349649 kern.info] [ 3.0ECB]emlxs1: NOTICE:1520: Firmware download. (AWC file: KERN: old=1.20a9 new=1.21a0 Update.)
Jan 23 13:39:38 test01.testdomain.com emlxs: [ID 349649 kern.info] [ 3.0EEB]emlxs1: NOTICE:1520: Firmware download. (DWC file: TEST: new=1.02a3 Update.)
Jan 23 13:39:38 test01.testdomain.com emlxs: [ID 349649 kern.info] [ 3.0EFF]emlxs1: NOTICE:1520: Firmware download. (DWC file: STUB: old=2.80a4 new=2.82a4 Update.)
Jan 23 13:39:38 test01.testdomain.com emlxs: [ID 349649 kern.info] [ 3.0F0E]emlxs1: NOTICE:1520: Firmware download. (DWC file: SLI1: old=2.80a4 new=2.82a3 Update.)
Jan 23 13:39:38 test01.testdomain.com emlxs: [ID 349649 kern.info] [ 3.0F1D]emlxs1: NOTICE:1520: Firmware download. (DWC file: SLI2: old=2.80a4 new=2.82a4 Update.)
Jan 23 13:39:38 test01.testdomain.com emlxs: [ID 349649 kern.info] [ 3.0F2C]emlxs1: NOTICE:1520: Firmware download. (DWC file: SLI3: old=2.80a4 new=2.82a4 Update.)
Jan 23 13:39:54 test01.testdomain.com emlxs: [ID 349649 kern.info] [ 3.0143]emlxs1: NOTICE:1521: Firmware download complete. (Status good.)
Jan 23 13:39:59 test01.testdomain.com emlxs: [ID 349649 kern.info] [ 5.055E]emlxs1: NOTICE: 720: Link up. (2Gb, fabric, initiator)

4. re-enable path in DMP (if applicable)
root@test01 # vxdmpadm enable ctlr=c2
Jan 23 13:41:44 test01.testdomain.com vxdmp: [ID 575547 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 enabled controller /pci@1,700000/pci@0/pci@0/SUNW,emlxs@0/fp@0,0 connected to disk array 18290

5. Double check that all paths recovered in dmp.

root@test01 # vxdmpadm getdmpnode
NAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME
==============================================================================
hitachi_usp-v0_1063 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1064 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1065 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1066 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1067 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1068 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1069 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0

6. Repeat steps 4-5 on the remaining HBA
root@test01 # vxdmpadm disable ctlr=c1
root@test01 # luxadm -e forcelip /devices/pci@0,600000/pci@0/pci@8/SUNW,emlxs@0/fp@0,0:devctl
root@test01 # vxdmpadm enable ctlr=c1

root@test01 # vxdmpadm getdmpnode
NAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME
==============================================================================
hitachi_usp-v0_1063 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1064 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1065 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1066 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1067 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1068 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0
hitachi_usp-v0_1069 ENABLED Hitachi_USP-V 2 2 0 hitachi_usp-v0

Using VEA with Veritas 5.1

Although I use the GUI less and less these days, I occasionally will use it for quick provisioning of cfs storage.

In version 5.1 Symantec stopped shipping VEA, instead guiding users towards their SFM Console.

I would be interested, but was turned off after I found out there was the possibility of system outages due to a bug it triggers in cfs. There are patched versions, but apparently this is present in the client that is in the 5.0MP3 versions, which Is have widely deployed.

By default, the vxsvc daemons are disabled, to enable them run:
svcadm enable vxsvc

The Gui itself can be downloaded at:
http://www.symantec.com/business/products/otherresources.jsp?pcid=pcat_storage&pvid=1621_1

Click on
“Veritas Cluster Server Java Console, Veritas Cluster Server Simulator, Veritas Enterprise Administrator Console”

After logging in, you will come to a page with the clients for Cluster Manger, Simulator and VEA for the various platforms.

Note: you may need to click on that link again, after logging in.