A place for Unix Thoughts and Ideas

Tag Archives: Veritas

Configuring ODM devices in a Solaris 11 Zone

I was happy to see that Symantec supplied a IPS repository for Storage Foundation 6.0pr1.

I was disappointed to see that the documentation for installed and enable Storage foundation for Solaris 11 zones was incomplete and didn’t work.

After digging through the documentation and performing a little troubleshooting, here is the procedure for installing and enable ODM support for Solaris 11 Zones.

1. The first step is to add the IPS repository as a publisher &  install the packages, then unset the publisher

root@testzone: # ls
VRTSpkgs.p5p  info
pkg set-publisher -P -g `pwd`/VRTSpkgs.p5p Symantec
pkg install --accept   VRTSvlic VRTSodm VRTSperl

If you are using the Zone with VCS, you can also install the 3 VCS packages specified in the install docs

pkg install --accept   VRTSvcs VRTSvcsag VRTSvcsea

Unset the publisher

pkg unset-publisher Symantec

2. Now we will update the zone configuration to add the lofs mount for the veritas license files, the odm device mapping and then the adding permission to the zone to make a odm mount. You will want to reboot the zone after this step.
Read more of this post


Mapping OS Devices to Veritas Device names in Veritas 5.1

With Veritas 5.1, if you are using native DMP, enclosure based naming is mandatory.

Now this is good as 5.1 has enhancements where the device name now includes the LDEV number for the array and it makes it easy for storage grows.

But this can kind of be a PITA if you are building fresh or you don’t have the LDEV number and are just trying to find the veritas device name for a set of luns.

This is a expansion of a 1 liner I posted awhile back, it has been expanded to include the Veritas Device name in the output.

Here it is and you can copy/paste it into a bash session. You can change the order and padding by modifying the awk statement

for c in $(iostat -En | grep Soft | awk '{print $1}' | cut -d t -f1 | sort | uniq); do 
for i in `iostat -En | grep Soft | awk '{print $1}' | grep "$c"`;do 
vxdisk list $i &>/dev/null || continue
DEV=`vxdisk list $i | grep Device | awk '{print $2}'`
SZ=$(iostat -En $i | grep Size | cut -d'<' -f2)
echo "$i ${SZ%% *} $DEV" | awk '{printf ( "%s\t%s %4d GB (%d MB)\n", $1, $3, $2/1024/1024/1024+.05, $2/1024/1024+.05) }'
done | sort -t d +1 -n; done

This is the output of vxdisk list

Read more of this post

Managing VCS zone dependencies in Veritas 5.1

I have been provisioning all my new servers with VCS 5.1sp1 and somewhere between 5.0mp4 and 5.1sp1 they changed the Zones Agent in some fundamental ways. Previously, zones were defined as a resource in the group and could be linked to other resources such as proxy/group/mount resources.

In 5.1, there is a zone resource, but the definition is handled via the ContainerInfo property on the Service group:

5.0 Config:

group ems_zones (
        SystemList = { testnode01 = 0, testnode02 = 1 }
        Parallel = 1
        AutoStartList = { testnode01, testnode02 }

        Zone ems_zones_01_02 (
                Critical = 0
                ZoneName @testnode02 = testzn_ems-02
                ZoneName @testnode01 = testzn_ems-01

        requires group cvm_mounts online local firm

5.1 Config:

group ems_zones (
        SystemList = { testnode02 = 1, testnode01 = 0 }
        ContainerInfo @testnode02 = { Name = testzn_ems-02, Type = Zone, Enabled = 1 }
        ContainerInfo @testnode01 = { Name = testzn_ems-01, Type = Zone, Enabled = 1 }
        Parallel = 1
        AutoStartList = { testnode02, testnode01 }

        Zone ems_zones (
                Critical = 0

        requires group cvm_mounts online local firm

Despite there still being a zone resource and there being support for resource dependencies, resource dependencies only work for dependencies that require the zone resources.

Read more of this post

Growing a Veritas Filesystem via the command line.

In my career I have gone from building volumes from the bottom up approach, to using vxassist, to VEA (with CFS cluster) and back to command line. My Symantec Reps have been raving about a new management console to replace VEA, but I’m leery of new software that comes with warnings about triggering kernel panics on existing older CFS clusters.

In the last year I have switched back to using the command line almost exclusively and I’m now going to illustrate the easiest and quickest way to add luns and grow filesystems.

In this example I’m going to be using lun 7 and 8 to grow my filesystem by 300GB. Although this resize can be done in one step, I’m splitting it out into two commands for illustration purposes.

First thing I’m going to check is to make sure that there is space left over in the filesystem for the new inodes.

If the filesystem is 100% full with no space left, do not proceed as if you attempt the resize, the operation will likely freeze and you’ll need to reboot to be able to complete the grow. If you are close to 100%, but have some space left over, you can try growing slowly, in chunks of MBs until you comfortably have enough free space to grow the volume.

I had read somewhere that this behavior should be gone by now, but members of my team still encounter it on occasion.

root@testserver # df -h /zones/.zonemounts/testzone-01/niqa3
Filesystem             size   used  avail capacity  Mounted on
                       200G   5.0G   183G     3%    /zones/.zonemounts/testzone-01/niqa3

In this case I have plenty of space so I’m going to proceed.

Read more of this post

Seeing a summary of I/O Throughput on a Solaris server

I recently migrated 16TB of storage between 2 systems & arrays using parallel copies with star.

As part of this, I wanted to know my total I/O bandwidth so I could tell when I hit the optimal number of parallel copy jobs and to estimate a completion time (btw the optimal number was 10 jobs).

Here is a simple way of seeing the IO throughput of the fibre/sas/scsi controllers on Solaris.

iostat -xCnM 5 | egrep '%|c.?$'
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    1.0    1.0    0.0    0.0  0.0  0.0    0.1    7.1   0   1 c0
  410.8  178.3   62.5    8.8 13.6  4.5   23.2    7.7   0 125 c1
  410.8  178.5   62.5    8.8 13.6  4.5   23.2    7.7   0 125 c3
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.4    0.0    0.0  0.0  0.0    0.0    5.7   0   0 c0
  692.9  227.1  151.0   20.4  0.0  8.1    0.0    8.8   0 395 c1
  678.1  253.9  151.9   21.3  0.0  8.1    0.0    8.7   0 377 c3
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0
  816.3  274.4  164.3   24.0  0.0  7.9    0.0    7.2   0 364 c1
  830.7  280.0  169.9   23.8  0.0  8.5    0.0    7.6   0 378 c3
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    6.6    0.0    0.1    0.0  0.0  0.0    0.0    4.2   0   2 c0
  785.8  273.8  153.1   24.4  0.2  7.6    0.2    7.2   0 355 c1
  832.0  260.8  168.4   21.8  0.1  8.1    0.1    7.5   0 377 c3

If all you storage is configured in Veritas, you can use vxstat to get a complete summary of reads and writes.

You can modify the vxstat commands to have it only show throughput for a specific disk group.

INT=5; while /bin/true; do CNT=$((`vxstat -o alldgs |wc -l ` + 2)); vxstat -o alldgs -i$INT -c2 | tail +$CNT | nawk -v secs=$INT ‘BEGIN{ writes=0; reads=0}{writes+=$6;reads +=$5} END {printf (“%.2f MB/s Reads %.2f MB/s Writes\n”, reads/2/1024/secs, writes/2/1024/secs) }’; done
46.58 MB/s Reads 334.42 MB/s Writes
47.25 MB/s Reads 320.77 MB/s Writes
47.55 MB/s Reads 340.67 MB/s Writes
45.85 MB/s Reads 498.19 MB/s Writes
52.51 MB/s Reads 478.23 MB/s Writes
42.32 MB/s Reads 465.49 MB/s Writes
31.30 MB/s Reads 439.65 MB/s Writes

every once in a while it will get a blip like
15937437.54 MB/S Reads 9646673.26 MB/S Writes

That is obviously wrong, not sure why I get it, but it can be ignored or filtered out by piping the output through

perl -ne ‘split; print if (!(@_[0] > 100000) || !(@_[3] > 100000))’

Putting it together:

INT=5; while /bin/true; do CNT=$((`vxstat -o alldgs |wc -l ` + 2)); vxstat -o alldgs -i$INT -c2 | tail +$CNT | nawk -v secs=$INT ‘BEGIN{ writes=0; reads=0}{writes+=$6;reads +=$5} END {printf (“%.2f MB/s Reads %.2f MB/s Writes\n”, reads/2/1024/secs, writes/2/1024/secs) }’; done | perl -ne ‘split; print if (!(@_[0] > 100000) || !(@_[3] > 100000))’

You could also tweak this a little bit for use with SNMPD for graphing.

Updated 2/9/11: Reads and Writes were swapped in text output.
Updated 2/6/12: Didn’t realize the iostat one-liner pasted was flat-out wrong and didn’t work.

Using VEA with Veritas 5.1

Although I use the GUI less and less these days, I occasionally will use it for quick provisioning of cfs storage.

In version 5.1 Symantec stopped shipping VEA, instead guiding users towards their SFM Console.

I would be interested, but was turned off after I found out there was the possibility of system outages due to a bug it triggers in cfs. There are patched versions, but apparently this is present in the client that is in the 5.0MP3 versions, which Is have widely deployed.

By default, the vxsvc daemons are disabled, to enable them run:
svcadm enable vxsvc

The Gui itself can be downloaded at:

Click on
“Veritas Cluster Server Java Console, Veritas Cluster Server Simulator, Veritas Enterprise Administrator Console”

After logging in, you will come to a page with the clients for Cluster Manger, Simulator and VEA for the various platforms.

Note: you may need to click on that link again, after logging in.

Wierd 11g RAC database instance failures

Since I’m currently writing about RAC clusters I thought I would mention this.

A long time ago when I was implementing 11gR1 for the first time, I ran into a very weird issue where the Cluster seemed to be working fine, but we could not get the database instance to start on more than 1 node of the cluster.

The errors were not very helpful and Oracle wasn’t any help and the configurations were identical.

We would start the instance on node 1, and when started on node 2, it would immediately go down.

Now we had 2 clusters, one in a local data center and one remote. The one in the local datacenter worked fine, the remote cluster did not.

It turned out that the remote cluster was mis-cabled.

Each set of private heartbeat links were on its own dedicated private VLAN, so everything “looked” ok from a VCS/CRS perspective.

However, each link was actually being run to separate switches and had to cross uplinks in-order to talk reach the other node (unsupported/incorrect configuration).

This caused latency and fragmentation of the packets that caused Oracle to terminate the secondary instance.

After rewiring so each set of private links where on the same switch (primaries to switch 1, secondaries to switch 2), everything worked perfectly.

Quickly add Luns to a Solaris host

Here is a 1 liner for having Solaris rescan all of its fibre cards for new luns and then have them detected by Veritas.

This only works if you are using the built in Leadville drivers (which you probably should be running anyway).

If you are not familiar with vxdiskconfig, it simply runs devfsadm, followed by vxdctl enable

cfgadm | grep fc-fabric | awk ‘{print $1}’ | xargs -I {} cfgadm -c configure {};vxdiskconfig;vxdisk list

I you are not using Veritas on your system, you could alternatively do
cfgadm | grep fc-fabric | awk ‘{print $1}’ | xargs -I {} cfgadm -c configure {}; devfsadm -v

BTW, if you are still using the lpfc drivers, I would strongly recommend moving to the Leadville drivers as they work more consistently for dynamic LUN addition.
I had been very hesitant of using the Leadville drivers, remembering back to when I used to work at HDS and they internally referred to it as the LeadWeight driver.

However, after experiencing multiple production outages when the lpfc driver would not properly detecting SAN path failure (such as a port failure or a target disappearing and then reappearing). We made the change and I have been much happier and our systems have been very reliable when dealing with transient SAN events ever since. Our performance has also been extremely good.

Quickly setting up disks for Veritas

Here is a quick trick for quickly labeling and initializing a group of disks for Veritas

Say you want to initialize luns 10-20 and they are all on c2t50060E800547725Dd* and force a smi label.

printf “label 0 y n n” > /tmp/cmd
for ((i=10;i<21;i++)); do format -d $i -f /tmp/cmd -e ;done
vxdisk scandiskss
for ((i=0;i<12;i++)); do /etc/vx/bin/vxdisksetup -i c2t50060E800547725Dd$i;done

Depending on how many unlabeled LUNs you have, this can take a long time.

Hide Local disks from Veritas

After receiving a frantic call one night when one of my colleagues initialized both disksuite mirror boot disks on one of our M5000s, I decided that it would be safer and much cleaner to mask all internal storage from Veritas.

Here is a quick way to do it.

vxdmpadm getctlr | grep scsi | awk ‘{print $1}’ |xargs -I {} vxdmpadm exclude vxvm ctlr={}

I don’t believe this works with versions prior to 5.0MP3 and you may want to run vxdmpadm getctlr | grep scsi by itself first to make sure it doesn’t accidentally disable any legacy direct attached scsi storage you may have.

BTW, if you have accidentally initialized your drives with Veritas but have not created any volumes, there is hope.

Veritas is very smart and if it detects or suspects there was data previously on the drive, it will write the private region to the end of the disk. For most that is where swap is located.

For us, all we had to do was recreate the label and apply with fmthard and we were able to bring the system back online.