rageek

A place for Unix Thoughts and Ideas

Growing a Veritas Filesystem via the command line.

In my career I have gone from building volumes from the bottom up approach, to using vxassist, to VEA (with CFS cluster) and back to command line. My Symantec Reps have been raving about a new management console to replace VEA, but I’m leery of new software that comes with warnings about triggering kernel panics on existing older CFS clusters.

In the last year I have switched back to using the command line almost exclusively and I’m now going to illustrate the easiest and quickest way to add luns and grow filesystems.

In this example I’m going to be using lun 7 and 8 to grow my filesystem by 300GB. Although this resize can be done in one step, I’m splitting it out into two commands for illustration purposes.

First thing I’m going to check is to make sure that there is space left over in the filesystem for the new inodes.

If the filesystem is 100% full with no space left, do not proceed as if you attempt the resize, the operation will likely freeze and you’ll need to reboot to be able to complete the grow. If you are close to 100%, but have some space left over, you can try growing slowly, in chunks of MBs until you comfortably have enough free space to grow the volume.

I had read somewhere that this behavior should be gone by now, but members of my team still encounter it on occasion.

root@testserver # df -h /zones/.zonemounts/testzone-01/niqa3
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/blddbdg/niqa3
                       200G   5.0G   183G     3%    /zones/.zonemounts/testzone-01/niqa3

In this case I have plenty of space so I’m going to proceed.

I’m now going to scan the fibre channel adapters for the new luns and label them and initialize them for Veritas

root@testserver # cfgadm | grep fc-fabric | awk '{print $1}' | xargs -I {} cfgadm -c configure {};vxdiskconfig
  VxVM  INFO V-5-2-1401 This command may take a few minutes to complete execution
  Executing Solaris command: devfsadm (part 1 of 2) at 12:19:34 PDT
  Executing VxVM command: vxdctl enable (part 2 of 2) at 12:19:35 PDT
  Command completed at 12:19:35 PDT
root@testserver # echo label > /tmp/cmd
root@testserver # format -f /tmp/cmd -d c2t50060E8005477215d7
root@testserver # format -f /tmp/cmd -d c2t50060E8005477215d8
root@testserver # vxdisk scandisks
root@testserver # vxdisk -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
hitachi_usp-v0_04b0 auto:ZFS       -            -           ZFS                  c2t50060E8005477215d0 hdprclm fc
hitachi_usp-v0_04d4 auto:cdsdisk   oraexpdg01  oraexpdg    online thinrclm      c2t50060E8005477215d1s2 hdprclm fc
hitachi_usp-v0_04d5 auto:cdsdisk   blddbdg01   blddbdg    online thinrclm      c2t50060E8005477215d2s2 hdprclm fc
hitachi_usp-v0_04d6 auto:cdsdisk   blddbdg02   blddbdg    online thinrclm      c2t50060E8005477215d3s2 hdprclm fc
hitachi_usp-v0_311e auto:cdsdisk   blddbdg04   blddbdg    online thinrclm      c2t50060E8005477215d6s2 hdprclm fc
hitachi_usp-v0_0428 auto:cdsdisk   oraexpdg02  oraexpdg    online thinrclm      c2t50060E8005477215d4s2 hdprclm fc
hitachi_usp-v0_0429 auto:cdsdisk   blddbdg03   blddbdg    online thinrclm      c2t50060E8005477215d5s2 hdprclm fc
hitachi_usp-v0_3125 auto:none      -            -           online invalid      c2t50060E8005477215d7s2 hdprclm fc
hitachi_usp-v0_3126 auto:none      -            -           online invalid      c2t50060E8005477215d8s2 hdprclm fc

root@testserver # /etc/vx/bin/vxdisksetup -i hitachi_usp-v0_3125
root@testserver # /etc/vx/bin/vxdisksetup -i hitachi_usp-v0_3126

root@testserver # vxdisk -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
hitachi_usp-v0_04b0 auto:ZFS       -            -           ZFS                  c2t50060E8005477215d0 hdprclm fc
hitachi_usp-v0_04d4 auto:cdsdisk   oraexpdg01  oraexpdg    online thinrclm      c2t50060E8005477215d1s2 hdprclm fc
hitachi_usp-v0_04d5 auto:cdsdisk   blddbdg01   blddbdg    online thinrclm      c2t50060E8005477215d2s2 hdprclm fc
hitachi_usp-v0_04d6 auto:cdsdisk   blddbdg02   blddbdg    online thinrclm      c2t50060E8005477215d3s2 hdprclm fc
hitachi_usp-v0_311e auto:cdsdisk   blddbdg04   blddbdg    online thinrclm      c2t50060E8005477215d6s2 hdprclm fc
hitachi_usp-v0_0428 auto:cdsdisk   oraexpdg02  oraexpdg    online thinrclm      c2t50060E8005477215d4s2 hdprclm fc
hitachi_usp-v0_0429 auto:cdsdisk   blddbdg03   blddbdg    online thinrclm      c2t50060E8005477215d5s2 hdprclm fc
hitachi_usp-v0_3125 auto:cdsdisk      -            -           online thinrclm      c2t50060E8005477215d7s2 hdprclm fc
hitachi_usp-v0_3126 auto:cdsdisk      -            -           online thinrclm      c2t50060E8005477215d8s2 hdprclm fc

If you are growing storage on a cfs cluster, you’ll want to go to the other node(s) and discover the storage there.  This can be handled with a

cfgadm | grep fc-fabric | awk '{print $1}' | xargs -I {} cfgadm -c configure {};vxdiskconfig;vxdisk list
Now that the disks are labled and initialized, I’m going to add them to the Diskgroup.
If this is on a cfs cluster, this will need to be done on the master node (this can be checked with vxdctl -c mode)
root@testserver # vxdg -g blddbdg adddisk blddbdg05=hitachi_usp-v0_3125
root@testserver # vxdg -g blddbdg adddisk blddbdg06=hitachi_usp-v0_3126

Using vxdg free to get the amount of Veritas block available. Note that these are 512byte blocks. So if you divide the length by 2, you’ll get the size in KBs, divide by 1024 for MBs

root@testserver # vxdg -g blddbdg free
DISK         DEVICE       TAG          OFFSET    LENGTH    FLAGS
blddbdg01   hitachi_usp-v0_04d5 hitachi_usp-v0_04d5 209627136 1792      -
blddbdg02   hitachi_usp-v0_04d6 hitachi_usp-v0_04d6 209627136 1792      -
blddbdg03   hitachi_usp-v0_0429 hitachi_usp-v0_0429 419346432 256       -
blddbdg04   hitachi_usp-v0_311e hitachi_usp-v0_311e 419346432 256       -
blddbdg05   hitachi_usp-v0_3125 hitachi_usp-v0_3125 0         209628928 -
blddbdg06   hitachi_usp-v0_3126 hitachi_usp-v0_3126 0         419346688 -

Doing the math 419346688/2 = 209673344, 209673344/1024 = 204759m

I’m now going to run vxresize to grow the volume and the filesystem and tell it to grab the space from blddbdg06. You can omit the disk name if you are pooling the storage on your server and don’t care where it is being allocated from.

root@testserver # /etc/vx/bin/vxresize -g blddbdg -b niqa3 +204759m blddbdg06
root@testserver # vxdg -g blddbdg free
 DISK         DEVICE       TAG          OFFSET    LENGTH    FLAGS
  blddbdg01   hitachi_usp-v0_04d5 hitachi_usp-v0_04d5 209627136 1792      -
  blddbdg02   hitachi_usp-v0_04d6 hitachi_usp-v0_04d6 209627136 1792      -
  blddbdg03   hitachi_usp-v0_0429 hitachi_usp-v0_0429 419346432 256       -
  blddbdg04   hitachi_usp-v0_311e hitachi_usp-v0_311e 419346432 256       -
  blddbdg05   hitachi_usp-v0_3125 hitachi_usp-v0_3125 0         209628928 -
  blddbdg06   hitachi_usp-v0_3126 hitachi_usp-v0_3126 419346432 256       -

The remaining space on blddbdg06 is now negligitable and it is time to add the rest of the space.

209628928/2 = 104814464K, 104814464/1024 = 102357M

root@testserver # /etc/vx/bin/vxresize -g blddbdg -b niqa3 +102357M blddbdg05
root@testserver # df -h /zones/.zonemounts/testzone-01/niqa3
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/bldlnpdg/niqa3
                       500G   6.1G   463G     2%    /zones/.zonemounts/sanb1dblnp-01/niqa3

Here are 2 additional ways to specify the size

root@testserver # /etc/vx/bin/vxresize -g blddbdg -b niqa3 +104814464k blddbdg05
or
root@testserver # /etc/vx/bin/vxresize -g blddbdg -b niqa3 +$((209628928/2))k blddbdg05

I also could have done the grown in one command with:

/etc/vx/bin/vxresize -g blddbdg -b niqa3 +307117mb lddbdg05 blddbdg06

Note: I have seen Veritas reject the grow via the command line and the gui due to some unknown internal math error.

If that happens, reduce the grow amount by a couple MBs and try again

5/23/12 – Updated with additional ways to specify the grow size.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: