A place for Unix Thoughts and Ideas

Category Archives: Solaris

Minding your ZFS pool and filesystem versions, and feature flags

After Code42 dropped Solaris support in the newest update of CrashPlan, I decided it was time to move my home server from Solaris to Linux.

Using OpenZFS on Linux, the plan was to use zfs send/recieve to migrate all my datasets from version 35 to version 28 pools while still on Solaris and then reboot my server onto a new Ubuntu based ZFS root.

This worked fine, except for a handful of filesystems that were created after going to Solaris 11.1, which by default created the ZFS filesystem as version 6.

If you are creating new pools on Solaris and want to retain the ability to migrate them to ZFS on Linux (or any of the OpenZFS platforms), you will want to stay at version 28 of the pool and version 5 of the filesystem. Also, avoid activating any of the feature flags on OpenZFS side if you want to have the ability to swing back to Solaris.

You will also need to watch the feature flags enabled on your new ZFS root pool as setting the wrong flag can instantly make it incompatible with grub and unable to boot your system. To recover you will have to boot off alternative media and create a new zfs pool for booting; which if you split all your filesystems in to different datasets, sounds worse than it actually is (support for additional features is already in GRUB’s Dev tree, so may not be a issue for long).

With OpenZFS, you’ll want to create the pool with the -d option to have all features turned off by default. Otherwise the default is to enable all features.

Example (pool creation):
zpool create testpool -o version=28 -O version=5 c3t0d0

Example (filesystem creation):
zfs create -o version=5 testpool/testdata

Example (OpenZFS pool creation):
zpool create testpool -d ata-ST4000DM000-1F2168_Z302HRLS-part1

Example (OpenZFS rpool Creation):
zpool create -f -d -o feature@async_destroy=enabled -o feature@empty_bpobj=enabled -o feature@lz4_compress=enabled -o ashift=12 -O compression=lz4 rpool mirror ${DISK1} ${DISK2}


Enabling Solaris 11.2 Kernel Zones whith VMWare Fusion

Here is a quick trick if you are looking to test out Solaris 11.2 Kernel Zones with Vmware Fusion.

I’m fairly sure that this trick will also work with the other VMWare products.

Out of the box, if you attempt to install a kernel-zone brand zone, you will see the following error:

# zoneadm -z myfirstkz install
Platform does not support the kernel-zone brand.
zoneadm: zone myfirstkz failed to verify

Running virtinfo confirms it:

root@s112test:/dev/lofi# virtinfo
NAME            CLASS     
vmware          current   
non-global-zone supported

1. First thing is that you need to be running i5/i7 generation processor that will support nested paging.
2. Shutdown your VM and then go into the container/folder for the VM and modify the .vmx file and add the following to the bottom:

vhv.enable = “TRUE”

3. Save the file and then restart your VM
4. Verify Support with virtinfo:

root@s112test:/dev/lofi# virtinfo
NAME            CLASS     
vmware          current   
non-global-zone supported
kernel-zone     supported

5. If you have accepted the default drive size of 16GB in VMWare Fusion, you will want to add an additional drive and create a new zfs pool for zone as the default rpool size is too small for a kernel zone and the zonecreate will abort during the zone creation.

Installation issues with LSI 9201 sas controller

As part of refreshing the storage on my home server which had a stack of 1TB WD GP drives which were starting to show their age in my weekly scrubs, I decided it was time to replace my LSI SAS3081 with a newer generation controller.

On eBay there is not shortage of LSI based 92xx controllers for around $80-100, so I picked up a LSI SAS9201-8i which is a IBM rebranded of a LSI 9211-8i.

Upon installing it into my system, the card’s bios initialized and showed all my drives, but the controller was not being see in Solaris 11.1.

I tested the card using a ubuntu rescue cd and confirmed the card was working, just not with Solaris 11. Since it was detected in ubuntu, I upgraded the firmware to fix the issue, but alas that didn’t work either.

Finally,  I decided I would boot from a Solaris 11.1 live CD, to eliminate my install as a issue; the card showed up immediately.

I grabbed all the prtconf -v info and driver files from etc and saved it off on a usb stick and then rebooted to continue testing.

On my install of Solaris 11.1, a grep of the prtconf output showed the OS could see the device, but it wasn’t attaching to it

pci1000,72 (driver not attached)
Hardware properties:
name=’assigned-addresses’ type=int items=15
name=’reg’ type=int items=20
name=’compatible’ type=string items=13
value=’pciex1000,72.1000.72.3′ + ‘pciex1000,72.1000.72’ + ‘pciex1000,72.3’ + ‘pciex1000,72’ + ‘pciexclass,010700’ + ‘pciexclass,0107’ + ‘pci1000,72.1000.72.3’ + ‘pci1000,72.1000.72’ + ‘pci1000,72’ + ‘pci1000,72.3’ + ‘pci1000,72’ + ‘pciclass,010700’ + ‘pciclass,0107′
name=’model’ type=string items=1
value=’Serial Attached SCSI Controller’

I then decided to use the add_drv command to try to link the pci information to the mpt_sas driver, figuring I had ran into a weird device driver issue, and then I got the biggest clue of all:

root@test_sys:/etc# add_drv -vi “pci1000,72@0” mpt_sas
Cannot find module (mpt_sas).

A search of /kernel/drv quickly identified that while there was a mpt drv, there was no mpt_sas driver module present.

After a quick pkg search

root@azurite:/kernel/drv/amd64# pkg search mpt_sas
driver_name driver mpt_sas pkg:/driver/storage/mpt_sas@0.5.11-
basename file kernel/drv/amd64/mpt_sas pkg:/driver/storage/mpt_sas@0.5.11-
basename file kernel/drv/sparcv9/mpt_sas pkg:/driver/storage/mpt_sas@0.5.11-
basename file kernel/drv/amd64/mpt_sas pkg:/driver/storage/mpt_sas@0.5.11-
basename file kernel/drv/sparcv9/mpt_sas pkg:/driver/storage/mpt_sas@0.5.11-
basename file kernel/kmdb/sparcv9/mpt_sas pkg:/developer/debug/mdb@0.5.11-
basename file kernel/kmdb/amd64/mpt_sas pkg:/developer/debug/mdb@0.5.11-
basename file kernel/kmdb/sparcv9/mpt_sas pkg:/developer/debug/mdb@0.5.11-
pkg.fmri set solaris/driver/storage/mpt_sas pkg:/driver/storage/mpt_sas@0.5.11-

After installing the driver package:

pkg install mpt_sas

The card was immediately bound to a driver and now works.

I guess the lesson here is that while just everything is installed with the base install of Solaris 10, make no assumptions of that on Solaris 11.

Programmatically determining the closest server

We recently updated our server naming standard to be location agnostic.

Due to this change, I had to work out a new mechanism to programmatically locate the nearest server for my imaging, build, and update scripts.

My end solution involves using ping to find the average ping time and comparing the average to determine the closest server.

In the case they are the same, it uses the first server.


case `uname -s` in
        FST=`ping  -vs $FIRSTSR 20 5 | awk -F/ '/^round|^rtt/{printf("%d\n",$6+.5)}'`
        SST=`ping  -vs $SECSR 20 5 | awk -F/ '/^round|^rtt/{printf("%d\n",$6+.5)}'`
        FST=`ping -s 20 -c 5 -v $FIRSTSR | awk -F/ '/^round|^rtt/{printf("%d\n",$6+.5)}'`
        SST=`ping -s 20 -c 5 -v $SECSR | awk -F/ '/^round|^rtt/{printf("%d\n",$6+.5)}'`

if [ $FST -le $SST ]; then
        echo "Using $FIRSTSR for nfs mount"
        echo "Using $SECSR for nfs mount"

Easily add log output to any shell script

I wrote this script a couple years ago for the purpose of being able to better track and save the output of my build scripts during the jumpstart process.

I was looking to create a easy way to have my script output to go to both the console and a log, without having to make extensive changes to all of my scripts.

The script works by creating new file descriptors and re-mapping stdout and stderr.

Here is a download link for log_include.sh

It can be added to any script by sourcing it at the top of the script.

if [ -f /admin/include_log.sh ]; then
    # Adds script logging output
    # run pre_exit prior to exiting/rebooting to close log and reset stdout/stderr
    . /admin/include_log.sh

By default this will write to the default log directory specified in log_include.sh.

When sourced in the file, it will immediately output to a default log directory with a log file named by the call script and date:

Saving output to /var/sadm/system/include_log/solaris_qa_082712_2318.log

Optionally, variables can be set in the calling script prior to sourcing to modify the logging behaviors:


_PIPEFILE – Specify name of Fifo file, defaults to /tmp/${Script basename}.pipe-_${date & time}
_CONSOLEOUT – Write output to Console in addition to stdout (no effect if running on console)
_CONSOLEDEV – Path to Console Character Device on System
_LOGFILENAME – The full path name of output log file
_LOGDIR – The directory to use for writing logs to, defaults to _DEFAULT_LOGDIR variable
_LOGFILEBASE – The base part of the filename to use, default is to use the calling script name

This should work in both Linux and Solaris

Prior to exiting your scripts, you will want to call the pre_exit function will close the log file and reset stdout/stderr

Configuring ODM devices in a Solaris 11 Zone

I was happy to see that Symantec supplied a IPS repository for Storage Foundation 6.0pr1.

I was disappointed to see that the documentation for installed and enable Storage foundation for Solaris 11 zones was incomplete and didn’t work.

After digging through the documentation and performing a little troubleshooting, here is the procedure for installing and enable ODM support for Solaris 11 Zones.

1. The first step is to add the IPS repository as a publisher &  install the packages, then unset the publisher

root@testzone: # ls
VRTSpkgs.p5p  info
pkg set-publisher -P -g `pwd`/VRTSpkgs.p5p Symantec
pkg install --accept   VRTSvlic VRTSodm VRTSperl

If you are using the Zone with VCS, you can also install the 3 VCS packages specified in the install docs

pkg install --accept   VRTSvcs VRTSvcsag VRTSvcsea

Unset the publisher

pkg unset-publisher Symantec

2. Now we will update the zone configuration to add the lofs mount for the veritas license files, the odm device mapping and then the adding permission to the zone to make a odm mount. You will want to reboot the zone after this step.
Read more of this post

Determining global zone name from inside a solaris 11 zone

The default security of Solaris zones masks the name of the host’s global zone.

In my environment we don’t have the requirement to mask the global zone name and it is very useful for our internal customers as wells as our engineering staff to have this info easily available.

There are a couple ways to go about this. I had 2 primary requirements for my solution. The information provided to the zone needed to be consistently accurate and up to date in the case that I migrate a zone to another hosts and I also do not want it to be obvious that this information is being provided to the zone.

One easy solution is to create a file on the global zone and loopback mount it into the zone.
A Very simple solution, but LOFS mounts appear in the zone’s df output and I think that is being a little too obvious.

Previously, I used a trick with the arp cache and probe-based IPMP to figure out the global zone.
But in Solaris 11, transitive probing and virtual nics have put an end to that.

My solution for the problem uses lofi and device mapping to make the information available to the zone.

Lofiadm is great tool which allows you to map files to devices, this is very handy for mounting iso’s or updating boot miniroots. The file can be arbitrary, but must be a multiple of 512bytes. This can be accomplished using the mkfile command and cat.

The first thing I do is generate the file in the global zone. I do by using the hostname command, and then creating another file to pad the original file to 512bytes.  Read more of this post

Mapping OS Devices to Veritas Device names in Veritas 5.1

With Veritas 5.1, if you are using native DMP, enclosure based naming is mandatory.

Now this is good as 5.1 has enhancements where the device name now includes the LDEV number for the array and it makes it easy for storage grows.

But this can kind of be a PITA if you are building fresh or you don’t have the LDEV number and are just trying to find the veritas device name for a set of luns.

This is a expansion of a 1 liner I posted awhile back, it has been expanded to include the Veritas Device name in the output.

Here it is and you can copy/paste it into a bash session. You can change the order and padding by modifying the awk statement

for c in $(iostat -En | grep Soft | awk '{print $1}' | cut -d t -f1 | sort | uniq); do 
for i in `iostat -En | grep Soft | awk '{print $1}' | grep "$c"`;do 
vxdisk list $i &>/dev/null || continue
DEV=`vxdisk list $i | grep Device | awk '{print $2}'`
SZ=$(iostat -En $i | grep Size | cut -d'<' -f2)
echo "$i ${SZ%% *} $DEV" | awk '{printf ( "%s\t%s %4d GB (%d MB)\n", $1, $3, $2/1024/1024/1024+.05, $2/1024/1024+.05) }'
done | sort -t d +1 -n; done

This is the output of vxdisk list

Read more of this post

Strickly Limiting ZFS arc cache size

On the majority of my servers I use ZFS just for the root filesystem and allowing the arc to grow uncheck is counterproductive for tracking server utilization and running some applications.

Consequently I severely limit the amount of memory used and set it at 100MB.

If your going to limit the arc cache, just about every ZFS tuning guide suggests capping the arc cache limit via zfs:zfs_arc_max

However, I was digging into the memory utilization of one of my Tibco servers and noticed that the ZFS arc cache was quite a bit larger than value specified in /etc/system

root@testserver # kstat zfs:0:arcstats:size | grep size | awk '{printf "%2dMB\n",  $2/1024/1024+0.5}'

I actually noticed this before in the past, but didn’t research any further due since it was insignificant compared to the free ram on that server.

I checked a couple other servers and noticed that it was consistently around 2GB for most of my servers.

root@testserver # grep zfs /etc/system
set zfs:zfs_arc_max = 104857600

In checking Kstat, I noticed a minimum parameter for zfs that I hadn’t noticed before that looked very similar to my arc size.

root@testserver # kstat -p zfs:0:arcstats | head -4
zfs:0:arcstats:c        2101237248
zfs:0:arcstats:class    misc
zfs:0:arcstats:c_max    104857600
zfs:0:arcstats:c_min    2101237248

Referring the Oracle Solaris Tunable guide, the zfs_arc_min parameter is set to 1/32nd of the physical memory or a minimum of 64MB. 2GB on a 64GB system, 4GB on 128GB one.

So I now include the maximum and minimum values in /etc/system and now the limits are occurring as predicted.

root@testserver # grep zfs_arc /etc/system
set zfs:zfs_arc_max = 104857600
set zfs:zfs_arc_min = 104857600

root@testserver # kstat -p zfs:0:arcstats | head -3
zfs:0:arcstats:c        104857600
zfs:0:arcstats:c_max    104857600
zfs:0:arcstats:c_min    104857600

root@testserver # kstat zfs:0:arcstats:size | grep size | awk '{printf "%2dMB\n",  $2/1024/1024+0.5}'

Now that I have realized that the majority of my servers have arc caches set to 1/32 of ram, I can take a good look at whether I should increase my intended defaults or leave them as is.

ILOM Quick Reference

Now that alom Mode has been eliminated with the T3/T4 platforms, ILOM mode is it going forward.

While I think alom is easier to remember (IMHO), it does now provide a interface that is consistent across sun platforms and i’m sure there are some other good reasons for going this direction.

Here is a quick reference for common task on the ILOM.

There is also a nice cheat sheet provided by Oracle for iLom Basic CLI Commands

Power On/Off/Reset system 

start /SYS
stop /SYS stop -force /SYS

reset  /SYS

Start console

start /SP/console
start -script /SP/console
start -script -force /SP/console

Updating firmware

load -source tftp://
load -source

Configure Network

Read more of this post