Storage Foundation HA 6.1 and Flexible Shared Storage

Veritas Flexible Storage Sharing

  • What is FSS?

Veritas Flexible Storage Sharing (FSS) is a new SFHA 6.1 feature that lets you configure a 'shared-nothing' storage volume. It is based upon Cluster Filesystem (CFS) but unlike base CFS it doesn't require shared storage (SAN, iSCSI, etc..).

You can read more about FSS here:
http://www.symantec.com/connect/blogs/configure-flexible-storage-sharing-using-veritas-operations-manager-61

Unlike the blog post above that uses the VOM GUI, I will be configuring FSS between two virtual machines in an old-fashioned style (CLI). I decided to write this blog because I couldn't find enough information in one place on how to setup FSS using the non-GUI stuff.

  • Lab setup and specs for each VM:

vcs14: RHEL6.6.z, 8Gb RAM, 32*8Tb local disks, 16*2Tb shared disks (these won't be used).
vcs15: RHEL6.6.z, 8Gb RAM, 32*8Tb local disks, 16*2Tb shared disks (these won't be used).

I won't go over the details of setting up a CFS cluster. Here are some of the things you would normally see after CFS has been setup (Think cfscluster config and cfscluster start):

[root@vcs14 ~]# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   2cad02 membership 01
Port b gen   2cad01 membership 01
Port d gen   2cad05 membership 01
Port f gen   2cad0f membership 01
Port h gen   2cad05 membership 01
Port u gen   2cad0d membership 01
Port v gen   2cad09 membership 01
Port w gen   2cad0b membership 01
Port y gen   2cad08 membership 01
[root@vcs14 ~]# vxdctl -c mode
mode: enabled: cluster active - MASTER
master: vcs14
[root@vcs14 ~]# lltconfig -a list
Link 0 (bond0):
  Node   0 vcs14      :   00:0C:29:D6:38:88  permanent
  Node   1 vcs15      :   00:0C:29:B0:34:23

Link 1 (bond1):
  Node   0 vcs14      :   00:0C:29:D6:38:92  permanent
  Node   1 vcs15      :   00:0C:29:B0:34:2D

Link 2 (bond2):
  Node   0 vcs14      :   00:0C:29:D6:38:9C  permanent
  Node   1 vcs15      :   00:0C:29:B0:34:37

[root@vcs14 ~]# cfscluster status

  Node             :  vcs14
  Cluster Manager  :  running
  CVM state        :  running
  MOUNT POINT    SHARED VOLUME  DISK GROUP        STATUS
  No mount point registered with cluster configuration


  Node             :  vcs15
  Cluster Manager  :  running
  CVM state        :  running
  MOUNT POINT    SHARED VOLUME  DISK GROUP        STATUS
  No mount point registered with cluster configuration


Here is vcs14:

[root@vcs14 ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:cdsdisk    -            -            online
disk_1       auto:cdsdisk    -            -            online
[....]
disk_14      auto:cdsdisk    -            -            online
disk_15      auto:cdsdisk    -            -            online
sdaa         auto:cdsdisk    -            -            online
sdab         auto:cdsdisk    -            -            online
sdac         auto:cdsdisk    -            -            online
sdad         auto:cdsdisk    -            -            online
sdae         auto:cdsdisk    -            -            online
sdaf         auto:cdsdisk    -            -            online
sdag         auto:cdsdisk    -            -            online
sdah         auto:cdsdisk    -            -            online
[....]
sdt          auto:cdsdisk    -            -            online
sdu          auto:cdsdisk    -            -            online
sdv          auto:cdsdisk    -            -            online
sdw          auto:cdsdisk    -            -            online
sdx          auto:cdsdisk    -            -            online
sdy          auto:cdsdisk    -            -            online
sdz          auto:cdsdisk    -            -            online

And its sibling vcs15:

[root@vcs15 ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:cdsdisk    -            -            online
disk_1       auto:cdsdisk    -            -            online
[....]
disk_14      auto:cdsdisk    -            -            online
disk_15      auto:cdsdisk    -            -            online
sdaa         auto:cdsdisk    -            -            online
sdab         auto:cdsdisk    -            -            online
sdac         auto:cdsdisk    -            -            online
sdad         auto:cdsdisk    -            -            online
sdae         auto:cdsdisk    -            -            online
sdaf         auto:cdsdisk    -            -            online
sdag         auto:cdsdisk    -            -            online
sdah         auto:cdsdisk    -            -            online
[....]
sdt          auto:cdsdisk    -            -            online
sdu          auto:cdsdisk    -            -            online
sdv          auto:cdsdisk    -            -            online
sdw          auto:cdsdisk    -            -            online
sdx          auto:cdsdisk    -            -            online
sdy          auto:cdsdisk    -            -            online
sdz          auto:cdsdisk    -            -            online

And here are our 8Tb disks:

[root@vcs14 ~]# lsblk -d|grep '^sd.*8T'
sdc       8:32   0    8T  0 disk
sdb       8:16   0    8T  0 disk
sdd       8:48   0    8T  0 disk
sdf       8:80   0    8T  0 disk
sde       8:64   0    8T  0 disk
sdg       8:96   0    8T  0 disk
sdh       8:112  0    8T  0 disk
sdi       8:128  0    8T  0 disk
sdl       8:176  0    8T  0 disk
sdm       8:192  0    8T  0 disk
sdj       8:144  0    8T  0 disk
sdk       8:160  0    8T  0 disk
sdp       8:240  0    8T  0 disk
sdn       8:208  0    8T  0 disk
sdr      65:16   0    8T  0 disk
sds      65:32   0    8T  0 disk
sdt      65:48   0    8T  0 disk
sdq      65:0    0    8T  0 disk
sdo       8:224  0    8T  0 disk
sdw      65:96   0    8T  0 disk
sdy      65:128  0    8T  0 disk
sdx      65:112  0    8T  0 disk
sdv      65:80   0    8T  0 disk
sdab     65:176  0    8T  0 disk
sdz      65:144  0    8T  0 disk
sdaa     65:160  0    8T  0 disk
sdac     65:192  0    8T  0 disk
sdae     65:224  0    8T  0 disk
sdad     65:208  0    8T  0 disk
sdu      65:64   0    8T  0 disk
sdaf     65:240  0    8T  0 disk
sdah     66:16   0    8T  0 disk
sdag     66:0    0    8T  0 disk

 [root@vcs15 ~]# lsblk -d|grep '^sd.*8T'
sdb       8:16   0    8T  0 disk
sdc       8:32   0    8T  0 disk
sdd       8:48   0    8T  0 disk
sde       8:64   0    8T  0 disk
sdf       8:80   0    8T  0 disk
sdh       8:112  0    8T  0 disk
sdi       8:128  0    8T  0 disk
sdg       8:96   0    8T  0 disk
sdk       8:160  0    8T  0 disk
sdl       8:176  0    8T  0 disk
sdj       8:144  0    8T  0 disk
sdm       8:192  0    8T  0 disk
sdn       8:208  0    8T  0 disk
sdp       8:240  0    8T  0 disk
sdo       8:224  0    8T  0 disk
sds      65:32   0    8T  0 disk
sdq      65:0    0    8T  0 disk
sdt      65:48   0    8T  0 disk
sdr      65:16   0    8T  0 disk
sdu      65:64   0    8T  0 disk
sdv      65:80   0    8T  0 disk
sdx      65:112  0    8T  0 disk
sdw      65:96   0    8T  0 disk
sdz      65:144  0    8T  0 disk
sdy      65:128  0    8T  0 disk
sdab     65:176  0    8T  0 disk
sdaa     65:160  0    8T  0 disk
sdaf     65:240  0    8T  0 disk
sdae     65:224  0    8T  0 disk
sdad     65:208  0    8T  0 disk
sdac     65:192  0    8T  0 disk
sdag     66:0    0    8T  0 disk
sdah     66:16   0    8T  0 disk

[root@vcs14 ~]# lsblk -d|grep '^sd.*8T'|wc -l
33

There are 32^H^H^H33 8Tb disks per system, like I said. :)


  • Meaningful disk names (Optional).


I don't like the default disk names, so let's change them to something meaningful (remember to vxdisksetup the disks if you haven't already):

[root@vcs14 ~]# lsblk -d|grep '^sd.*8T'|awk '{ print $1 }' > /tmp/disks
[root@vcs14 ~]# seq -w 0 32 > /tmp/nums
[root@vcs14 ~]# paste /tmp/disks /tmp/nums |awk '{ print "vxdmpadm setattr dmpnode",$1,"name=ssd_" $2 }' > /tmp/names.sh
[root@vcs14 ~]# bash /tmp/names.sh
[root@vcs14 ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:cdsdisk    -            -            online
disk_1       auto:cdsdisk    -            -            online
[....]
disk_14      auto:cdsdisk    -            -            online
disk_15      auto:cdsdisk    -            -            online
ssd_00       auto:cdsdisk    -            -            online
ssd_01       auto:cdsdisk    -            -            online
ssd_02       auto:cdsdisk    -            -            online
ssd_03       auto:cdsdisk    -            -            online
ssd_04       auto:cdsdisk    -            -            online
[....]
ssd_28       auto:cdsdisk    -            -            online
ssd_29       auto:cdsdisk    -            -            online
ssd_30       auto:cdsdisk    -            -            online
ssd_31       auto:cdsdisk    -            -            online
ssd_32       auto:cdsdisk    -            -            online


Let's do the same for vcs15 ( you get the picture..) :

[root@vcs15 ~]# lsblk -d|grep '^sd.*8T'|awk '{ print $1 }' > /tmp/disks
[root@vcs15 ~]# seq -w 0 32 > /tmp/nums
[root@vcs15 ~]# paste /tmp/disks /tmp/nums |awk '{ print "vxdmpadm setattr dmpnode",$1,"name=ssd_" $2 }' > /tmp/names.sh
[root@vcs15 ~]# bash /tmp/names.sh
[root@vcs15 ~]# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:cdsdisk    -            -            online
disk_1       auto:cdsdisk    -            -            online
[....]
disk_14      auto:cdsdisk    -            -            online
disk_15      auto:cdsdisk    -            -            online
ssd_00       auto:cdsdisk    -            -            online
ssd_01       auto:cdsdisk    -            -            online
ssd_02       auto:cdsdisk    -            -            online
ssd_03       auto:cdsdisk    -            -            online
ssd_04       auto:cdsdisk    -            -            online
[....]
ssd_28       auto:cdsdisk    -            -            online
ssd_29       auto:cdsdisk    -            -            online
ssd_30       auto:cdsdisk    -            -            online
ssd_31       auto:cdsdisk    -            -            online
ssd_32       auto:cdsdisk    -            -            online

  •  Now export the disks to the remote cluster nodes:


[root@vcs14 ~]# vxdisk list|grep ^ssd|awk '{ print $1}' |xargs -n1 vxdisk export
[root@vcs15 ~]# vxdisk list|grep ^ssd|awk '{ print $1}' |xargs -n1 vxdisk export

Look at the vxdisk output, now each node sees the disks that were exported on its sibling:

On vcs14, we see the disks from vcs15

[root@vcs14 ~]# vxdisk list|egrep 'DEVICE|ssd'
DEVICE       TYPE            DISK         GROUP        STATUS
ssd_00       auto:cdsdisk    -            -            online exported
ssd_01       auto:cdsdisk    -            -            online exported
ssd_02       auto:cdsdisk    -            -            online exported
ssd_03       auto:cdsdisk    -            -            online exported
ssd_04       auto:cdsdisk    -            -            online exported
ssd_05       auto:cdsdisk    -            -            online exported
ssd_06       auto:cdsdisk    -            -            online exported
ssd_07       auto:cdsdisk    -            -            online exported
ssd_08       auto:cdsdisk    -            -            online exported
ssd_09       auto:cdsdisk    -            -            online exported
ssd_10       auto:cdsdisk    -            -            online exported
ssd_11       auto:cdsdisk    -            -            online exported
ssd_12       auto:cdsdisk    -            -            online exported
ssd_13       auto:cdsdisk    -            -            online exported
ssd_14       auto:cdsdisk    -            -            online exported
ssd_15       auto:cdsdisk    -            -            online exported
ssd_16       auto:cdsdisk    -            -            online exported
ssd_17       auto:cdsdisk    -            -            online exported
ssd_18       auto:cdsdisk    -            -            online exported
ssd_19       auto:cdsdisk    -            -            online exported
ssd_20       auto:cdsdisk    -            -            online exported
ssd_21       auto:cdsdisk    -            -            online exported
ssd_22       auto:cdsdisk    -            -            online exported
ssd_23       auto:cdsdisk    -            -            online exported
ssd_24       auto:cdsdisk    -            -            online exported
ssd_25       auto:cdsdisk    -            -            online exported
ssd_26       auto:cdsdisk    -            -            online exported
ssd_27       auto:cdsdisk    -            -            online exported
ssd_28       auto:cdsdisk    -            -            online exported
ssd_29       auto:cdsdisk    -            -            online exported
ssd_30       auto:cdsdisk    -            -            online exported
ssd_31       auto:cdsdisk    -            -            online exported
ssd_32       auto:cdsdisk    -            -            online exported
vcs15_ssd_00 auto:cdsdisk    -            -            online remote
vcs15_ssd_01 auto:cdsdisk    -            -            online remote
vcs15_ssd_02 auto:cdsdisk    -            -            online remote
vcs15_ssd_03 auto:cdsdisk    -            -            online remote
vcs15_ssd_04 auto:cdsdisk    -            -            online remote
vcs15_ssd_05 auto:cdsdisk    -            -            online remote
vcs15_ssd_06 auto:cdsdisk    -            -            online remote
vcs15_ssd_07 auto:cdsdisk    -            -            online remote
vcs15_ssd_08 auto:cdsdisk    -            -            online remote
vcs15_ssd_09 auto:cdsdisk    -            -            online remote
vcs15_ssd_10 auto:cdsdisk    -            -            online remote
vcs15_ssd_11 auto:cdsdisk    -            -            online remote
vcs15_ssd_12 auto:cdsdisk    -            -            online remote
vcs15_ssd_13 auto:cdsdisk    -            -            online remote
vcs15_ssd_14 auto:cdsdisk    -            -            online remote
vcs15_ssd_15 auto:cdsdisk    -            -            online remote
vcs15_ssd_16 auto:cdsdisk    -            -            online remote
vcs15_ssd_17 auto:cdsdisk    -            -            online remote
vcs15_ssd_18 auto:cdsdisk    -            -            online remote
vcs15_ssd_19 auto:cdsdisk    -            -            online remote
vcs15_ssd_20 auto:cdsdisk    -            -            online remote
vcs15_ssd_21 auto:cdsdisk    -            -            online remote
vcs15_ssd_22 auto:cdsdisk    -            -            online remote
vcs15_ssd_23 auto:cdsdisk    -            -            online remote
vcs15_ssd_24 auto:cdsdisk    -            -            online remote
vcs15_ssd_25 auto:cdsdisk    -            -            online remote
vcs15_ssd_26 auto:cdsdisk    -            -            online remote
vcs15_ssd_27 auto:cdsdisk    -            -            online remote
vcs15_ssd_28 auto:cdsdisk    -            -            online remote
vcs15_ssd_29 auto:cdsdisk    -            -            online remote
vcs15_ssd_30 auto:cdsdisk    -            -            online remote
vcs15_ssd_31 auto:cdsdisk    -            -            online remote
vcs15_ssd_32 auto:cdsdisk    -            -            online remote

 And on vx15 we have vcs14's disks:

 [root@vcs15 ~]# vxdisk list|egrep 'DEVICE|ssd'
DEVICE       TYPE            DISK         GROUP        STATUS
ssd_00       auto:cdsdisk    -            -            online exported
ssd_01       auto:cdsdisk    -            -            online exported
ssd_02       auto:cdsdisk    -            -            online exported
ssd_03       auto:cdsdisk    -            -            online exported
ssd_04       auto:cdsdisk    -            -            online exported
ssd_05       auto:cdsdisk    -            -            online exported
ssd_06       auto:cdsdisk    -            -            online exported
ssd_07       auto:cdsdisk    -            -            online exported
ssd_08       auto:cdsdisk    -            -            online exported
ssd_09       auto:cdsdisk    -            -            online exported
ssd_10       auto:cdsdisk    -            -            online exported
ssd_11       auto:cdsdisk    -            -            online exported
ssd_12       auto:cdsdisk    -            -            online exported
ssd_13       auto:cdsdisk    -            -            online exported
ssd_14       auto:cdsdisk    -            -            online exported
ssd_15       auto:cdsdisk    -            -            online exported
ssd_16       auto:cdsdisk    -            -            online exported
ssd_17       auto:cdsdisk    -            -            online exported
ssd_18       auto:cdsdisk    -            -            online exported
ssd_19       auto:cdsdisk    -            -            online exported
ssd_20       auto:cdsdisk    -            -            online exported
ssd_21       auto:cdsdisk    -            -            online exported
ssd_22       auto:cdsdisk    -            -            online exported
ssd_23       auto:cdsdisk    -            -            online exported
ssd_24       auto:cdsdisk    -            -            online exported
ssd_25       auto:cdsdisk    -            -            online exported
ssd_26       auto:cdsdisk    -            -            online exported
ssd_27       auto:cdsdisk    -            -            online exported
ssd_28       auto:cdsdisk    -            -            online exported
ssd_29       auto:cdsdisk    -            -            online exported
ssd_30       auto:cdsdisk    -            -            online exported
ssd_31       auto:cdsdisk    -            -            online exported
ssd_32       auto:cdsdisk    -            -            online exported
vcs14_ssd_00 auto:cdsdisk    -            -            online remote
vcs14_ssd_01 auto:cdsdisk    -            -            online remote
vcs14_ssd_02 auto:cdsdisk    -            -            online remote
vcs14_ssd_03 auto:cdsdisk    -            -            online remote
vcs14_ssd_04 auto:cdsdisk    -            -            online remote
vcs14_ssd_05 auto:cdsdisk    -            -            online remote
vcs14_ssd_06 auto:cdsdisk    -            -            online remote
vcs14_ssd_07 auto:cdsdisk    -            -            online remote
vcs14_ssd_08 auto:cdsdisk    -            -            online remote
vcs14_ssd_09 auto:cdsdisk    -            -            online remote
vcs14_ssd_10 auto:cdsdisk    -            -            online remote
vcs14_ssd_11 auto:cdsdisk    -            -            online remote
vcs14_ssd_12 auto:cdsdisk    -            -            online remote
vcs14_ssd_13 auto:cdsdisk    -            -            online remote
vcs14_ssd_14 auto:cdsdisk    -            -            online remote
vcs14_ssd_15 auto:cdsdisk    -            -            online remote
vcs14_ssd_16 auto:cdsdisk    -            -            online remote
vcs14_ssd_17 auto:cdsdisk    -            -            online remote
vcs14_ssd_18 auto:cdsdisk    -            -            online remote
vcs14_ssd_19 auto:cdsdisk    -            -            online remote
vcs14_ssd_20 auto:cdsdisk    -            -            online remote
vcs14_ssd_21 auto:cdsdisk    -            -            online remote
vcs14_ssd_22 auto:cdsdisk    -            -            online remote
vcs14_ssd_23 auto:cdsdisk    -            -            online remote
vcs14_ssd_24 auto:cdsdisk    -            -            online remote
vcs14_ssd_25 auto:cdsdisk    -            -            online remote
vcs14_ssd_26 auto:cdsdisk    -            -            online remote
vcs14_ssd_27 auto:cdsdisk    -            -            online remote
vcs14_ssd_28 auto:cdsdisk    -            -            online remote
vcs14_ssd_29 auto:cdsdisk    -            -            online remote
vcs14_ssd_30 auto:cdsdisk    -            -            online remote
vcs14_ssd_31 auto:cdsdisk    -            -            online remote
vcs14_ssd_32 auto:cdsdisk    -            -            online remote

  • Configure an FSS diskgroup (do this on only one node):


[root@vcs14 ~]# vxdisk list|egrep 'ssd'|awk '{ print $1}' > /tmp/alldisks

* Create the diskgroup on one node (notice how it shows up on both nodes):
[root@vcs14 ~]# vxdg -o fss -s init FSS00dg $(cat /tmp/alldisks )
[root@vcs14 ~]# vxdg list
NAME         STATE           ID
FSS00dg      enabled,shared,cds   1423072143.104.vcs14

 [root@vcs15 ~]# vxdg list
NAME         STATE           ID
FSS00dg      enabled,shared,cds   1423072143.104.vcs14

And some info from vxdisk list:

[root@vcs14 ~]# vxdisk list|egrep '^(DEVI|ssd|vcs)'
DEVICE       TYPE            DISK         GROUP        STATUS
ssd_00       auto:cdsdisk    ssd_00       FSS00dg      online exported shared
ssd_01       auto:cdsdisk    ssd_01       FSS00dg      online exported shared
ssd_02       auto:cdsdisk    ssd_02       FSS00dg      online exported shared
ssd_03       auto:cdsdisk    ssd_03       FSS00dg      online exported shared
ssd_04       auto:cdsdisk    ssd_04       FSS00dg      online exported shared
ssd_05       auto:cdsdisk    ssd_05       FSS00dg      online exported shared
ssd_06       auto:cdsdisk    ssd_06       FSS00dg      online exported shared
ssd_07       auto:cdsdisk    ssd_07       FSS00dg      online exported shared
ssd_08       auto:cdsdisk    ssd_08       FSS00dg      online exported shared
ssd_09       auto:cdsdisk    ssd_09       FSS00dg      online exported shared
ssd_10       auto:cdsdisk    ssd_10       FSS00dg      online exported shared
ssd_11       auto:cdsdisk    ssd_11       FSS00dg      online exported shared
ssd_12       auto:cdsdisk    ssd_12       FSS00dg      online exported shared
ssd_13       auto:cdsdisk    ssd_13       FSS00dg      online exported shared
ssd_14       auto:cdsdisk    ssd_14       FSS00dg      online exported shared
ssd_15       auto:cdsdisk    ssd_15       FSS00dg      online exported shared
ssd_16       auto:cdsdisk    ssd_16       FSS00dg      online exported shared
ssd_17       auto:cdsdisk    ssd_17       FSS00dg      online exported shared
ssd_18       auto:cdsdisk    ssd_18       FSS00dg      online exported shared
ssd_19       auto:cdsdisk    ssd_19       FSS00dg      online exported shared
ssd_20       auto:cdsdisk    ssd_20       FSS00dg      online exported shared
ssd_21       auto:cdsdisk    ssd_21       FSS00dg      online exported shared
ssd_22       auto:cdsdisk    ssd_22       FSS00dg      online exported shared
ssd_23       auto:cdsdisk    ssd_23       FSS00dg      online exported shared
ssd_24       auto:cdsdisk    ssd_24       FSS00dg      online exported shared
ssd_25       auto:cdsdisk    ssd_25       FSS00dg      online exported shared
ssd_26       auto:cdsdisk    ssd_26       FSS00dg      online exported shared
ssd_27       auto:cdsdisk    ssd_27       FSS00dg      online exported shared
ssd_28       auto:cdsdisk    ssd_28       FSS00dg      online exported shared
ssd_29       auto:cdsdisk    ssd_29       FSS00dg      online exported shared
ssd_30       auto:cdsdisk    ssd_30       FSS00dg      online exported shared
ssd_31       auto:cdsdisk    ssd_31       FSS00dg      online exported shared
ssd_32       auto:cdsdisk    ssd_32       FSS00dg      online exported shared
vcs15_ssd_00 auto:cdsdisk    vcs15_ssd_00  FSS00dg      online shared remote
vcs15_ssd_01 auto:cdsdisk    vcs15_ssd_01  FSS00dg      online shared remote
vcs15_ssd_02 auto:cdsdisk    vcs15_ssd_02  FSS00dg      online shared remote
vcs15_ssd_03 auto:cdsdisk    vcs15_ssd_03  FSS00dg      online shared remote
vcs15_ssd_04 auto:cdsdisk    vcs15_ssd_04  FSS00dg      online shared remote
vcs15_ssd_05 auto:cdsdisk    vcs15_ssd_05  FSS00dg      online shared remote
vcs15_ssd_06 auto:cdsdisk    vcs15_ssd_06  FSS00dg      online shared remote
vcs15_ssd_07 auto:cdsdisk    vcs15_ssd_07  FSS00dg      online shared remote
vcs15_ssd_08 auto:cdsdisk    vcs15_ssd_08  FSS00dg      online shared remote
vcs15_ssd_09 auto:cdsdisk    vcs15_ssd_09  FSS00dg      online shared remote
vcs15_ssd_10 auto:cdsdisk    vcs15_ssd_10  FSS00dg      online shared remote
vcs15_ssd_11 auto:cdsdisk    vcs15_ssd_11  FSS00dg      online shared remote
vcs15_ssd_12 auto:cdsdisk    vcs15_ssd_12  FSS00dg      online shared remote
vcs15_ssd_13 auto:cdsdisk    vcs15_ssd_13  FSS00dg      online shared remote
vcs15_ssd_14 auto:cdsdisk    vcs15_ssd_14  FSS00dg      online shared remote
vcs15_ssd_15 auto:cdsdisk    vcs15_ssd_15  FSS00dg      online shared remote
vcs15_ssd_16 auto:cdsdisk    vcs15_ssd_16  FSS00dg      online shared remote
vcs15_ssd_17 auto:cdsdisk    vcs15_ssd_17  FSS00dg      online shared remote
vcs15_ssd_18 auto:cdsdisk    vcs15_ssd_18  FSS00dg      online shared remote
vcs15_ssd_19 auto:cdsdisk    vcs15_ssd_19  FSS00dg      online shared remote
vcs15_ssd_20 auto:cdsdisk    vcs15_ssd_20  FSS00dg      online shared remote
vcs15_ssd_21 auto:cdsdisk    vcs15_ssd_21  FSS00dg      online shared remote
vcs15_ssd_22 auto:cdsdisk    vcs15_ssd_22  FSS00dg      online shared remote
vcs15_ssd_23 auto:cdsdisk    vcs15_ssd_23  FSS00dg      online shared remote
vcs15_ssd_24 auto:cdsdisk    vcs15_ssd_24  FSS00dg      online shared remote
vcs15_ssd_25 auto:cdsdisk    vcs15_ssd_25  FSS00dg      online shared remote
vcs15_ssd_26 auto:cdsdisk    vcs15_ssd_26  FSS00dg      online shared remote
vcs15_ssd_27 auto:cdsdisk    vcs15_ssd_27  FSS00dg      online shared remote
vcs15_ssd_28 auto:cdsdisk    vcs15_ssd_28  FSS00dg      online shared remote
vcs15_ssd_29 auto:cdsdisk    vcs15_ssd_29  FSS00dg      online shared remote
vcs15_ssd_30 auto:cdsdisk    vcs15_ssd_30  FSS00dg      online shared remote
vcs15_ssd_31 auto:cdsdisk    vcs15_ssd_31  FSS00dg      online shared remote
vcs15_ssd_32 auto:cdsdisk    vcs15_ssd_32  FSS00dg      online shared remote

  • Create a volume in the FSS diskgroup to hold our data:


[root@vcs14 ~]# vxdctl -c mode
mode: enabled: cluster active - MASTER
master: vcs14

[root@vcs14 ~]# vxassist -g FSS00dg maxsize nmirror=2 layout=mirror-stripe stripeunit=512k
Maximum volume size: 566933245952 (276822874Mb)

* To be on the safe side: 256tb = 262144g, pick 262144g - 2g : 262142g
(See the footnotes for more information on this)

[root@vcs14 ~]# vxassist -g FSS00dg make fss0_lv 262142g nmirror=2 layout=mirror-stripe stripeunit=512k

Let's look at the newly created volume:

[root@vcs14 ~]# vxprint -Ath
Disk group: FSS00dg

 DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         RVG          KSTATE   STATE    NVOLUME
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
SR NAME         KSTATE

 dg FSS00dg      default      default  35000    1423072143.104.vcs14

 dm ssd_00       ssd_00       auto     65536    17179803344 -
dm ssd_01       ssd_01       auto     65536    17179803344 -
dm ssd_02       ssd_02       auto     65536    17179803344 -
dm ssd_03       ssd_03       auto     65536    17179803344 -
dm ssd_04       ssd_04       auto     65536    17179803344 -
[....]
dm ssd_28       ssd_28       auto     65536    17179803344 -
dm ssd_29       ssd_29       auto     65536    17179803344 -
dm ssd_30       ssd_30       auto     65536    17179803344 -
dm ssd_31       ssd_31       auto     65536    17179803344 -
dm ssd_32       ssd_32       auto     65536    17179803344 -
dm vcs15_ssd_00 vcs15_ssd_00 auto     65536    17179803344 REMOTE
dm vcs15_ssd_01 vcs15_ssd_01 auto     65536    17179803344 REMOTE
dm vcs15_ssd_02 vcs15_ssd_02 auto     65536    17179803344 REMOTE
dm vcs15_ssd_03 vcs15_ssd_03 auto     65536    17179803344 REMOTE
dm vcs15_ssd_04 vcs15_ssd_04 auto     65536    17179803344 REMOTE
[....]
dm vcs15_ssd_28 vcs15_ssd_28 auto     65536    17179803344 REMOTE
dm vcs15_ssd_29 vcs15_ssd_29 auto     65536    17179803344 REMOTE
dm vcs15_ssd_30 vcs15_ssd_30 auto     65536    17179803344 REMOTE
dm vcs15_ssd_31 vcs15_ssd_31 auto     65536    17179803344 REMOTE
dm vcs15_ssd_32 vcs15_ssd_32 auto     65536    17179803344 REMOTE

 v  fss0_lv      -            ENABLED  ACTIVE   549755813888 SELECT -       fsgen
pl fss0_lv-01   fss0_lv      ENABLED  ACTIVE   549755814912 STRIPE 3/1024  RW
sd vcs15_ssd_00-01 fss0_lv-01 vcs15_ssd_00 0   17179803344 0/0    vcs15_ssd_00 ENA
sd vcs15_ssd_03-01 fss0_lv-01 vcs15_ssd_03 0   17179803344 0/17179803344 vcs15_ssd_03 ENA
sd vcs15_ssd_06-01 fss0_lv-01 vcs15_ssd_06 0   17179803344 0/34359606688 vcs15_ssd_06 ENA
sd vcs15_ssd_09-01 fss0_lv-01 vcs15_ssd_09 0   17179803344 0/51539410032 vcs15_ssd_09 ENA
sd vcs15_ssd_12-01 fss0_lv-01 vcs15_ssd_12 0   17179803344 0/68719213376 vcs15_ssd_12 ENA
sd vcs15_ssd_15-01 fss0_lv-01 vcs15_ssd_15 0   17179803344 0/85899016720 vcs15_ssd_15 ENA
sd vcs15_ssd_18-01 fss0_lv-01 vcs15_ssd_18 0   17179803344 0/103078820064 vcs15_ssd_18 ENA
sd vcs15_ssd_21-01 fss0_lv-01 vcs15_ssd_21 0   17179803344 0/120258623408 vcs15_ssd_21 ENA
sd vcs15_ssd_24-01 fss0_lv-01 vcs15_ssd_24 0   17179803344 0/137438426752 vcs15_ssd_24 ENA
sd vcs15_ssd_27-01 fss0_lv-01 vcs15_ssd_27 0   17179803344 0/154618230096 vcs15_ssd_27 ENA
sd vcs15_ssd_30-01 fss0_lv-01 vcs15_ssd_30 0   11453904864 0/171798033440 vcs15_ssd_30 ENA
sd vcs15_ssd_01-01 fss0_lv-01 vcs15_ssd_01 0   17179803344 1/0    vcs15_ssd_01 ENA
sd vcs15_ssd_04-01 fss0_lv-01 vcs15_ssd_04 0   17179803344 1/17179803344 vcs15_ssd_04 ENA
sd vcs15_ssd_07-01 fss0_lv-01 vcs15_ssd_07 0   17179803344 1/34359606688 vcs15_ssd_07 ENA
sd vcs15_ssd_10-01 fss0_lv-01 vcs15_ssd_10 0   17179803344 1/51539410032 vcs15_ssd_10 ENA
sd vcs15_ssd_13-01 fss0_lv-01 vcs15_ssd_13 0   17179803344 1/68719213376 vcs15_ssd_13 ENA
sd vcs15_ssd_16-01 fss0_lv-01 vcs15_ssd_16 0   17179803344 1/85899016720 vcs15_ssd_16 ENA
sd vcs15_ssd_19-01 fss0_lv-01 vcs15_ssd_19 0   17179803344 1/103078820064 vcs15_ssd_19 ENA
sd vcs15_ssd_22-01 fss0_lv-01 vcs15_ssd_22 0   17179803344 1/120258623408 vcs15_ssd_22 ENA
sd vcs15_ssd_25-01 fss0_lv-01 vcs15_ssd_25 0   17179803344 1/137438426752 vcs15_ssd_25 ENA
sd vcs15_ssd_28-01 fss0_lv-01 vcs15_ssd_28 0   17179803344 1/154618230096 vcs15_ssd_28 ENA
sd vcs15_ssd_31-01 fss0_lv-01 vcs15_ssd_31 0   11453904864 1/171798033440 vcs15_ssd_31 ENA
sd vcs15_ssd_02-01 fss0_lv-01 vcs15_ssd_02 0   17179803344 2/0    vcs15_ssd_02 ENA
sd vcs15_ssd_05-01 fss0_lv-01 vcs15_ssd_05 0   17179803344 2/17179803344 vcs15_ssd_05 ENA
sd vcs15_ssd_08-01 fss0_lv-01 vcs15_ssd_08 0   17179803344 2/34359606688 vcs15_ssd_08 ENA
sd vcs15_ssd_11-01 fss0_lv-01 vcs15_ssd_11 0   17179803344 2/51539410032 vcs15_ssd_11 ENA
sd vcs15_ssd_14-01 fss0_lv-01 vcs15_ssd_14 0   17179803344 2/68719213376 vcs15_ssd_14 ENA
sd vcs15_ssd_17-01 fss0_lv-01 vcs15_ssd_17 0   17179803344 2/85899016720 vcs15_ssd_17 ENA
sd vcs15_ssd_20-01 fss0_lv-01 vcs15_ssd_20 0   17179803344 2/103078820064 vcs15_ssd_20 ENA
sd vcs15_ssd_23-01 fss0_lv-01 vcs15_ssd_23 0   17179803344 2/120258623408 vcs15_ssd_23 ENA
sd vcs15_ssd_26-01 fss0_lv-01 vcs15_ssd_26 0   17179803344 2/137438426752 vcs15_ssd_26 ENA
sd vcs15_ssd_29-01 fss0_lv-01 vcs15_ssd_29 0   17179803344 2/154618230096 vcs15_ssd_29 ENA
sd vcs15_ssd_32-01 fss0_lv-01 vcs15_ssd_32 0   11453904864 2/171798033440 vcs15_ssd_32 ENA
pl fss0_lv-02   fss0_lv      ENABLED  ACTIVE   549755814912 STRIPE 3/1024  RW
sd ssd_00-01    fss0_lv-02   ssd_00   0        17179803344 0/0    ssd_00   ENA
sd ssd_03-01    fss0_lv-02   ssd_03   0        17179803344 0/17179803344 ssd_03 ENA
sd ssd_06-01    fss0_lv-02   ssd_06   0        17179803344 0/34359606688 ssd_06 ENA
sd ssd_09-01    fss0_lv-02   ssd_09   0        17179803344 0/51539410032 ssd_09 ENA
sd ssd_12-01    fss0_lv-02   ssd_12   0        17179803344 0/68719213376 ssd_12 ENA
sd ssd_15-01    fss0_lv-02   ssd_15   0        17179803344 0/85899016720 ssd_15 ENA
sd ssd_18-01    fss0_lv-02   ssd_18   0        17179803344 0/103078820064 ssd_18 ENA
sd ssd_21-01    fss0_lv-02   ssd_21   0        17179803344 0/120258623408 ssd_21 ENA
sd ssd_24-01    fss0_lv-02   ssd_24   0        17179803344 0/137438426752 ssd_24 ENA
sd ssd_27-01    fss0_lv-02   ssd_27   0        17179803344 0/154618230096 ssd_27 ENA
sd ssd_30-01    fss0_lv-02   ssd_30   0        11453904864 0/171798033440 ssd_30 ENA
sd ssd_01-01    fss0_lv-02   ssd_01   0        17179803344 1/0    ssd_01   ENA
sd ssd_04-01    fss0_lv-02   ssd_04   0        17179803344 1/17179803344 ssd_04 ENA
sd ssd_07-01    fss0_lv-02   ssd_07   0        17179803344 1/34359606688 ssd_07 ENA
sd ssd_10-01    fss0_lv-02   ssd_10   0        17179803344 1/51539410032 ssd_10 ENA
sd ssd_13-01    fss0_lv-02   ssd_13   0        17179803344 1/68719213376 ssd_13 ENA
sd ssd_16-01    fss0_lv-02   ssd_16   0        17179803344 1/85899016720 ssd_16 ENA
sd ssd_19-01    fss0_lv-02   ssd_19   0        17179803344 1/103078820064 ssd_19 ENA
sd ssd_22-01    fss0_lv-02   ssd_22   0        17179803344 1/120258623408 ssd_22 ENA
sd ssd_25-01    fss0_lv-02   ssd_25   0        17179803344 1/137438426752 ssd_25 ENA
sd ssd_28-01    fss0_lv-02   ssd_28   0        17179803344 1/154618230096 ssd_28 ENA
sd ssd_31-01    fss0_lv-02   ssd_31   0        11453904864 1/171798033440 ssd_31 ENA
sd ssd_02-01    fss0_lv-02   ssd_02   0        17179803344 2/0    ssd_02   ENA
sd ssd_05-01    fss0_lv-02   ssd_05   0        17179803344 2/17179803344 ssd_05 ENA
sd ssd_08-01    fss0_lv-02   ssd_08   0        17179803344 2/34359606688 ssd_08 ENA
sd ssd_11-01    fss0_lv-02   ssd_11   0        17179803344 2/51539410032 ssd_11 ENA
sd ssd_14-01    fss0_lv-02   ssd_14   0        17179803344 2/68719213376 ssd_14 ENA
sd ssd_17-01    fss0_lv-02   ssd_17   0        17179803344 2/85899016720 ssd_17 ENA
sd ssd_20-01    fss0_lv-02   ssd_20   0        17179803344 2/103078820064 ssd_20 ENA
sd ssd_23-01    fss0_lv-02   ssd_23   0        17179803344 2/120258623408 ssd_23 ENA
sd ssd_26-01    fss0_lv-02   ssd_26   0        17179803344 2/137438426752 ssd_26 ENA
sd ssd_29-01    fss0_lv-02   ssd_29   0        17179803344 2/154618230096 ssd_29 ENA
sd ssd_32-01    fss0_lv-02   ssd_32   0        11453904864 2/171798033440 ssd_32 ENA
dc fss0_lv_dco  fss0_lv      fss0_lv_dcl
v  fss0_lv_dcl  -            ENABLED  ACTIVE   18980992 SELECT    -        gen
pl fss0_lv_dcl-01 fss0_lv_dcl ENABLED ACTIVE   18980992 CONCAT    -        RW
sd ssd_30-02    fss0_lv_dcl-01 ssd_30 11453904864 18980992 0      ssd_30   ENA
pl fss0_lv_dcl-02 fss0_lv_dcl ENABLED ACTIVE   18980992 CONCAT    -        RW
sd vcs15_ssd_30-02 fss0_lv_dcl-02 vcs15_ssd_30 11453904864 18980992 0 vcs15_ssd_30 ENA

It looks good! Let's do the same on the other node:

[root@vcs15 ~]# vxprint -Ath
Disk group: FSS00dg

 DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         RVG          KSTATE   STATE    NVOLUME
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
SR NAME         KSTATE

 dg FSS00dg      default      default  35000    1423072143.104.vcs14

 dm ssd_00       vcs14_ssd_00 auto     65536    17179803344 REMOTE
dm ssd_01       vcs14_ssd_01 auto     65536    17179803344 REMOTE
dm ssd_02       vcs14_ssd_02 auto     65536    17179803344 REMOTE
dm ssd_03       vcs14_ssd_03 auto     65536    17179803344 REMOTE
dm ssd_04       vcs14_ssd_04 auto     65536    17179803344 REMOTE
[....]
dm ssd_28       vcs14_ssd_28 auto     65536    17179803344 REMOTE
dm ssd_29       vcs14_ssd_29 auto     65536    17179803344 REMOTE
dm ssd_30       vcs14_ssd_30 auto     65536    17179803344 REMOTE
dm ssd_31       vcs14_ssd_31 auto     65536    17179803344 REMOTE
dm ssd_32       vcs14_ssd_32 auto     65536    17179803344 REMOTE
dm vcs15_ssd_00 ssd_00       auto     65536    17179803344 -
dm vcs15_ssd_01 ssd_01       auto     65536    17179803344 -
dm vcs15_ssd_02 ssd_02       auto     65536    17179803344 -
dm vcs15_ssd_03 ssd_03       auto     65536    17179803344 -
dm vcs15_ssd_04 ssd_04       auto     65536    17179803344 -
[....]
dm vcs15_ssd_28 ssd_28       auto     65536    17179803344 -
dm vcs15_ssd_29 ssd_29       auto     65536    17179803344 -
dm vcs15_ssd_30 ssd_30       auto     65536    17179803344 -
dm vcs15_ssd_31 ssd_31       auto     65536    17179803344 -
dm vcs15_ssd_32 ssd_32       auto     65536    17179803344 -

 v  fss0_lv      -            ENABLED  ACTIVE   549755813888 SELECT -       fsgen
pl fss0_lv-01   fss0_lv      ENABLED  ACTIVE   549755814912 STRIPE 3/1024  RW
sd vcs15_ssd_00-01 fss0_lv-01 vcs15_ssd_00 0   17179803344 0/0    ssd_00   ENA
sd vcs15_ssd_03-01 fss0_lv-01 vcs15_ssd_03 0   17179803344 0/17179803344 ssd_03 ENA
sd vcs15_ssd_06-01 fss0_lv-01 vcs15_ssd_06 0   17179803344 0/34359606688 ssd_06 ENA
sd vcs15_ssd_09-01 fss0_lv-01 vcs15_ssd_09 0   17179803344 0/51539410032 ssd_09 ENA
sd vcs15_ssd_12-01 fss0_lv-01 vcs15_ssd_12 0   17179803344 0/68719213376 ssd_12 ENA
sd vcs15_ssd_15-01 fss0_lv-01 vcs15_ssd_15 0   17179803344 0/85899016720 ssd_15 ENA
sd vcs15_ssd_18-01 fss0_lv-01 vcs15_ssd_18 0   17179803344 0/103078820064 ssd_18 ENA
sd vcs15_ssd_21-01 fss0_lv-01 vcs15_ssd_21 0   17179803344 0/120258623408 ssd_21 ENA
sd vcs15_ssd_24-01 fss0_lv-01 vcs15_ssd_24 0   17179803344 0/137438426752 ssd_24 ENA
sd vcs15_ssd_27-01 fss0_lv-01 vcs15_ssd_27 0   17179803344 0/154618230096 ssd_27 ENA
sd vcs15_ssd_30-01 fss0_lv-01 vcs15_ssd_30 0   11453904864 0/171798033440 ssd_30 ENA
sd vcs15_ssd_01-01 fss0_lv-01 vcs15_ssd_01 0   17179803344 1/0    ssd_01   ENA
sd vcs15_ssd_04-01 fss0_lv-01 vcs15_ssd_04 0   17179803344 1/17179803344 ssd_04 ENA
sd vcs15_ssd_07-01 fss0_lv-01 vcs15_ssd_07 0   17179803344 1/34359606688 ssd_07 ENA
sd vcs15_ssd_10-01 fss0_lv-01 vcs15_ssd_10 0   17179803344 1/51539410032 ssd_10 ENA
sd vcs15_ssd_13-01 fss0_lv-01 vcs15_ssd_13 0   17179803344 1/68719213376 ssd_13 ENA
sd vcs15_ssd_16-01 fss0_lv-01 vcs15_ssd_16 0   17179803344 1/85899016720 ssd_16 ENA
sd vcs15_ssd_19-01 fss0_lv-01 vcs15_ssd_19 0   17179803344 1/103078820064 ssd_19 ENA
sd vcs15_ssd_22-01 fss0_lv-01 vcs15_ssd_22 0   17179803344 1/120258623408 ssd_22 ENA
sd vcs15_ssd_25-01 fss0_lv-01 vcs15_ssd_25 0   17179803344 1/137438426752 ssd_25 ENA
sd vcs15_ssd_28-01 fss0_lv-01 vcs15_ssd_28 0   17179803344 1/154618230096 ssd_28 ENA
sd vcs15_ssd_31-01 fss0_lv-01 vcs15_ssd_31 0   11453904864 1/171798033440 ssd_31 ENA
sd vcs15_ssd_02-01 fss0_lv-01 vcs15_ssd_02 0   17179803344 2/0    ssd_02   ENA
sd vcs15_ssd_05-01 fss0_lv-01 vcs15_ssd_05 0   17179803344 2/17179803344 ssd_05 ENA
sd vcs15_ssd_08-01 fss0_lv-01 vcs15_ssd_08 0   17179803344 2/34359606688 ssd_08 ENA
sd vcs15_ssd_11-01 fss0_lv-01 vcs15_ssd_11 0   17179803344 2/51539410032 ssd_11 ENA
sd vcs15_ssd_14-01 fss0_lv-01 vcs15_ssd_14 0   17179803344 2/68719213376 ssd_14 ENA
sd vcs15_ssd_17-01 fss0_lv-01 vcs15_ssd_17 0   17179803344 2/85899016720 ssd_17 ENA
sd vcs15_ssd_20-01 fss0_lv-01 vcs15_ssd_20 0   17179803344 2/103078820064 ssd_20 ENA
sd vcs15_ssd_23-01 fss0_lv-01 vcs15_ssd_23 0   17179803344 2/120258623408 ssd_23 ENA
sd vcs15_ssd_26-01 fss0_lv-01 vcs15_ssd_26 0   17179803344 2/137438426752 ssd_26 ENA
sd vcs15_ssd_29-01 fss0_lv-01 vcs15_ssd_29 0   17179803344 2/154618230096 ssd_29 ENA
sd vcs15_ssd_32-01 fss0_lv-01 vcs15_ssd_32 0   11453904864 2/171798033440 ssd_32 ENA
pl fss0_lv-02   fss0_lv      ENABLED  ACTIVE   549755814912 STRIPE 3/1024  RW
sd ssd_00-01    fss0_lv-02   ssd_00   0        17179803344 0/0    vcs14_ssd_00 ENA
sd ssd_03-01    fss0_lv-02   ssd_03   0        17179803344 0/17179803344 vcs14_ssd_03 ENA
sd ssd_06-01    fss0_lv-02   ssd_06   0        17179803344 0/34359606688 vcs14_ssd_06 ENA
sd ssd_09-01    fss0_lv-02   ssd_09   0        17179803344 0/51539410032 vcs14_ssd_09 ENA
sd ssd_12-01    fss0_lv-02   ssd_12   0        17179803344 0/68719213376 vcs14_ssd_12 ENA
sd ssd_15-01    fss0_lv-02   ssd_15   0        17179803344 0/85899016720 vcs14_ssd_15 ENA
sd ssd_18-01    fss0_lv-02   ssd_18   0        17179803344 0/103078820064 vcs14_ssd_18 ENA
sd ssd_21-01    fss0_lv-02   ssd_21   0        17179803344 0/120258623408 vcs14_ssd_21 ENA
sd ssd_24-01    fss0_lv-02   ssd_24   0        17179803344 0/137438426752 vcs14_ssd_24 ENA
sd ssd_27-01    fss0_lv-02   ssd_27   0        17179803344 0/154618230096 vcs14_ssd_27 ENA
sd ssd_30-01    fss0_lv-02   ssd_30   0        11453904864 0/171798033440 vcs14_ssd_30 ENA
sd ssd_01-01    fss0_lv-02   ssd_01   0        17179803344 1/0    vcs14_ssd_01 ENA
sd ssd_04-01    fss0_lv-02   ssd_04   0        17179803344 1/17179803344 vcs14_ssd_04 ENA
sd ssd_07-01    fss0_lv-02   ssd_07   0        17179803344 1/34359606688 vcs14_ssd_07 ENA
sd ssd_10-01    fss0_lv-02   ssd_10   0        17179803344 1/51539410032 vcs14_ssd_10 ENA
sd ssd_13-01    fss0_lv-02   ssd_13   0        17179803344 1/68719213376 vcs14_ssd_13 ENA
sd ssd_16-01    fss0_lv-02   ssd_16   0        17179803344 1/85899016720 vcs14_ssd_16 ENA
sd ssd_19-01    fss0_lv-02   ssd_19   0        17179803344 1/103078820064 vcs14_ssd_19 ENA
sd ssd_22-01    fss0_lv-02   ssd_22   0        17179803344 1/120258623408 vcs14_ssd_22 ENA
sd ssd_25-01    fss0_lv-02   ssd_25   0        17179803344 1/137438426752 vcs14_ssd_25 ENA
sd ssd_28-01    fss0_lv-02   ssd_28   0        17179803344 1/154618230096 vcs14_ssd_28 ENA
sd ssd_31-01    fss0_lv-02   ssd_31   0        11453904864 1/171798033440 vcs14_ssd_31 ENA
sd ssd_02-01    fss0_lv-02   ssd_02   0        17179803344 2/0    vcs14_ssd_02 ENA
sd ssd_05-01    fss0_lv-02   ssd_05   0        17179803344 2/17179803344 vcs14_ssd_05 ENA
sd ssd_08-01    fss0_lv-02   ssd_08   0        17179803344 2/34359606688 vcs14_ssd_08 ENA
sd ssd_11-01    fss0_lv-02   ssd_11   0        17179803344 2/51539410032 vcs14_ssd_11 ENA
sd ssd_14-01    fss0_lv-02   ssd_14   0        17179803344 2/68719213376 vcs14_ssd_14 ENA
sd ssd_17-01    fss0_lv-02   ssd_17   0        17179803344 2/85899016720 vcs14_ssd_17 ENA
sd ssd_20-01    fss0_lv-02   ssd_20   0        17179803344 2/103078820064 vcs14_ssd_20 ENA
sd ssd_23-01    fss0_lv-02   ssd_23   0        17179803344 2/120258623408 vcs14_ssd_23 ENA
sd ssd_26-01    fss0_lv-02   ssd_26   0        17179803344 2/137438426752 vcs14_ssd_26 ENA
sd ssd_29-01    fss0_lv-02   ssd_29   0        17179803344 2/154618230096 vcs14_ssd_29 ENA
sd ssd_32-01    fss0_lv-02   ssd_32   0        11453904864 2/171798033440 vcs14_ssd_32 ENA
dc fss0_lv_dco  fss0_lv      fss0_lv_dcl
v  fss0_lv_dcl  -            ENABLED  ACTIVE   18980992 SELECT    -        gen
pl fss0_lv_dcl-01 fss0_lv_dcl ENABLED ACTIVE   18980992 CONCAT    -        RW
sd ssd_30-02    fss0_lv_dcl-01 ssd_30 11453904864 18980992 0      vcs14_ssd_30 ENA
pl fss0_lv_dcl-02 fss0_lv_dcl ENABLED ACTIVE   18980992 CONCAT    -        RW
sd vcs15_ssd_30-02 fss0_lv_dcl-02 vcs15_ssd_30 11453904864 18980992 0 ssd_30 ENA

  • That big filesystem


Now, create a filesystem on that volume so we can make use of it (Do this on only one node):

[root@vcs14 ~]# mkfs -t vxfs /dev/vx/dsk/FSS00dg/fss0_lv
    version 10 layout
    549755813888 sectors, 34359738368 blocks of size 8192, log size 32768 blocks
    rcq size 8192 blocks
    largefiles supported
    maxlink supported

CFSmount the Filesystem (Do this on only one node):

[root@vcs14 ~]# cfsmntadm add FSS00dg fss0_lv /shared/fss0 fss00_SG all=crw
  Mount Point is being added...
  /shared/fss0 added to the cluster-configuration

 [root@vcs14 ~]# cfsmount /shared/fss0
  Mounting...
  [/dev/vx/dsk/FSS00dg/fss0_lv] mounted successfully at /shared/fss0 on vcs14
  [/dev/vx/dsk/FSS00dg/fss0_lv] mounted successfully at /shared/fss0 on vcs15

  • Configure a Service Group (Optional)

Add a service group to make it show in the Cluster configuration and autostart the right things (tm):

[root@vcs14 ~]# haconf -makerw
[root@vcs14 ~]# hares -modify vxfsckd ActivationMode  cvmdg sw FSS00dg sw -sys vcs14
[root@vcs14 ~]# hares -modify vxfsckd ActivationMode  cvmdg sw FSS00dg sw -sys vcs15
[root@vcs14 ~]# hagrp -add fss00_SG
[root@vcs14 ~]# hagrp -modify fss00_SG SystemList  vcs14 0 vcs15 1
[root@vcs14 ~]# hagrp -modify fss00_SG Parallel 1
[root@vcs14 ~]# hagrp -modify fss00_SG AutoStartList  vcs14 vcs15
[root@vcs14 ~]# hagrp -modify fss00_SG SourceFile "./main.cf"
[root@vcs14 ~]# hares -add cfsmount_res CFSMount fss00_SG
[root@vcs14 ~]# hares -modify cfsmount_res BlockDevice "/dev/vx/dsk/FSS00dg/fss0_lv"
[root@vcs14 ~]# hares -modify cfsmount_res MountPoint "/shared/fss0"
[root@vcs14 ~]# hares -add cvmvoldg_res CVMVolDg fss00_SG
[root@vcs14 ~]# hares -modify cvmvoldg_res CVMVolume  fss0_lv
[root@vcs14 ~]# hares -modify cvmvoldg_res CVMDiskGroup FSS00dg
[root@vcs14 ~]# haconf -makerw

This should result in this output:

[root@vcs14 ~]# hagrp -state fss00_SG
#Group       Attribute             System     Value
fss00_SG     State                 vcs14      |ONLINE|
fss00_SG     State                 vcs15      |ONLINE|
[root@vcs14 ~]# hares -state|grep -i cfs
cfsmount_res         State                        vcs14      ONLINE
cfsmount_res         State                        vcs15      ONLINE

[root@vcs14 ~]# mount -v|grep fss0
/dev/vx/dsk/FSS00dg/fss0_lv on /shared/fss0 type vxfs (rw,mntlock=VCS,cluster,crw,delaylog,largefiles,mincache=dsync,noatime,nomtime,ioerror=mdisable)

 [root@vcs15 ~]# mount -v|grep fss0
/dev/vx/dsk/FSS00dg/fss0_lv on /shared/fss0 type vxfs (rw,mntlock=VCS,cluster,crw,delaylog,largefiles,mincache=dsync,noatime,nomtime,ioerror=mdisable)

[root@vcs14 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootdg-lv_root
                       36G   22G   12G  65% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/sda1             477M  100M  353M  22% /boot
tmpfs                 788M  216K  788M   1% /tmp
tmpfs                 4.0K     0  4.0K   0% /dev/vx
tmpfs                  64M  204K   64M   1% /var/VRTSvcs/lock
/dev/vx/dsk/FSS00dg/fss0_lv
                      256T  8.7G  256T   1% /shared/fss0

# netrun -q -e 'df -h -t vxfs' vcs1{4,5}
Progress:  0% |-----------+-----------+-----------+-----------| 100%
              ################################################# Done.
vcs14: Filesystem            Size  Used Avail Use% Mounted on
vcs14: /dev/vx/dsk/FSS00dg/fss0_lv
vcs14:                       256T  8.7G  256T   1% /shared/fss0
vcs15: Filesystem            Size  Used Avail Use% Mounted on
vcs15: /dev/vx/dsk/FSS00dg/fss0_lv
vcs15:                       256T  8.7G  256T   1% /shared/fss0

  • Run some tests

Let's do some writes from both nodes (remember that these are Virtual Machines with emulated disks so the performance would be better if I had real physical DAS storage):

[root@vcs15 ~]# hacli -cmd 'dd if=/dev/zero of=/shared/fss0/$(uname -n).dump count=1024 bs=4k'
vcs14:
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 2.04956 s, 2.0 MB/s

 vcs15:
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 2.07718 s, 2.0 MB/s

 [root@vcs15 ~]# hacli -cmd 'ls -ld /shared/fss0/*dump'
vcs14:
-rw-r--r-- 1 root root 4194304 Feb  4 13:28 /shared/fss0/vcs14.dump
-rw-r--r-- 1 root root 4194304 Feb  4 13:28 /shared/fss0/vcs15.dump

 vcs15:
-rw-r--r-- 1 root root 4194304 Feb  4 13:28 /shared/fss0/vcs14.dump
-rw-r--r-- 1 root root 4194304 Feb  4 13:28 /shared/fss0/vcs15.dump

[root@vcs15 ~]# hacli -cmd 'cksum /shared/fss0/*dump'
vcs14:
3413741448 4194304 /shared/fss0/vcs14.dump
3413741448 4194304 /shared/fss0/vcs15.dump

 vcs15:
3413741448 4194304 /shared/fss0/vcs14.dump
3413741448 4194304 /shared/fss0/vcs15.dump

  • Why 262142g?


256Tb (the max size of a VxFS filesystem) is 262144gb. When I attempted to create a Filesystem on my 256Tb volume, I was greeted with this error:

[root@vcs14 ~]# mkfs -t vxfs /dev/vx/dsk/FSS00dg/fss00_lv
UX:vxfs mkfs.vxfs: ERROR: V-3-26258: File system size cannot exceed 262144 GB

Also, I noticed that sometimes vxvol wouldn't start a 256Tb volume so I dropped the ball and went for 262142gb for both the volume and filesystem.

  • SFCFSHA software versions:


[root@vcs14 ~]# rpm -q VRTSvxfs VRTSvxvm
VRTSvxfs-6.1.1.202-RHEL6.x86_64
VRTSvxvm-6.1.1.100-RHEL6.x86_64

If you are on SF6.1+latest patches (As of Jan 2015) and you have a CFS filesystem that's over 100Tb in size, contact Veritas and ask for the following patch: fs-rhel6_x86_64-HotFix-6.1.1.007.tar.gz or fs-rhel6_x86_64-HotFix-6.1.1.202.tar.gz.
The tracking number for this issue is e3233276.
Without the patch, when unmounting the filesystem, you might get a cpu soft lockup. Nothing happens but it slows down the whole unmounting procedure. Here's the sign you're running into this issue:
[root@server101 CFSMount]# umount /vx/vol01

Message from syslogd@server101 at Oct 23 21:36:03 ...
kernel:BUG: soft lockup - CPU#0 stuck for 67s! [vx_ctl_thread:19524]


  • How to enable 'hacli -cmd':


[root@vcs14 ~]# hacli -cmd uname
VCS WARNING V-16-1-50112 hacli is disabled
[root@vcs14 ~]# haconf -makerw
[root@vcs14 ~]# haclus -modify HacliUserLevel COMMANDROOT
[root@vcs14 ~]# haclus -value HacliUserLevel
COMMANDROOT
[root@vcs14 ~]# haconf -dump -makero
[root@vcs14 ~]# haclus -value HacliUserLevel
COMMANDROOT
[root@vcs14 ~]# hacli -cmd uname
vcs15:
Linux

 vcs14:
Linux

Comments

Post a Comment

Popular posts from this blog

LSI MegaRaid HBA's, overheating and one ugly hack

Image
Summer is here! Like many others, I have LSI MegaRaid HBA's that I use in desktop machines. These things are great but they tend to overheat and quite a few people have reported high temperature  findings (97C reported by the chip when idle'ing in both my Dell T410 and my Dell T5610): $  sudo megaclisas-status -- Controller information -- -- ID | H/W Model                  | RAM    | Temp | Firmware      c0    | LSI MegaRAID SAS 9271-8i   | 1024MB | 97C  | FW: 23.32.0-0009  c1    | LSI MegaRAID SAS 9280-4i4e | 512MB  | N/A  | FW: 12.15.0-0205  [...] I had never really bothered about the temperature but when I started to rebuild my T410's boot LD (Raid-1) to swap the 2Tb drives with 4Tb drives I had, things started to get complicated quickly. As soon as the mirror started rebuilding, the ROC temp (sitting around 97C) skyrocketted to 102C and so...
Read more

Some Tips about running a Dell PowerEdge Tower Server as your workstation

Image
Some use workstations as servers. I'm using servers as workstations. Over the years, I've changed computing gear on quite a few occasions. I've been using Tower Servers for the past 5 years and would like to share some tips to help others. But why would anyone want to do that?? - Servers are well integrated systems and are usually seriously designed and tested. - They offer greater expandability (6x3.5" hotswap bays in my previous T410, now 8x3.5" in my T430). - They usually include some kind of Remote Access Card (RAC) which is great for remote'ing in when all else has failed. - I can get tons of server equipment on ebay that will be compatible with that system. - Where else can I get 192Gb of ECC DDR4 RDIMM, dual 6-core Xeons and 8 hotswap bays? Tip #1 : Choose your chassis with care. Not all servers are created equal: - Rack servers are usually thin and noisy (those 8k rpm fans have the job of cool that 2U enclosure). It is n...
Read more

VMWare Worksation 12 on Fedora Core 23/24 (fc23 and fc24)

Image
Update (20170321): VMware released VMW 12.5.4, which fixes all of the current issues with Fedora and the latest kernels (AFAIK). The hacks described on this page ARE NO LONGER NEEDED NOR MAINTAINED . Update (20160921): VMware released VMW 12.5, which is supposed to work fine out of the box.  I updated  Update_VMW.sh  to avoid patch anything -else- than VMW 11.x or 12.{0,1}. Please report back if you need assistance. Update (20160630):  I updated  Update_VMW.sh  to support Fedora Core 24 (fc24) for which the changes needed are the same. Please report back if you need assistance. Update (20160525): I updated my  Update_VMW.sh  script to patch the VMware kernel modules so they work on kernel 4.4.x and above (FC23+). Tested successfully with VMW 12.1.1. Update (20151209): This is still needed on 12.1.0 but I wrote a small tool to make everyone's life easier:  Update_VMW.sh VMWare workstation 12.0.1 Pro stopped working when I upgraded...
Read more