Storage Foundation HA 6.1 and Flexible Shared Storage
Veritas Flexible Storage Sharing
What is FSS?
You can read more about FSS here:
http://www.symantec.com/connect/blogs/configure-flexible-storage-sharing-using-veritas-operations-manager-61
Unlike the blog post above that uses the VOM GUI, I will be configuring FSS between two virtual machines in an old-fashioned style (CLI). I decided to write this blog because I couldn't find enough information in one place on how to setup FSS using the non-GUI stuff.
Lab setup and specs for each VM:
vcs15: RHEL6.6.z, 8Gb RAM, 32*8Tb local disks, 16*2Tb shared disks (these won't be used).
I won't go over the details of setting up a CFS cluster. Here are some of the things you would normally see after CFS has been setup (Think cfscluster config and cfscluster start):
[root@vcs14 ~]# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen 2cad02 membership 01
Port b gen 2cad01 membership 01
Port d gen 2cad05 membership 01
Port f gen 2cad0f membership 01
Port h gen 2cad05 membership 01
Port u gen 2cad0d membership 01
Port v gen 2cad09 membership 01
Port w gen 2cad0b membership 01
Port y gen 2cad08 membership 01
[root@vcs14 ~]# vxdctl -c mode
mode: enabled: cluster active - MASTER
master: vcs14
[root@vcs14 ~]# lltconfig -a list
Link 0 (bond0):
Node 0 vcs14 : 00:0C:29:D6:38:88 permanent
Node 1 vcs15 : 00:0C:29:B0:34:23
Link 1 (bond1):
Node 0 vcs14 : 00:0C:29:D6:38:92 permanent
Node 1 vcs15 : 00:0C:29:B0:34:2D
Link 2 (bond2):
Node 0 vcs14 : 00:0C:29:D6:38:9C permanent
Node 1 vcs15 : 00:0C:29:B0:34:37
[root@vcs14 ~]# cfscluster status
Node : vcs14
Cluster Manager : running
CVM state : running
MOUNT POINT SHARED VOLUME DISK GROUP STATUS
No mount point registered with cluster configuration
Node : vcs15
Cluster Manager : running
CVM state : running
MOUNT POINT SHARED VOLUME DISK GROUP STATUS
No mount point registered with cluster configuration
Here is vcs14:
[root@vcs14 ~]# vxdisk list
DEVICE TYPE DISK GROUP STATUS
disk_0 auto:cdsdisk - - online
disk_1 auto:cdsdisk - - online
[....]
disk_14 auto:cdsdisk - - online
disk_15 auto:cdsdisk - - online
sdaa auto:cdsdisk - - online
sdab auto:cdsdisk - - online
sdac auto:cdsdisk - - online
sdad auto:cdsdisk - - online
sdae auto:cdsdisk - - online
sdaf auto:cdsdisk - - online
sdag auto:cdsdisk - - online
sdah auto:cdsdisk - - online
[....]
sdt auto:cdsdisk - - online
sdu auto:cdsdisk - - online
sdv auto:cdsdisk - - online
sdw auto:cdsdisk - - online
sdx auto:cdsdisk - - online
sdy auto:cdsdisk - - online
sdz auto:cdsdisk - - online
And its sibling vcs15:
[root@vcs15 ~]# vxdisk list
DEVICE TYPE DISK GROUP STATUS
disk_0 auto:cdsdisk - - online
disk_1 auto:cdsdisk - - online
[....]
disk_14 auto:cdsdisk - - online
disk_15 auto:cdsdisk - - online
sdaa auto:cdsdisk - - online
sdab auto:cdsdisk - - online
sdac auto:cdsdisk - - online
sdad auto:cdsdisk - - online
sdae auto:cdsdisk - - online
sdaf auto:cdsdisk - - online
sdag auto:cdsdisk - - online
sdah auto:cdsdisk - - online
[....]
sdt auto:cdsdisk - - online
sdu auto:cdsdisk - - online
sdv auto:cdsdisk - - online
sdw auto:cdsdisk - - online
sdx auto:cdsdisk - - online
sdy auto:cdsdisk - - online
sdz auto:cdsdisk - - online
And here are our 8Tb disks:
[root@vcs14 ~]# lsblk -d|grep '^sd.*8T'
sdc 8:32 0 8T 0 disk
sdb 8:16 0 8T 0 disk
sdd 8:48 0 8T 0 disk
sdf 8:80 0 8T 0 disk
sde 8:64 0 8T 0 disk
sdg 8:96 0 8T 0 disk
sdh 8:112 0 8T 0 disk
sdi 8:128 0 8T 0 disk
sdl 8:176 0 8T 0 disk
sdm 8:192 0 8T 0 disk
sdj 8:144 0 8T 0 disk
sdk 8:160 0 8T 0 disk
sdp 8:240 0 8T 0 disk
sdn 8:208 0 8T 0 disk
sdr 65:16 0 8T 0 disk
sds 65:32 0 8T 0 disk
sdt 65:48 0 8T 0 disk
sdq 65:0 0 8T 0 disk
sdo 8:224 0 8T 0 disk
sdw 65:96 0 8T 0 disk
sdy 65:128 0 8T 0 disk
sdx 65:112 0 8T 0 disk
sdv 65:80 0 8T 0 disk
sdab 65:176 0 8T 0 disk
sdz 65:144 0 8T 0 disk
sdaa 65:160 0 8T 0 disk
sdac 65:192 0 8T 0 disk
sdae 65:224 0 8T 0 disk
sdad 65:208 0 8T 0 disk
sdu 65:64 0 8T 0 disk
sdaf 65:240 0 8T 0 disk
sdah 66:16 0 8T 0 disk
sdag 66:0 0 8T 0 disk
[root@vcs15 ~]# lsblk -d|grep '^sd.*8T'
sdb 8:16 0 8T 0 disk
sdc 8:32 0 8T 0 disk
sdd 8:48 0 8T 0 disk
sde 8:64 0 8T 0 disk
sdf 8:80 0 8T 0 disk
sdh 8:112 0 8T 0 disk
sdi 8:128 0 8T 0 disk
sdg 8:96 0 8T 0 disk
sdk 8:160 0 8T 0 disk
sdl 8:176 0 8T 0 disk
sdj 8:144 0 8T 0 disk
sdm 8:192 0 8T 0 disk
sdn 8:208 0 8T 0 disk
sdp 8:240 0 8T 0 disk
sdo 8:224 0 8T 0 disk
sds 65:32 0 8T 0 disk
sdq 65:0 0 8T 0 disk
sdt 65:48 0 8T 0 disk
sdr 65:16 0 8T 0 disk
sdu 65:64 0 8T 0 disk
sdv 65:80 0 8T 0 disk
sdx 65:112 0 8T 0 disk
sdw 65:96 0 8T 0 disk
sdz 65:144 0 8T 0 disk
sdy 65:128 0 8T 0 disk
sdab 65:176 0 8T 0 disk
sdaa 65:160 0 8T 0 disk
sdaf 65:240 0 8T 0 disk
sdae 65:224 0 8T 0 disk
sdad 65:208 0 8T 0 disk
sdac 65:192 0 8T 0 disk
sdag 66:0 0 8T 0 disk
sdah 66:16 0 8T 0 disk
[root@vcs14 ~]# lsblk -d|grep '^sd.*8T'|wc -l
33
There are 32^H^H^H33 8Tb disks per system, like I said. :)
Meaningful disk names (Optional).
I don't like the default disk names, so let's change them to something meaningful (remember to vxdisksetup the disks if you haven't already):
[root@vcs14 ~]# lsblk -d|grep '^sd.*8T'|awk '{ print $1 }' > /tmp/disks
[root@vcs14 ~]# seq -w 0 32 > /tmp/nums
[root@vcs14 ~]# paste /tmp/disks /tmp/nums |awk '{ print "vxdmpadm setattr dmpnode",$1,"name=ssd_" $2 }' > /tmp/names.sh
[root@vcs14 ~]# bash /tmp/names.sh
[root@vcs14 ~]# vxdisk list
DEVICE TYPE DISK GROUP STATUS
disk_0 auto:cdsdisk - - online
disk_1 auto:cdsdisk - - online
[....]
disk_14 auto:cdsdisk - - online
disk_15 auto:cdsdisk - - online
ssd_00 auto:cdsdisk - - online
ssd_01 auto:cdsdisk - - online
ssd_02 auto:cdsdisk - - online
ssd_03 auto:cdsdisk - - online
ssd_04 auto:cdsdisk - - online
[....]
ssd_28 auto:cdsdisk - - online
ssd_29 auto:cdsdisk - - online
ssd_30 auto:cdsdisk - - online
ssd_31 auto:cdsdisk - - online
ssd_32 auto:cdsdisk - - online
Let's do the same for vcs15 ( you get the picture..) :
[root@vcs15 ~]# lsblk -d|grep '^sd.*8T'|awk '{ print $1 }' > /tmp/disks
[root@vcs15 ~]# seq -w 0 32 > /tmp/nums
[root@vcs15 ~]# paste /tmp/disks /tmp/nums |awk '{ print "vxdmpadm setattr dmpnode",$1,"name=ssd_" $2 }' > /tmp/names.sh
[root@vcs15 ~]# bash /tmp/names.sh
[root@vcs15 ~]# vxdisk list
DEVICE TYPE DISK GROUP STATUS
disk_0 auto:cdsdisk - - online
disk_1 auto:cdsdisk - - online
[....]
disk_14 auto:cdsdisk - - online
disk_15 auto:cdsdisk - - online
ssd_00 auto:cdsdisk - - online
ssd_01 auto:cdsdisk - - online
ssd_02 auto:cdsdisk - - online
ssd_03 auto:cdsdisk - - online
ssd_04 auto:cdsdisk - - online
[....]
ssd_28 auto:cdsdisk - - online
ssd_29 auto:cdsdisk - - online
ssd_30 auto:cdsdisk - - online
ssd_31 auto:cdsdisk - - online
ssd_32 auto:cdsdisk - - online
Now export the disks to the remote cluster nodes:
[root@vcs14 ~]# vxdisk list|grep ^ssd|awk '{ print $1}' |xargs -n1 vxdisk export
[root@vcs15 ~]# vxdisk list|grep ^ssd|awk '{ print $1}' |xargs -n1 vxdisk export
Look at the vxdisk output, now each node sees the disks that were exported on its sibling:
On vcs14, we see the disks from vcs15
[root@vcs14 ~]# vxdisk list|egrep 'DEVICE|ssd'
DEVICE TYPE DISK GROUP STATUS
ssd_00 auto:cdsdisk - - online exported
ssd_01 auto:cdsdisk - - online exported
ssd_02 auto:cdsdisk - - online exported
ssd_03 auto:cdsdisk - - online exported
ssd_04 auto:cdsdisk - - online exported
ssd_05 auto:cdsdisk - - online exported
ssd_06 auto:cdsdisk - - online exported
ssd_07 auto:cdsdisk - - online exported
ssd_08 auto:cdsdisk - - online exported
ssd_09 auto:cdsdisk - - online exported
ssd_10 auto:cdsdisk - - online exported
ssd_11 auto:cdsdisk - - online exported
ssd_12 auto:cdsdisk - - online exported
ssd_13 auto:cdsdisk - - online exported
ssd_14 auto:cdsdisk - - online exported
ssd_15 auto:cdsdisk - - online exported
ssd_16 auto:cdsdisk - - online exported
ssd_17 auto:cdsdisk - - online exported
ssd_18 auto:cdsdisk - - online exported
ssd_19 auto:cdsdisk - - online exported
ssd_20 auto:cdsdisk - - online exported
ssd_21 auto:cdsdisk - - online exported
ssd_22 auto:cdsdisk - - online exported
ssd_23 auto:cdsdisk - - online exported
ssd_24 auto:cdsdisk - - online exported
ssd_25 auto:cdsdisk - - online exported
ssd_26 auto:cdsdisk - - online exported
ssd_27 auto:cdsdisk - - online exported
ssd_28 auto:cdsdisk - - online exported
ssd_29 auto:cdsdisk - - online exported
ssd_30 auto:cdsdisk - - online exported
ssd_31 auto:cdsdisk - - online exported
ssd_32 auto:cdsdisk - - online exported
vcs15_ssd_00 auto:cdsdisk - - online remote
vcs15_ssd_01 auto:cdsdisk - - online remote
vcs15_ssd_02 auto:cdsdisk - - online remote
vcs15_ssd_03 auto:cdsdisk - - online remote
vcs15_ssd_04 auto:cdsdisk - - online remote
vcs15_ssd_05 auto:cdsdisk - - online remote
vcs15_ssd_06 auto:cdsdisk - - online remote
vcs15_ssd_07 auto:cdsdisk - - online remote
vcs15_ssd_08 auto:cdsdisk - - online remote
vcs15_ssd_09 auto:cdsdisk - - online remote
vcs15_ssd_10 auto:cdsdisk - - online remote
vcs15_ssd_11 auto:cdsdisk - - online remote
vcs15_ssd_12 auto:cdsdisk - - online remote
vcs15_ssd_13 auto:cdsdisk - - online remote
vcs15_ssd_14 auto:cdsdisk - - online remote
vcs15_ssd_15 auto:cdsdisk - - online remote
vcs15_ssd_16 auto:cdsdisk - - online remote
vcs15_ssd_17 auto:cdsdisk - - online remote
vcs15_ssd_18 auto:cdsdisk - - online remote
vcs15_ssd_19 auto:cdsdisk - - online remote
vcs15_ssd_20 auto:cdsdisk - - online remote
vcs15_ssd_21 auto:cdsdisk - - online remote
vcs15_ssd_22 auto:cdsdisk - - online remote
vcs15_ssd_23 auto:cdsdisk - - online remote
vcs15_ssd_24 auto:cdsdisk - - online remote
vcs15_ssd_25 auto:cdsdisk - - online remote
vcs15_ssd_26 auto:cdsdisk - - online remote
vcs15_ssd_27 auto:cdsdisk - - online remote
vcs15_ssd_28 auto:cdsdisk - - online remote
vcs15_ssd_29 auto:cdsdisk - - online remote
vcs15_ssd_30 auto:cdsdisk - - online remote
vcs15_ssd_31 auto:cdsdisk - - online remote
vcs15_ssd_32 auto:cdsdisk - - online remote
And on vx15 we have vcs14's disks:
[root@vcs15 ~]# vxdisk list|egrep 'DEVICE|ssd'
DEVICE TYPE DISK GROUP STATUS
ssd_00 auto:cdsdisk - - online exported
ssd_01 auto:cdsdisk - - online exported
ssd_02 auto:cdsdisk - - online exported
ssd_03 auto:cdsdisk - - online exported
ssd_04 auto:cdsdisk - - online exported
ssd_05 auto:cdsdisk - - online exported
ssd_06 auto:cdsdisk - - online exported
ssd_07 auto:cdsdisk - - online exported
ssd_08 auto:cdsdisk - - online exported
ssd_09 auto:cdsdisk - - online exported
ssd_10 auto:cdsdisk - - online exported
ssd_11 auto:cdsdisk - - online exported
ssd_12 auto:cdsdisk - - online exported
ssd_13 auto:cdsdisk - - online exported
ssd_14 auto:cdsdisk - - online exported
ssd_15 auto:cdsdisk - - online exported
ssd_16 auto:cdsdisk - - online exported
ssd_17 auto:cdsdisk - - online exported
ssd_18 auto:cdsdisk - - online exported
ssd_19 auto:cdsdisk - - online exported
ssd_20 auto:cdsdisk - - online exported
ssd_21 auto:cdsdisk - - online exported
ssd_22 auto:cdsdisk - - online exported
ssd_23 auto:cdsdisk - - online exported
ssd_24 auto:cdsdisk - - online exported
ssd_25 auto:cdsdisk - - online exported
ssd_26 auto:cdsdisk - - online exported
ssd_27 auto:cdsdisk - - online exported
ssd_28 auto:cdsdisk - - online exported
ssd_29 auto:cdsdisk - - online exported
ssd_30 auto:cdsdisk - - online exported
ssd_31 auto:cdsdisk - - online exported
ssd_32 auto:cdsdisk - - online exported
vcs14_ssd_00 auto:cdsdisk - - online remote
vcs14_ssd_01 auto:cdsdisk - - online remote
vcs14_ssd_02 auto:cdsdisk - - online remote
vcs14_ssd_03 auto:cdsdisk - - online remote
vcs14_ssd_04 auto:cdsdisk - - online remote
vcs14_ssd_05 auto:cdsdisk - - online remote
vcs14_ssd_06 auto:cdsdisk - - online remote
vcs14_ssd_07 auto:cdsdisk - - online remote
vcs14_ssd_08 auto:cdsdisk - - online remote
vcs14_ssd_09 auto:cdsdisk - - online remote
vcs14_ssd_10 auto:cdsdisk - - online remote
vcs14_ssd_11 auto:cdsdisk - - online remote
vcs14_ssd_12 auto:cdsdisk - - online remote
vcs14_ssd_13 auto:cdsdisk - - online remote
vcs14_ssd_14 auto:cdsdisk - - online remote
vcs14_ssd_15 auto:cdsdisk - - online remote
vcs14_ssd_16 auto:cdsdisk - - online remote
vcs14_ssd_17 auto:cdsdisk - - online remote
vcs14_ssd_18 auto:cdsdisk - - online remote
vcs14_ssd_19 auto:cdsdisk - - online remote
vcs14_ssd_20 auto:cdsdisk - - online remote
vcs14_ssd_21 auto:cdsdisk - - online remote
vcs14_ssd_22 auto:cdsdisk - - online remote
vcs14_ssd_23 auto:cdsdisk - - online remote
vcs14_ssd_24 auto:cdsdisk - - online remote
vcs14_ssd_25 auto:cdsdisk - - online remote
vcs14_ssd_26 auto:cdsdisk - - online remote
vcs14_ssd_27 auto:cdsdisk - - online remote
vcs14_ssd_28 auto:cdsdisk - - online remote
vcs14_ssd_29 auto:cdsdisk - - online remote
vcs14_ssd_30 auto:cdsdisk - - online remote
vcs14_ssd_31 auto:cdsdisk - - online remote
vcs14_ssd_32 auto:cdsdisk - - online remote
Configure an FSS diskgroup (do this on only one node):
[root@vcs14 ~]# vxdisk list|egrep 'ssd'|awk '{ print $1}' > /tmp/alldisks
* Create the diskgroup on one node (notice how it shows up on both nodes):
[root@vcs14 ~]# vxdg -o fss -s init FSS00dg $(cat /tmp/alldisks )
[root@vcs14 ~]# vxdg list
NAME STATE ID
FSS00dg enabled,shared,cds 1423072143.104.vcs14
[root@vcs15 ~]# vxdg list
NAME STATE ID
FSS00dg enabled,shared,cds 1423072143.104.vcs14
And some info from vxdisk list:
[root@vcs14 ~]# vxdisk list|egrep '^(DEVI|ssd|vcs)'
DEVICE TYPE DISK GROUP STATUS
ssd_00 auto:cdsdisk ssd_00 FSS00dg online exported shared
ssd_01 auto:cdsdisk ssd_01 FSS00dg online exported shared
ssd_02 auto:cdsdisk ssd_02 FSS00dg online exported shared
ssd_03 auto:cdsdisk ssd_03 FSS00dg online exported shared
ssd_04 auto:cdsdisk ssd_04 FSS00dg online exported shared
ssd_05 auto:cdsdisk ssd_05 FSS00dg online exported shared
ssd_06 auto:cdsdisk ssd_06 FSS00dg online exported shared
ssd_07 auto:cdsdisk ssd_07 FSS00dg online exported shared
ssd_08 auto:cdsdisk ssd_08 FSS00dg online exported shared
ssd_09 auto:cdsdisk ssd_09 FSS00dg online exported shared
ssd_10 auto:cdsdisk ssd_10 FSS00dg online exported shared
ssd_11 auto:cdsdisk ssd_11 FSS00dg online exported shared
ssd_12 auto:cdsdisk ssd_12 FSS00dg online exported shared
ssd_13 auto:cdsdisk ssd_13 FSS00dg online exported shared
ssd_14 auto:cdsdisk ssd_14 FSS00dg online exported shared
ssd_15 auto:cdsdisk ssd_15 FSS00dg online exported shared
ssd_16 auto:cdsdisk ssd_16 FSS00dg online exported shared
ssd_17 auto:cdsdisk ssd_17 FSS00dg online exported shared
ssd_18 auto:cdsdisk ssd_18 FSS00dg online exported shared
ssd_19 auto:cdsdisk ssd_19 FSS00dg online exported shared
ssd_20 auto:cdsdisk ssd_20 FSS00dg online exported shared
ssd_21 auto:cdsdisk ssd_21 FSS00dg online exported shared
ssd_22 auto:cdsdisk ssd_22 FSS00dg online exported shared
ssd_23 auto:cdsdisk ssd_23 FSS00dg online exported shared
ssd_24 auto:cdsdisk ssd_24 FSS00dg online exported shared
ssd_25 auto:cdsdisk ssd_25 FSS00dg online exported shared
ssd_26 auto:cdsdisk ssd_26 FSS00dg online exported shared
ssd_27 auto:cdsdisk ssd_27 FSS00dg online exported shared
ssd_28 auto:cdsdisk ssd_28 FSS00dg online exported shared
ssd_29 auto:cdsdisk ssd_29 FSS00dg online exported shared
ssd_30 auto:cdsdisk ssd_30 FSS00dg online exported shared
ssd_31 auto:cdsdisk ssd_31 FSS00dg online exported shared
ssd_32 auto:cdsdisk ssd_32 FSS00dg online exported shared
vcs15_ssd_00 auto:cdsdisk vcs15_ssd_00 FSS00dg online shared remote
vcs15_ssd_01 auto:cdsdisk vcs15_ssd_01 FSS00dg online shared remote
vcs15_ssd_02 auto:cdsdisk vcs15_ssd_02 FSS00dg online shared remote
vcs15_ssd_03 auto:cdsdisk vcs15_ssd_03 FSS00dg online shared remote
vcs15_ssd_04 auto:cdsdisk vcs15_ssd_04 FSS00dg online shared remote
vcs15_ssd_05 auto:cdsdisk vcs15_ssd_05 FSS00dg online shared remote
vcs15_ssd_06 auto:cdsdisk vcs15_ssd_06 FSS00dg online shared remote
vcs15_ssd_07 auto:cdsdisk vcs15_ssd_07 FSS00dg online shared remote
vcs15_ssd_08 auto:cdsdisk vcs15_ssd_08 FSS00dg online shared remote
vcs15_ssd_09 auto:cdsdisk vcs15_ssd_09 FSS00dg online shared remote
vcs15_ssd_10 auto:cdsdisk vcs15_ssd_10 FSS00dg online shared remote
vcs15_ssd_11 auto:cdsdisk vcs15_ssd_11 FSS00dg online shared remote
vcs15_ssd_12 auto:cdsdisk vcs15_ssd_12 FSS00dg online shared remote
vcs15_ssd_13 auto:cdsdisk vcs15_ssd_13 FSS00dg online shared remote
vcs15_ssd_14 auto:cdsdisk vcs15_ssd_14 FSS00dg online shared remote
vcs15_ssd_15 auto:cdsdisk vcs15_ssd_15 FSS00dg online shared remote
vcs15_ssd_16 auto:cdsdisk vcs15_ssd_16 FSS00dg online shared remote
vcs15_ssd_17 auto:cdsdisk vcs15_ssd_17 FSS00dg online shared remote
vcs15_ssd_18 auto:cdsdisk vcs15_ssd_18 FSS00dg online shared remote
vcs15_ssd_19 auto:cdsdisk vcs15_ssd_19 FSS00dg online shared remote
vcs15_ssd_20 auto:cdsdisk vcs15_ssd_20 FSS00dg online shared remote
vcs15_ssd_21 auto:cdsdisk vcs15_ssd_21 FSS00dg online shared remote
vcs15_ssd_22 auto:cdsdisk vcs15_ssd_22 FSS00dg online shared remote
vcs15_ssd_23 auto:cdsdisk vcs15_ssd_23 FSS00dg online shared remote
vcs15_ssd_24 auto:cdsdisk vcs15_ssd_24 FSS00dg online shared remote
vcs15_ssd_25 auto:cdsdisk vcs15_ssd_25 FSS00dg online shared remote
vcs15_ssd_26 auto:cdsdisk vcs15_ssd_26 FSS00dg online shared remote
vcs15_ssd_27 auto:cdsdisk vcs15_ssd_27 FSS00dg online shared remote
vcs15_ssd_28 auto:cdsdisk vcs15_ssd_28 FSS00dg online shared remote
vcs15_ssd_29 auto:cdsdisk vcs15_ssd_29 FSS00dg online shared remote
vcs15_ssd_30 auto:cdsdisk vcs15_ssd_30 FSS00dg online shared remote
vcs15_ssd_31 auto:cdsdisk vcs15_ssd_31 FSS00dg online shared remote
vcs15_ssd_32 auto:cdsdisk vcs15_ssd_32 FSS00dg online shared remote
Create a volume in the FSS diskgroup to hold our data:
[root@vcs14 ~]# vxdctl -c mode
mode: enabled: cluster active - MASTER
master: vcs14
[root@vcs14 ~]# vxassist -g FSS00dg maxsize nmirror=2 layout=mirror-stripe stripeunit=512k
Maximum volume size: 566933245952 (276822874Mb)
* To be on the safe side: 256tb = 262144g, pick 262144g - 2g : 262142g
(See the footnotes for more information on this)
[root@vcs14 ~]# vxassist -g FSS00dg make fss0_lv 262142g nmirror=2 layout=mirror-stripe stripeunit=512k
Let's look at the newly created volume:
[root@vcs14 ~]# vxprint -Ath
Disk group: FSS00dg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME RVG KSTATE STATE NVOLUME
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
EX NAME ASSOC VC PERMS MODE STATE
SR NAME KSTATE
dg FSS00dg default default 35000 1423072143.104.vcs14
dm ssd_00 ssd_00 auto 65536 17179803344 -
dm ssd_01 ssd_01 auto 65536 17179803344 -
dm ssd_02 ssd_02 auto 65536 17179803344 -
dm ssd_03 ssd_03 auto 65536 17179803344 -
dm ssd_04 ssd_04 auto 65536 17179803344 -
[....]
dm ssd_28 ssd_28 auto 65536 17179803344 -
dm ssd_29 ssd_29 auto 65536 17179803344 -
dm ssd_30 ssd_30 auto 65536 17179803344 -
dm ssd_31 ssd_31 auto 65536 17179803344 -
dm ssd_32 ssd_32 auto 65536 17179803344 -
dm vcs15_ssd_00 vcs15_ssd_00 auto 65536 17179803344 REMOTE
dm vcs15_ssd_01 vcs15_ssd_01 auto 65536 17179803344 REMOTE
dm vcs15_ssd_02 vcs15_ssd_02 auto 65536 17179803344 REMOTE
dm vcs15_ssd_03 vcs15_ssd_03 auto 65536 17179803344 REMOTE
dm vcs15_ssd_04 vcs15_ssd_04 auto 65536 17179803344 REMOTE
[....]
dm vcs15_ssd_28 vcs15_ssd_28 auto 65536 17179803344 REMOTE
dm vcs15_ssd_29 vcs15_ssd_29 auto 65536 17179803344 REMOTE
dm vcs15_ssd_30 vcs15_ssd_30 auto 65536 17179803344 REMOTE
dm vcs15_ssd_31 vcs15_ssd_31 auto 65536 17179803344 REMOTE
dm vcs15_ssd_32 vcs15_ssd_32 auto 65536 17179803344 REMOTE
v fss0_lv - ENABLED ACTIVE 549755813888 SELECT - fsgen
pl fss0_lv-01 fss0_lv ENABLED ACTIVE 549755814912 STRIPE 3/1024 RW
sd vcs15_ssd_00-01 fss0_lv-01 vcs15_ssd_00 0 17179803344 0/0 vcs15_ssd_00 ENA
sd vcs15_ssd_03-01 fss0_lv-01 vcs15_ssd_03 0 17179803344 0/17179803344 vcs15_ssd_03 ENA
sd vcs15_ssd_06-01 fss0_lv-01 vcs15_ssd_06 0 17179803344 0/34359606688 vcs15_ssd_06 ENA
sd vcs15_ssd_09-01 fss0_lv-01 vcs15_ssd_09 0 17179803344 0/51539410032 vcs15_ssd_09 ENA
sd vcs15_ssd_12-01 fss0_lv-01 vcs15_ssd_12 0 17179803344 0/68719213376 vcs15_ssd_12 ENA
sd vcs15_ssd_15-01 fss0_lv-01 vcs15_ssd_15 0 17179803344 0/85899016720 vcs15_ssd_15 ENA
sd vcs15_ssd_18-01 fss0_lv-01 vcs15_ssd_18 0 17179803344 0/103078820064 vcs15_ssd_18 ENA
sd vcs15_ssd_21-01 fss0_lv-01 vcs15_ssd_21 0 17179803344 0/120258623408 vcs15_ssd_21 ENA
sd vcs15_ssd_24-01 fss0_lv-01 vcs15_ssd_24 0 17179803344 0/137438426752 vcs15_ssd_24 ENA
sd vcs15_ssd_27-01 fss0_lv-01 vcs15_ssd_27 0 17179803344 0/154618230096 vcs15_ssd_27 ENA
sd vcs15_ssd_30-01 fss0_lv-01 vcs15_ssd_30 0 11453904864 0/171798033440 vcs15_ssd_30 ENA
sd vcs15_ssd_01-01 fss0_lv-01 vcs15_ssd_01 0 17179803344 1/0 vcs15_ssd_01 ENA
sd vcs15_ssd_04-01 fss0_lv-01 vcs15_ssd_04 0 17179803344 1/17179803344 vcs15_ssd_04 ENA
sd vcs15_ssd_07-01 fss0_lv-01 vcs15_ssd_07 0 17179803344 1/34359606688 vcs15_ssd_07 ENA
sd vcs15_ssd_10-01 fss0_lv-01 vcs15_ssd_10 0 17179803344 1/51539410032 vcs15_ssd_10 ENA
sd vcs15_ssd_13-01 fss0_lv-01 vcs15_ssd_13 0 17179803344 1/68719213376 vcs15_ssd_13 ENA
sd vcs15_ssd_16-01 fss0_lv-01 vcs15_ssd_16 0 17179803344 1/85899016720 vcs15_ssd_16 ENA
sd vcs15_ssd_19-01 fss0_lv-01 vcs15_ssd_19 0 17179803344 1/103078820064 vcs15_ssd_19 ENA
sd vcs15_ssd_22-01 fss0_lv-01 vcs15_ssd_22 0 17179803344 1/120258623408 vcs15_ssd_22 ENA
sd vcs15_ssd_25-01 fss0_lv-01 vcs15_ssd_25 0 17179803344 1/137438426752 vcs15_ssd_25 ENA
sd vcs15_ssd_28-01 fss0_lv-01 vcs15_ssd_28 0 17179803344 1/154618230096 vcs15_ssd_28 ENA
sd vcs15_ssd_31-01 fss0_lv-01 vcs15_ssd_31 0 11453904864 1/171798033440 vcs15_ssd_31 ENA
sd vcs15_ssd_02-01 fss0_lv-01 vcs15_ssd_02 0 17179803344 2/0 vcs15_ssd_02 ENA
sd vcs15_ssd_05-01 fss0_lv-01 vcs15_ssd_05 0 17179803344 2/17179803344 vcs15_ssd_05 ENA
sd vcs15_ssd_08-01 fss0_lv-01 vcs15_ssd_08 0 17179803344 2/34359606688 vcs15_ssd_08 ENA
sd vcs15_ssd_11-01 fss0_lv-01 vcs15_ssd_11 0 17179803344 2/51539410032 vcs15_ssd_11 ENA
sd vcs15_ssd_14-01 fss0_lv-01 vcs15_ssd_14 0 17179803344 2/68719213376 vcs15_ssd_14 ENA
sd vcs15_ssd_17-01 fss0_lv-01 vcs15_ssd_17 0 17179803344 2/85899016720 vcs15_ssd_17 ENA
sd vcs15_ssd_20-01 fss0_lv-01 vcs15_ssd_20 0 17179803344 2/103078820064 vcs15_ssd_20 ENA
sd vcs15_ssd_23-01 fss0_lv-01 vcs15_ssd_23 0 17179803344 2/120258623408 vcs15_ssd_23 ENA
sd vcs15_ssd_26-01 fss0_lv-01 vcs15_ssd_26 0 17179803344 2/137438426752 vcs15_ssd_26 ENA
sd vcs15_ssd_29-01 fss0_lv-01 vcs15_ssd_29 0 17179803344 2/154618230096 vcs15_ssd_29 ENA
sd vcs15_ssd_32-01 fss0_lv-01 vcs15_ssd_32 0 11453904864 2/171798033440 vcs15_ssd_32 ENA
pl fss0_lv-02 fss0_lv ENABLED ACTIVE 549755814912 STRIPE 3/1024 RW
sd ssd_00-01 fss0_lv-02 ssd_00 0 17179803344 0/0 ssd_00 ENA
sd ssd_03-01 fss0_lv-02 ssd_03 0 17179803344 0/17179803344 ssd_03 ENA
sd ssd_06-01 fss0_lv-02 ssd_06 0 17179803344 0/34359606688 ssd_06 ENA
sd ssd_09-01 fss0_lv-02 ssd_09 0 17179803344 0/51539410032 ssd_09 ENA
sd ssd_12-01 fss0_lv-02 ssd_12 0 17179803344 0/68719213376 ssd_12 ENA
sd ssd_15-01 fss0_lv-02 ssd_15 0 17179803344 0/85899016720 ssd_15 ENA
sd ssd_18-01 fss0_lv-02 ssd_18 0 17179803344 0/103078820064 ssd_18 ENA
sd ssd_21-01 fss0_lv-02 ssd_21 0 17179803344 0/120258623408 ssd_21 ENA
sd ssd_24-01 fss0_lv-02 ssd_24 0 17179803344 0/137438426752 ssd_24 ENA
sd ssd_27-01 fss0_lv-02 ssd_27 0 17179803344 0/154618230096 ssd_27 ENA
sd ssd_30-01 fss0_lv-02 ssd_30 0 11453904864 0/171798033440 ssd_30 ENA
sd ssd_01-01 fss0_lv-02 ssd_01 0 17179803344 1/0 ssd_01 ENA
sd ssd_04-01 fss0_lv-02 ssd_04 0 17179803344 1/17179803344 ssd_04 ENA
sd ssd_07-01 fss0_lv-02 ssd_07 0 17179803344 1/34359606688 ssd_07 ENA
sd ssd_10-01 fss0_lv-02 ssd_10 0 17179803344 1/51539410032 ssd_10 ENA
sd ssd_13-01 fss0_lv-02 ssd_13 0 17179803344 1/68719213376 ssd_13 ENA
sd ssd_16-01 fss0_lv-02 ssd_16 0 17179803344 1/85899016720 ssd_16 ENA
sd ssd_19-01 fss0_lv-02 ssd_19 0 17179803344 1/103078820064 ssd_19 ENA
sd ssd_22-01 fss0_lv-02 ssd_22 0 17179803344 1/120258623408 ssd_22 ENA
sd ssd_25-01 fss0_lv-02 ssd_25 0 17179803344 1/137438426752 ssd_25 ENA
sd ssd_28-01 fss0_lv-02 ssd_28 0 17179803344 1/154618230096 ssd_28 ENA
sd ssd_31-01 fss0_lv-02 ssd_31 0 11453904864 1/171798033440 ssd_31 ENA
sd ssd_02-01 fss0_lv-02 ssd_02 0 17179803344 2/0 ssd_02 ENA
sd ssd_05-01 fss0_lv-02 ssd_05 0 17179803344 2/17179803344 ssd_05 ENA
sd ssd_08-01 fss0_lv-02 ssd_08 0 17179803344 2/34359606688 ssd_08 ENA
sd ssd_11-01 fss0_lv-02 ssd_11 0 17179803344 2/51539410032 ssd_11 ENA
sd ssd_14-01 fss0_lv-02 ssd_14 0 17179803344 2/68719213376 ssd_14 ENA
sd ssd_17-01 fss0_lv-02 ssd_17 0 17179803344 2/85899016720 ssd_17 ENA
sd ssd_20-01 fss0_lv-02 ssd_20 0 17179803344 2/103078820064 ssd_20 ENA
sd ssd_23-01 fss0_lv-02 ssd_23 0 17179803344 2/120258623408 ssd_23 ENA
sd ssd_26-01 fss0_lv-02 ssd_26 0 17179803344 2/137438426752 ssd_26 ENA
sd ssd_29-01 fss0_lv-02 ssd_29 0 17179803344 2/154618230096 ssd_29 ENA
sd ssd_32-01 fss0_lv-02 ssd_32 0 11453904864 2/171798033440 ssd_32 ENA
dc fss0_lv_dco fss0_lv fss0_lv_dcl
v fss0_lv_dcl - ENABLED ACTIVE 18980992 SELECT - gen
pl fss0_lv_dcl-01 fss0_lv_dcl ENABLED ACTIVE 18980992 CONCAT - RW
sd ssd_30-02 fss0_lv_dcl-01 ssd_30 11453904864 18980992 0 ssd_30 ENA
pl fss0_lv_dcl-02 fss0_lv_dcl ENABLED ACTIVE 18980992 CONCAT - RW
sd vcs15_ssd_30-02 fss0_lv_dcl-02 vcs15_ssd_30 11453904864 18980992 0 vcs15_ssd_30 ENA
It looks good! Let's do the same on the other node:
[root@vcs15 ~]# vxprint -Ath
Disk group: FSS00dg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME RVG KSTATE STATE NVOLUME
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
EX NAME ASSOC VC PERMS MODE STATE
SR NAME KSTATE
dg FSS00dg default default 35000 1423072143.104.vcs14
dm ssd_00 vcs14_ssd_00 auto 65536 17179803344 REMOTE
dm ssd_01 vcs14_ssd_01 auto 65536 17179803344 REMOTE
dm ssd_02 vcs14_ssd_02 auto 65536 17179803344 REMOTE
dm ssd_03 vcs14_ssd_03 auto 65536 17179803344 REMOTE
dm ssd_04 vcs14_ssd_04 auto 65536 17179803344 REMOTE
[....]
dm ssd_28 vcs14_ssd_28 auto 65536 17179803344 REMOTE
dm ssd_29 vcs14_ssd_29 auto 65536 17179803344 REMOTE
dm ssd_30 vcs14_ssd_30 auto 65536 17179803344 REMOTE
dm ssd_31 vcs14_ssd_31 auto 65536 17179803344 REMOTE
dm ssd_32 vcs14_ssd_32 auto 65536 17179803344 REMOTE
dm vcs15_ssd_00 ssd_00 auto 65536 17179803344 -
dm vcs15_ssd_01 ssd_01 auto 65536 17179803344 -
dm vcs15_ssd_02 ssd_02 auto 65536 17179803344 -
dm vcs15_ssd_03 ssd_03 auto 65536 17179803344 -
dm vcs15_ssd_04 ssd_04 auto 65536 17179803344 -
[....]
dm vcs15_ssd_28 ssd_28 auto 65536 17179803344 -
dm vcs15_ssd_29 ssd_29 auto 65536 17179803344 -
dm vcs15_ssd_30 ssd_30 auto 65536 17179803344 -
dm vcs15_ssd_31 ssd_31 auto 65536 17179803344 -
dm vcs15_ssd_32 ssd_32 auto 65536 17179803344 -
v fss0_lv - ENABLED ACTIVE 549755813888 SELECT - fsgen
pl fss0_lv-01 fss0_lv ENABLED ACTIVE 549755814912 STRIPE 3/1024 RW
sd vcs15_ssd_00-01 fss0_lv-01 vcs15_ssd_00 0 17179803344 0/0 ssd_00 ENA
sd vcs15_ssd_03-01 fss0_lv-01 vcs15_ssd_03 0 17179803344 0/17179803344 ssd_03 ENA
sd vcs15_ssd_06-01 fss0_lv-01 vcs15_ssd_06 0 17179803344 0/34359606688 ssd_06 ENA
sd vcs15_ssd_09-01 fss0_lv-01 vcs15_ssd_09 0 17179803344 0/51539410032 ssd_09 ENA
sd vcs15_ssd_12-01 fss0_lv-01 vcs15_ssd_12 0 17179803344 0/68719213376 ssd_12 ENA
sd vcs15_ssd_15-01 fss0_lv-01 vcs15_ssd_15 0 17179803344 0/85899016720 ssd_15 ENA
sd vcs15_ssd_18-01 fss0_lv-01 vcs15_ssd_18 0 17179803344 0/103078820064 ssd_18 ENA
sd vcs15_ssd_21-01 fss0_lv-01 vcs15_ssd_21 0 17179803344 0/120258623408 ssd_21 ENA
sd vcs15_ssd_24-01 fss0_lv-01 vcs15_ssd_24 0 17179803344 0/137438426752 ssd_24 ENA
sd vcs15_ssd_27-01 fss0_lv-01 vcs15_ssd_27 0 17179803344 0/154618230096 ssd_27 ENA
sd vcs15_ssd_30-01 fss0_lv-01 vcs15_ssd_30 0 11453904864 0/171798033440 ssd_30 ENA
sd vcs15_ssd_01-01 fss0_lv-01 vcs15_ssd_01 0 17179803344 1/0 ssd_01 ENA
sd vcs15_ssd_04-01 fss0_lv-01 vcs15_ssd_04 0 17179803344 1/17179803344 ssd_04 ENA
sd vcs15_ssd_07-01 fss0_lv-01 vcs15_ssd_07 0 17179803344 1/34359606688 ssd_07 ENA
sd vcs15_ssd_10-01 fss0_lv-01 vcs15_ssd_10 0 17179803344 1/51539410032 ssd_10 ENA
sd vcs15_ssd_13-01 fss0_lv-01 vcs15_ssd_13 0 17179803344 1/68719213376 ssd_13 ENA
sd vcs15_ssd_16-01 fss0_lv-01 vcs15_ssd_16 0 17179803344 1/85899016720 ssd_16 ENA
sd vcs15_ssd_19-01 fss0_lv-01 vcs15_ssd_19 0 17179803344 1/103078820064 ssd_19 ENA
sd vcs15_ssd_22-01 fss0_lv-01 vcs15_ssd_22 0 17179803344 1/120258623408 ssd_22 ENA
sd vcs15_ssd_25-01 fss0_lv-01 vcs15_ssd_25 0 17179803344 1/137438426752 ssd_25 ENA
sd vcs15_ssd_28-01 fss0_lv-01 vcs15_ssd_28 0 17179803344 1/154618230096 ssd_28 ENA
sd vcs15_ssd_31-01 fss0_lv-01 vcs15_ssd_31 0 11453904864 1/171798033440 ssd_31 ENA
sd vcs15_ssd_02-01 fss0_lv-01 vcs15_ssd_02 0 17179803344 2/0 ssd_02 ENA
sd vcs15_ssd_05-01 fss0_lv-01 vcs15_ssd_05 0 17179803344 2/17179803344 ssd_05 ENA
sd vcs15_ssd_08-01 fss0_lv-01 vcs15_ssd_08 0 17179803344 2/34359606688 ssd_08 ENA
sd vcs15_ssd_11-01 fss0_lv-01 vcs15_ssd_11 0 17179803344 2/51539410032 ssd_11 ENA
sd vcs15_ssd_14-01 fss0_lv-01 vcs15_ssd_14 0 17179803344 2/68719213376 ssd_14 ENA
sd vcs15_ssd_17-01 fss0_lv-01 vcs15_ssd_17 0 17179803344 2/85899016720 ssd_17 ENA
sd vcs15_ssd_20-01 fss0_lv-01 vcs15_ssd_20 0 17179803344 2/103078820064 ssd_20 ENA
sd vcs15_ssd_23-01 fss0_lv-01 vcs15_ssd_23 0 17179803344 2/120258623408 ssd_23 ENA
sd vcs15_ssd_26-01 fss0_lv-01 vcs15_ssd_26 0 17179803344 2/137438426752 ssd_26 ENA
sd vcs15_ssd_29-01 fss0_lv-01 vcs15_ssd_29 0 17179803344 2/154618230096 ssd_29 ENA
sd vcs15_ssd_32-01 fss0_lv-01 vcs15_ssd_32 0 11453904864 2/171798033440 ssd_32 ENA
pl fss0_lv-02 fss0_lv ENABLED ACTIVE 549755814912 STRIPE 3/1024 RW
sd ssd_00-01 fss0_lv-02 ssd_00 0 17179803344 0/0 vcs14_ssd_00 ENA
sd ssd_03-01 fss0_lv-02 ssd_03 0 17179803344 0/17179803344 vcs14_ssd_03 ENA
sd ssd_06-01 fss0_lv-02 ssd_06 0 17179803344 0/34359606688 vcs14_ssd_06 ENA
sd ssd_09-01 fss0_lv-02 ssd_09 0 17179803344 0/51539410032 vcs14_ssd_09 ENA
sd ssd_12-01 fss0_lv-02 ssd_12 0 17179803344 0/68719213376 vcs14_ssd_12 ENA
sd ssd_15-01 fss0_lv-02 ssd_15 0 17179803344 0/85899016720 vcs14_ssd_15 ENA
sd ssd_18-01 fss0_lv-02 ssd_18 0 17179803344 0/103078820064 vcs14_ssd_18 ENA
sd ssd_21-01 fss0_lv-02 ssd_21 0 17179803344 0/120258623408 vcs14_ssd_21 ENA
sd ssd_24-01 fss0_lv-02 ssd_24 0 17179803344 0/137438426752 vcs14_ssd_24 ENA
sd ssd_27-01 fss0_lv-02 ssd_27 0 17179803344 0/154618230096 vcs14_ssd_27 ENA
sd ssd_30-01 fss0_lv-02 ssd_30 0 11453904864 0/171798033440 vcs14_ssd_30 ENA
sd ssd_01-01 fss0_lv-02 ssd_01 0 17179803344 1/0 vcs14_ssd_01 ENA
sd ssd_04-01 fss0_lv-02 ssd_04 0 17179803344 1/17179803344 vcs14_ssd_04 ENA
sd ssd_07-01 fss0_lv-02 ssd_07 0 17179803344 1/34359606688 vcs14_ssd_07 ENA
sd ssd_10-01 fss0_lv-02 ssd_10 0 17179803344 1/51539410032 vcs14_ssd_10 ENA
sd ssd_13-01 fss0_lv-02 ssd_13 0 17179803344 1/68719213376 vcs14_ssd_13 ENA
sd ssd_16-01 fss0_lv-02 ssd_16 0 17179803344 1/85899016720 vcs14_ssd_16 ENA
sd ssd_19-01 fss0_lv-02 ssd_19 0 17179803344 1/103078820064 vcs14_ssd_19 ENA
sd ssd_22-01 fss0_lv-02 ssd_22 0 17179803344 1/120258623408 vcs14_ssd_22 ENA
sd ssd_25-01 fss0_lv-02 ssd_25 0 17179803344 1/137438426752 vcs14_ssd_25 ENA
sd ssd_28-01 fss0_lv-02 ssd_28 0 17179803344 1/154618230096 vcs14_ssd_28 ENA
sd ssd_31-01 fss0_lv-02 ssd_31 0 11453904864 1/171798033440 vcs14_ssd_31 ENA
sd ssd_02-01 fss0_lv-02 ssd_02 0 17179803344 2/0 vcs14_ssd_02 ENA
sd ssd_05-01 fss0_lv-02 ssd_05 0 17179803344 2/17179803344 vcs14_ssd_05 ENA
sd ssd_08-01 fss0_lv-02 ssd_08 0 17179803344 2/34359606688 vcs14_ssd_08 ENA
sd ssd_11-01 fss0_lv-02 ssd_11 0 17179803344 2/51539410032 vcs14_ssd_11 ENA
sd ssd_14-01 fss0_lv-02 ssd_14 0 17179803344 2/68719213376 vcs14_ssd_14 ENA
sd ssd_17-01 fss0_lv-02 ssd_17 0 17179803344 2/85899016720 vcs14_ssd_17 ENA
sd ssd_20-01 fss0_lv-02 ssd_20 0 17179803344 2/103078820064 vcs14_ssd_20 ENA
sd ssd_23-01 fss0_lv-02 ssd_23 0 17179803344 2/120258623408 vcs14_ssd_23 ENA
sd ssd_26-01 fss0_lv-02 ssd_26 0 17179803344 2/137438426752 vcs14_ssd_26 ENA
sd ssd_29-01 fss0_lv-02 ssd_29 0 17179803344 2/154618230096 vcs14_ssd_29 ENA
sd ssd_32-01 fss0_lv-02 ssd_32 0 11453904864 2/171798033440 vcs14_ssd_32 ENA
dc fss0_lv_dco fss0_lv fss0_lv_dcl
v fss0_lv_dcl - ENABLED ACTIVE 18980992 SELECT - gen
pl fss0_lv_dcl-01 fss0_lv_dcl ENABLED ACTIVE 18980992 CONCAT - RW
sd ssd_30-02 fss0_lv_dcl-01 ssd_30 11453904864 18980992 0 vcs14_ssd_30 ENA
pl fss0_lv_dcl-02 fss0_lv_dcl ENABLED ACTIVE 18980992 CONCAT - RW
sd vcs15_ssd_30-02 fss0_lv_dcl-02 vcs15_ssd_30 11453904864 18980992 0 ssd_30 ENA
That big filesystem
Now, create a filesystem on that volume so we can make use of it (Do this on only one node):
[root@vcs14 ~]# mkfs -t vxfs /dev/vx/dsk/FSS00dg/fss0_lv
version 10 layout
549755813888 sectors, 34359738368 blocks of size 8192, log size 32768 blocks
rcq size 8192 blocks
largefiles supported
maxlink supported
CFSmount the Filesystem (Do this on only one node):
[root@vcs14 ~]# cfsmntadm add FSS00dg fss0_lv /shared/fss0 fss00_SG all=crw
Mount Point is being added...
/shared/fss0 added to the cluster-configuration
[root@vcs14 ~]# cfsmount /shared/fss0
Mounting...
[/dev/vx/dsk/FSS00dg/fss0_lv] mounted successfully at /shared/fss0 on vcs14
[/dev/vx/dsk/FSS00dg/fss0_lv] mounted successfully at /shared/fss0 on vcs15
Configure a Service Group (Optional)
[root@vcs14 ~]# haconf -makerw
[root@vcs14 ~]# hares -modify vxfsckd ActivationMode cvmdg sw FSS00dg sw -sys vcs14
[root@vcs14 ~]# hares -modify vxfsckd ActivationMode cvmdg sw FSS00dg sw -sys vcs15
[root@vcs14 ~]# hagrp -add fss00_SG
[root@vcs14 ~]# hagrp -modify fss00_SG SystemList vcs14 0 vcs15 1
[root@vcs14 ~]# hagrp -modify fss00_SG Parallel 1
[root@vcs14 ~]# hagrp -modify fss00_SG AutoStartList vcs14 vcs15
[root@vcs14 ~]# hagrp -modify fss00_SG SourceFile "./main.cf"
[root@vcs14 ~]# hares -add cfsmount_res CFSMount fss00_SG
[root@vcs14 ~]# hares -modify cfsmount_res BlockDevice "/dev/vx/dsk/FSS00dg/fss0_lv"
[root@vcs14 ~]# hares -modify cfsmount_res MountPoint "/shared/fss0"
[root@vcs14 ~]# hares -add cvmvoldg_res CVMVolDg fss00_SG
[root@vcs14 ~]# hares -modify cvmvoldg_res CVMVolume fss0_lv
[root@vcs14 ~]# hares -modify cvmvoldg_res CVMDiskGroup FSS00dg
[root@vcs14 ~]# haconf -makerw
This should result in this output:
[root@vcs14 ~]# hagrp -state fss00_SG
#Group Attribute System Value
fss00_SG State vcs14 |ONLINE|
fss00_SG State vcs15 |ONLINE|
[root@vcs14 ~]# hares -state|grep -i cfs
cfsmount_res State vcs14 ONLINE
cfsmount_res State vcs15 ONLINE
[root@vcs14 ~]# mount -v|grep fss0
/dev/vx/dsk/FSS00dg/fss0_lv on /shared/fss0 type vxfs (rw,mntlock=VCS,cluster,crw,delaylog,largefiles,mincache=dsync,noatime,nomtime,ioerror=mdisable)
[root@vcs15 ~]# mount -v|grep fss0
/dev/vx/dsk/FSS00dg/fss0_lv on /shared/fss0 type vxfs (rw,mntlock=VCS,cluster,crw,delaylog,largefiles,mincache=dsync,noatime,nomtime,ioerror=mdisable)
[root@vcs14 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootdg-lv_root
36G 22G 12G 65% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 477M 100M 353M 22% /boot
tmpfs 788M 216K 788M 1% /tmp
tmpfs 4.0K 0 4.0K 0% /dev/vx
tmpfs 64M 204K 64M 1% /var/VRTSvcs/lock
/dev/vx/dsk/FSS00dg/fss0_lv
256T 8.7G 256T 1% /shared/fss0
# netrun -q -e 'df -h -t vxfs' vcs1{4,5}
Progress: 0% |-----------+-----------+-----------+-----------| 100%
################################################# Done.
vcs14: Filesystem Size Used Avail Use% Mounted on
vcs14: /dev/vx/dsk/FSS00dg/fss0_lv
vcs14: 256T 8.7G 256T 1% /shared/fss0
vcs15: Filesystem Size Used Avail Use% Mounted on
vcs15: /dev/vx/dsk/FSS00dg/fss0_lv
vcs15: 256T 8.7G 256T 1% /shared/fss0
Run some tests
[root@vcs15 ~]# hacli -cmd 'dd if=/dev/zero of=/shared/fss0/$(uname -n).dump count=1024 bs=4k'
vcs14:
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 2.04956 s, 2.0 MB/s
vcs15:
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 2.07718 s, 2.0 MB/s
[root@vcs15 ~]# hacli -cmd 'ls -ld /shared/fss0/*dump'
vcs14:
-rw-r--r-- 1 root root 4194304 Feb 4 13:28 /shared/fss0/vcs14.dump
-rw-r--r-- 1 root root 4194304 Feb 4 13:28 /shared/fss0/vcs15.dump
vcs15:
-rw-r--r-- 1 root root 4194304 Feb 4 13:28 /shared/fss0/vcs14.dump
-rw-r--r-- 1 root root 4194304 Feb 4 13:28 /shared/fss0/vcs15.dump
[root@vcs15 ~]# hacli -cmd 'cksum /shared/fss0/*dump'
vcs14:
3413741448 4194304 /shared/fss0/vcs14.dump
3413741448 4194304 /shared/fss0/vcs15.dump
vcs15:
3413741448 4194304 /shared/fss0/vcs14.dump
3413741448 4194304 /shared/fss0/vcs15.dump
Why 262142g?
256Tb (the max size of a VxFS filesystem) is 262144gb. When I attempted to create a Filesystem on my 256Tb volume, I was greeted with this error:
[root@vcs14 ~]# mkfs -t vxfs /dev/vx/dsk/FSS00dg/fss00_lv
UX:vxfs mkfs.vxfs: ERROR: V-3-26258: File system size cannot exceed 262144 GB
Also, I noticed that sometimes vxvol wouldn't start a 256Tb volume so I dropped the ball and went for 262142gb for both the volume and filesystem.
SFCFSHA software versions:
[root@vcs14 ~]# rpm -q VRTSvxfs VRTSvxvm
VRTSvxfs-6.1.1.202-RHEL6.x86_64
VRTSvxvm-6.1.1.100-RHEL6.x86_64
If you are on SF6.1+latest patches (As of Jan 2015) and you have a CFS filesystem that's over 100Tb in size, contact Veritas and ask for the following patch: fs-rhel6_x86_64-HotFix-6.1.1.007.tar.gz or fs-rhel6_x86_64-HotFix-6.1.1.202.tar.gz.
The tracking number for this issue is e3233276.
Without the patch, when unmounting the filesystem, you might get a cpu soft lockup. Nothing happens but it slows down the whole unmounting procedure. Here's the sign you're running into this issue:
[root@server101 CFSMount]# umount /vx/vol01
Message from syslogd@server101 at Oct 23 21:36:03 ...
kernel:BUG: soft lockup - CPU#0 stuck for 67s! [vx_ctl_thread:19524]
How to enable 'hacli -cmd':
[root@vcs14 ~]# hacli -cmd uname
VCS WARNING V-16-1-50112 hacli is disabled
[root@vcs14 ~]# haconf -makerw
[root@vcs14 ~]# haclus -modify HacliUserLevel COMMANDROOT
[root@vcs14 ~]# haclus -value HacliUserLevel
COMMANDROOT
[root@vcs14 ~]# haconf -dump -makero
[root@vcs14 ~]# haclus -value HacliUserLevel
COMMANDROOT
[root@vcs14 ~]# hacli -cmd uname
vcs15:
Linux
vcs14:
Linux
Comments
Post a Comment