OpenSolaris

  subsites   code review   repo   packages   bugs   defect   polls   planet
You are not signed in. Sign in or register.

Testing Open HA Cluster, Geographic Edition

The geo module of the Open High Availability (Open HA) Cluster Automated Test Suite is the open-source version of the Geo-Cluster test harness. This test harness currently supports Sun StorageTek™ Availability Suite (AVS) data replication software. For more information or for complete documentation, refer to the Open HA Cluster Automated Test Suite User's Guide. For the most up-to-date build instructions, see the Building and Installing the Open HA Cluster Automated Test Suite page.

The following procedures explain how to run the Open HA Cluster Automated Test Suite to qualify your build of Open HA Cluster, Geographic Edition.

Prerequisites

Before you run the geo test suite, ensure that you have met the following prerequisites.

Cluster Hardware

  • The geo-cluster must consist of two clusters that have at least two nodes each.

  • Each cluster must have at least three free shared disks that are connected to the same set of cluster nodes.
    • Two of the shared disks are used for data volumes.
    • The third disk can be used for the AVS configuration location.

  • The shared disks must be SMI labeled.

Cluster Software

  • The following software must be installed on all nodes of both clusters:
    • Open HA Cluster, Geographic Edition
    • Sun StorEdge™ Availability Suite
    • SUNWscts (Open HA Cluster Automated Test Suite)
    • SUNWscnfs (HA for NFS data service)

  • When you run scconf -pvv | grep d#, where d# is the DID name of the disk you are using for the test, the scconf command must return some output.

Test Server

The SUNWscts (Open HA Cluster Automated Test Suite) package must be installed on the test server.

Top

How to Create the Test Request File

Create a test request file of the geo-cluster configuration before you run the test suite.
  1. Copy the ~scate/tset/geo/config/testfile.template file to create the test request file. 
    $ cd ~scate/tset/geo/config
$ cp testfile.template /var/tmp/geo_testfile.conf
  2. Open the new test request file for editing. 
    $ vi /var/tmp/geo_testfile.conf
  3. Change the values of the variables listed in the file.

    Refer to the comments at the top of the testfile.template file.

  4. When you are finished, save and exit the file.

  5. Ensure that the logical host names that are specified in the test request file have entries in the /etc/inet/hosts file on each cluster node.

Top

How to Run the geo-avs Test Suite

  1. Ensure that Open HA Cluster, Geographic Edition software is running on all nodes of both clusters. 
    # geoadm start
  2. Ensure that trust exists between the two clusters.

    On the first cluster, issue the following command: 

    # geops add-trust -c cluster2
    cluster2 is the name of the second cluster.

    On the second cluster, issue the following command: 

    # geops add-trust -c cluster1
    cluster1 is the name of the first cluster.

  3. Log in to the Test Server as user scate.

  4. Run the test. 
    $ ~scate/tset/bin/run_geo -f /path/test-request-file geo:scenario

    You can set scenario to one of the following values:

    • geo_avs_nonreboot
    • geo_avs_rbac_nonreboot
    • geo_avs_mult_pgs_nonreboot
    • geo_avs_reboot
    • geo_avs_mult_pgs_reboot

    See the ~scate/tset/geo/tet_scen file for more options.

  5. After the test finishes, check the test results on the Test Server under the ~scate/geo_results/sc_clusternode1_clusternode2/results/log.pid/ directory.
The following example test request file runs the geo-avs test suite on cluster breccia and cluster hadar. 
#
# Geo-Cluster Test Request File
#

NODENAME=pbric1
SEQUENCE=geo

CLUSTER_1_NODE=pbric1
CLUSTER_1_FAILOVER_ADDRESS_1=breccia-1
CLUSTER_1_FAILOVER_ADDRESS_2=breccia-2
CLUSTER_1_FAILOVER_ADDRESS_3=breccia-3
CLUSTER_1_FAILOVER_ADDRESS_4=breccia-4
CLUSTER_1_DISK_1=d4
CLUSTER_1_DISK_2=d5

CLUSTER_2_NODE=phadar1
CLUSTER_2_FAILOVER_ADDRESS_1=hadar-1
CLUSTER_2_FAILOVER_ADDRESS_2=hadar-2
CLUSTER_2_FAILOVER_ADDRESS_3=hadar-3
CLUSTER_2_FAILOVER_ADDRESS_4=hadar-4
CLUSTER_2_DISK_1=d5
CLUSTER_2_DISK_2=d6

RAW_DEVICE_TYPE=sds
DATA_VOL_SIZE=240     
GEO_DELAY=6
PG_TIMEOUT=600
Top

How to Monitor Test Results

Test result directory - The test result is located on the node where the run_geo command is started. It is in the /usr/scate/geo_results/sc_clusternode1_clusternode2/results/log.pid directory, where clusternode1 is the name of a node in the first cluster, and clusternode2 is the name of a node in the second cluster. When the test is finished, you can look at the summary and test log files under this test result directory.

Test log files - If the summary file indicates that there are failures, you can go to the test log file to look further into the failures. Almost all command executions in the geo-avs test suite capture the output of the command executions and save the output in /var/tmp/geo* files in case of failures. So if a command fails, check the /var/tmp/geo* files to see the message that the command returned.

Here is an example of a line in a test log file: 

520|0 0 00017162 0 302|Nov 27 12:45:15 : COMMAND: 
/usr/scate/tset/geo/bin/check_geoadm_status_pg pg1 halite-1 "Configuration" "OK" 
/var/tmp/geoRabqHH 1 root phalite4 rsh, ret = 0
The preceding example shows that command execution returned successfully (ret = 0). If ret were 1, you could look at the /var/tmp/geoRabqHH file, which is saved on the Sun Cluster Automated Test Environment (SCATE) server.

Location of a test log file - To find out whether a /var/tmp/geo* file is on the SCATE server or one of the cluster nodes, check the third column of the output. The following information is taken from the previous example: 

520|0 0 00017162
In this example, the 000 prefix represents the SCATE server. Because this prefix indicates that the command was executed on the SCATE server, the /var/tmp/geo* files are located on the SCATE server.

This prefix can be 001, 002, 003, and so forth. The prefix 001 indicates that the command was executed on TET_NODE_1 and the prefix 002 indicates that the command was executed on TET_NODE_2, and so forth.

In some cases, the SCATE server issues a command to be run on a cluster node. In this test, the test log contains the following entry: 

  520|0 0 00013369 0 148|Dec 02 21:05:15 : run_remote: cmd=, node=
Monitoring live logs - During test execution, you can monitor the test execution as it runs by viewing the live log located under the /usr/scate/geo_results/sc_clusternode1_clusternode2/results/0001e directory. You can run tail -f on the live log to view the latest output.

If you kill a test manually, the live log is saved in the live_logs/ subdirectory of the test result directory.

Top

How to Cleaning Up After You Kill a Test Manually

Perform all of the following tasks to clean up the geo-cluster configuration after you manually kill a test.
  • Remove all protection groups.
  • If a cluster is still in a partnership, leave the partnership.
  • Manually kill all dangling processes started by the test.
  • Remove resources and resource groups that are created by the test.
  • Run the umount command to unmount and then remove the /global/sndrii* and /global/geo* directories.
  • Remove all metasets named scate-sndrii-dg-1 and scate-sndrii-dg-2, which are created by the tests.
  • Remove all entries containing scate-sndrii in the /etc/vfstab file on all cluster nodes.
  • Remove all /var/cluster/geo/avs/scate*volset.ini files on all cluster nodes.
  • Ensure that the scate entry in the /etc/user_attr file is scate::::type=normal.
Top