OpenSolaris

  subsites   code review   repo   packages   bugs   defect   polls   planet
You are not signed in. Sign in or register.

Testing Open HA Cluster

This document provides information about how to run Smoke Tests for Open HA Cluster. Smoke Tests include tests for HA Communication Infrastructure and Services Software in Open HA Cluster. For the most up-to-date build instructions, see the
Building and Installing the Open HA Cluster Automated Test Suite page. This document contains the following sections:

Configuration Requirements for the Smoke Test Suite

Before you run the Smoke test suite, ensure that you have met the following prerequisites.

Cluster Node Requirements

  • A cluster with at least two nodes. Both SPARC and x86 configurations are supported.
  • The Open High Availability Cluster (OHAC) software installed on all cluster nodes.
  • The SUNWspro (The Studio Compiler) package installed on all cluster nodes.
  • The Open HA Cluster Automated Test software package (SUNWscts) installed on all cluster nodes. You must provide the path to the C Compiler during installation of the SUNWscts test package.
Top

Client Machine Requirements

  • One external client machine outside the cluster. It can be SPARC or x86.
  • The Open HA Cluster Automated Test software package (SUNWscts) installed on the client machine.
Top

Test Server Machine Requirements

  • One test server machine. It can be SPARC or x86.
  • Open HA Cluster Automated Test software package (SUNWscts) installed on the test server machine.
Top

Individual Test Prerequisites

For failover and scalable tests:
  • One hostname for configuring a Logical Host or Shared Address resource.
  • One global raw device, either a Solaris Volume Manager metadevice or a /dev/global device for hardware RAID.

For zone tests:
NOTE: You can use the same hostname and global raw device in all three types of tests (failover, scalable, and zone). Top

How to Start the SCATE Server Process

To determine whether the SCATE server process is already running, use one of the following commands:
$ /usr/scate/svc/bin/scate status
or
$ pgrep scateserver
  • Log on to the test server as superuser.
  • Start the SCATE server process on the test server, if the SCATE server process is not already running.
    # /usr/scate/svc/bin/scate start
    
    Top

    How to Create the Test Request File

    1. Log on to the test server as user scate.
    2. Run the smokerun utility from the test server to create a test request file.
      $ /usr/scate/tset/bin/smokerun
      

    The following is a sample screenshot of a smokerun invocation.

    Name of a cluster node ? [phys-foo-1] pono1
    
    Obtaining cluster configuration information from node pono1...
    
    Specify Test Results Log Directory that is NFS exported and world writable.
    Ex : /home/username/log (or) /net/machine/export/home/log
    Test Results Log Directory? /home/mytest/scate_logs
    
    Test Request Filename ? [/home/mytest/scate_logs/TestRequest/testrequest1.out] 
    
    -------------
    Test Category
    -------------
    
            1. Smoke Tests for HA Communication Infrastructure
            2. Smoke Tests for HA Services
    
    
    Enter the number that corresponds to the tests you want to run.
    To select more than one option, Enter a comma seperated list.
    
    Enter your selection : 2
    
            1. Failover Services
            2. Scalable Services
            3. Private Interconnect
            4. Non-global Zone
    
    Enter the number that corresponds to the test you want to run.
    To select more than one test, you can either enter a comma
    seperated list (1,2,3...) or specify a range (1-3) or
    enter "all" to run all the tests.
    
    Enter your selection : 1
    
    ***** Generating Entry for Failover Services *****
    Some tests you selected require information about Network Hostname. 
    
    Please enter the hostname to be used for configuring Logical Host and Shared Address resource :ono-1
    
    Getting node list from the cluster...
    Please enter comma seperated node list on which "ono-1" will be hosted [pono1,pono2] : Return
    
    Validating the Network hostname...
    Getting the "NetIfList"(IPMP group info). Please wait...
    
    ********************************************************************************
    Tests require one global raw device. Device Name should be of the form
    /dev/md/foo-1/rdsk/dx for SVM or /dev/global/rdsk/dxsy for a H/W RAID device.
    
    The device should NOT be mounted either globally or locally on any cluster node.
     Also, the device should NOT be mounted at boot thru /etc/vfstab. If the device
    is present in /etc/vfstab, set 'mount at boot' field to 'no' on all the nodes.
    ********************************************************************************
    
    Enter the raw disk slice(s) and end the list with a blank line.
    Disk name? /dev/md/ono-1/rdsk/d0
    Disk name? Return
    
    You selected 1 disks
    Raw Disk 1 = /dev/md/ono-1/rdsk/d0
    Is this correct? [y/n] ? y
    
    You selected 1 disks
    Raw Disk 1 = /dev/md/ono-1/rdsk/d0
    
    Test Request File is in /home/mytest/scate_logs/TestRequest/testrequest1.out
    
    Run the tests using smokerun -f /home/mytest/scate_logs/TestRequest/testrequest1.out
    

    The following is an example test request file for Smoke Failover Services testing:

    # SCATE Test Request File
    # Generated: n1c33 Thu May 22 07:34:20 PDT 2008
    
    LOGDIR=/home/mytest/scate_logs
    
    CLUSTER
            NODENAME=pono1
    END
    
    # Smoke - Failover Services
    REQUEST cti
        SEQUENCE=sanity
        SCENARIO=failover
        #SCENARIO=tc_mytest
        #TEST_NUMBER="{1}"
        #TEST_NUMBER="{1-3,5}"
        #TEST_NUMBER=all
        RUN_NEWCLI=TRUE
        NETWORK_HOSTNAME_1=ono-1
        NETWORK_HOSTNAME_ADAPTERS_1="sc_ipmp0@1,sc_ipmp0@2"
        NETWORK_HOSTNAME_HOSTS_1=pono1,pono2
        NETWORK_HOSTNAME_TYPE_1=0
        RAW_DISK_1=/dev/md/ono-1/rdsk/d0
        FS_TYPE=ufs
    END
    
    Top

    How to Run the Smoke Test

    The test request file is created under the log directory that is passed to the smokerun utility. In the previous example, the test request file name is /home/mytest/scate_logs/TestRequest/testrequest1.out. To run the test suite, invoke the following command on the test server as user scate.

    $ /usr/scate/svc/bin/smokerun -f /path/testrequestfile
    
    Example:
    $ /usr/scate/svc/bin/smokerun -f /home/mytest/scate_logs/TestRequest/testrequest1.out
    
    Executing the tests in /home/mytest/scate_logs/TestRequest/testrequest1.out...
    Info: (From Server) Host is rsh'able.
    Operation Success: (From Server) Job ID is 1
    Operation Success: (From Server) Log directory is /home/mytest/scate_logs/05-22-08_09.57.48_pono1_1
    
    Top

    How to Run Individual Test Cases

    For CTI Requests

    1. Edit the test request file to add values to the following parameters:
         SCENARIO=tc_testcase
         TEST_NUMBER="{N}"
      
      For "{N}", specify the numbers of the tests to run by using the following format, including the quotation marks (" ") and the braces ({ }):
      "{1}"
      Specify a single test
      "{1,3}"
      Specify multiple tests
      "{1-3}"
      Specify a range of tests
    2. As user scate, run the smokerun utility, specifying the request file that you edited.
      $ smokerun -f requestfile
      

    For Other Requests

    1. Edit the test request file to add values to the following parameters:
         SCENARIO=tc_testcase
         TEST_NUMBER="N"
      
      For "N", specify the numbers of the tests to run by using the following format, including the quotation marks (" ") but without the braces you use in CTI request files:
      "1"
      Specify a single test
      "1,3"
      Specify multiple tests
      "1-3"
      Specify a range of tests
    2. As user scate, run the smokerun utility, specifying the request file that you edited.
      $ smokerun -f requestfile
      
    Top

    How to Stop the SCATE Server Process

    # /usr/scate/svc/bin/scate stop
    
    Top

    Logs and Reports for the Test Suites

    The test suites generate the following logs and reports.

    Test Logs

    The test logs for the smokerun test invocation are created under the log directory that is passed to the smokerun utility. The name of a test log is constructed as follows:

        /Logdirectory/timestamp_clusternode_N/
    
    Logdirectory
    The directory that is passed to the smokerun utility
    timestamp
    Includes the date and time of the test suite invocation
    clusternode
    The name of the cluster node that is passed to the smokerun utility
    N
    A running job number

    In the previous example, the test logs for the smokerun test invocation are in the /home/mytest/scate_logs/05-22-08_09.57.48_pono1_1/ directory.

    Top

    SCATE Report

    A SCATE_REPORT file is created at the top level directory /Logdirectory/timestamp_clusternode_N/SCATE_REPORT/. This file has PASS/FAIL status for all the individual test suites and also a pointer to the detailed logs for the failed test suites.

    Top

    Individual Test Logs

    Test logs for individual entries in the test request file are in the following locations, where reqno is a running test request number and testcase is the test case name:

    Summary Results:
    /Logdirectory/timestamp_clusternode_N/reqno_testcase/summary
    Centralized Log:
    CTI: /Logdirectory/timestamp_clusternode_N/reqno_testcase/results/testlog
    Other: /Logdirectory/timestamp_clusternode_N/reqno_testcase/clusternode.log
    Individual Test Logs:
    CTI: /Logdirectory/timestamp_clusternode_N/reqno_testcase/results/sorted_results
    Other: /Logdirectory/timestamp_clusternode_N/reqno_testcase/*_fi_tests

    All the commands that are executed in the Smoke tests, along with the output and the log messages from the code, are captured in the test log files as shown in the following example output.

    520|0 1 00123575 1 43|May 22 08:10:35 : Sysid 1, file McExecCmd.c, line 129 : ex
    ecute_cmd: gosu ksh -c '/usr/cluster/bin/clrt register SUNW.tsvc2'
    
    The following are the sample contents of the summary file:
    TC /tests/tc_failover
    
    TP 1 tc_failover PASS
    TP 2 tc_failover PASS
    TP 3 tc_failover PASS
    
    Top

    Monitoring Live Logs

    You can use the scatestatus -l command to monitor the progress of the smokerun test invocations. The live logs for individual test suites for 'cti' REQUEST entries in the test request file are available in the following location:

        /Logdirectory/timestamp_clusternode_N/1_xxx/results/0001e/*.AAA
    
    Use the tail -f command on the live log to show the log messages from the tests that are currently in progress.

    If the tests are aborted for some reason, the logs from the tests that were in progress are stored under the live_logs/ subdirectory.

    Top