Recover the MSM cluster

Restore an MSM cluster using the VMware vSphere Client. Restore when an error occurs when joining nodes to the cluster or when the connectivity for one of the MSM nodes in the cluster is down.

About this task

Perform this task only during the upgrade process.

Power on MSM nodes in the reverse order that they were powered off, starting with the seed node.

Before you begin

Verify the order that the MSM nodes were powered off. If the order is unknown, begin with the MSM nodes in the seed node of VMware vCenter.

Procedure

  1. Power on the MSM seed node using the VMware vCenter vSphere Client. Open a console connection to the VM, and wait for the login prompt.
  2. Perform the following steps for each MSM node in the same data center in the reverse order from shutdown:
    1. Power on the next MSM node using the VMware vCenter vSphere Client. Continue powering on each MSM node in the cluster in five minute increments. Repeat the recovery process for every MSM node in each data center of your clustered environment.
    2. After all the MSM nodes in all data centers are powered back on, on each of the MSM nodes in the clustered environment, type:

      service tomcat restart

    3. To check the services, type:

      vision start

    4. To monitor the status of the cluster, type:

      /opt/cassandra/bin/nodetool status

    5. After all MSM nodes are up and running in the same data center, verify that the IP address of each data center MSM node is displayed. Type:

      rabbitmqctl cluster_status

  3. After all the nodes in all data centers are powered on, perform the following:
    1. Check the VxBlock Central dashboard to ensure you can view your Converged Systems. If not, to restart the Tomcat service, type:

      service tomcat restart

    2. Check the Compliance and Remediation of the VxBlock Central dashboard to ensure that the compliance status is displayed. If not, type:

      service vision-mvb-compliance restart