Revision as of 00:59, March 11, 2016 by Sschlich (talk | contribs) (→‎Running Nodetool on a Feature Server Host)
Jump to: navigation, search

Feature Server Cassandra Maintenance

The nodetool is a Cassandra utility for managing a Cassandra cluster. Use it for Feature Server Cassandra Maintenance.

Regular maintenance repairs inconsistencies across all data ranges, and ensures that replicated data is consistent across all nodes. You should perform maintenance on a node after a Feature Server upgrade and when new Feature Server nodes were added to the cluster. Apply additional maintenance steps after you remove a node from the cluster. See section [xxxxxxxxxxxxxxxxxxxxx|Maintenance after Removing Node from Cluster] for more details.

As a best practice, you should schedule the regular maintenance weekly, e.g. using cron on Linux or as a Task Scheduler on Windows.

Running Nodetool on a Feature Server Host

[https://docs.datastax.com/en/cassandra/1.2/cassandra/tools/toolsNodetool_r.html The Nodetool utility] has command line interface. The steps below describe how to run nodetool commands on Cassandra when it is deployed on a host, as part of a Feature Server deployment.

1. Change the current directory in the Linux shell terminal or in the Windows command prompt window: 

cd <Feature Server Installation Directory>/work/jetty-<vms_host>-<http_port>-fs.war-_fs-any-/webapp/WEB-INF/lib
where...
<Feature Server Installation Directory> is where Feature Server is installed for your company.
<vms_host> is the IP address of the vms host parameter that Feature Server started with, defined in launcher.xml or in the command line. The default is 0.0.0.0.
<http_port> is the port number that Feature Server started with, defined in launcher.xml. The default is 8080.

2. Run the Nodetool utility.

In Linux: (.jar is separated by a colon ":")
java -cp libthrift-0.7.0.jar:cassandra-thrift-<Cassandra version>.jar:commons-cli-1.1.jar:cassandra-all-<Cassandra version>.jar org.apache.cassandra.tools.NodeCmd -h <Cassandra host> -p ''<jmx_port> <nodetool command>''
In Windows: (.jar is separated by a semicolon ";")
java -cp libthrift-0.7.0.jar;cassandra-thrift-<Cassandra version>.jar;commons-cli-1.1.jar;cassandra-all-<Cassandra version>.jar org.apache.cassandra.tools.NodeCmd -h <Cassandra host> -p ''<jmx_port> <nodetool command>''
where...
<Cassandra version> could be either 1.1.6 or 1.1.12 depending on Feature Server version installed. Please check the actual cassandra-all-*.jar file name in current directory.
<Cassandra host> hostname or IP address where Feature Server is running.
<jmx_port> is the JMX port number that Feature Server started with, defined in launcher.xml. Default=9192.
<nodetool command> can be ring [keyspace name] or removetoken <token> or repair or flush.

For example, the following command line runs nodetool and sends <nodetool command> ring to Feature Server running on the local host: 

cd genesys/fs/work/jetty-0.0.0.0-8080-fs.war-_fs-any-/webapp/WEB-INF/lib

java -cp libthrift-0.7.0.jar:cassandra-thrift-1.1.6.jar:commons-cli-1.1.jar:cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p 9192 ring sipfs

Regular Maintenance Run nodetool repair on all Feature Server hosts in Cassandra cluster, see section“Running Nodetool on a Feature Server Host” detailing about how to run nodetool.

Maintenance after Removal a Node from Cluster When you remove a node from the ring, you must also remove the corresponding tokens. Use these software procedures: 1. Run nodetool ring to obtain tokens of nodes to be removed 2. Run nodetool removetoken to remove nodes from the ring. 3. Run nodetool repair. 4. Run nodetool ring to validate removal.

For example, run nodetool ring to obtain tokens of nodes to be removed cd genesys/fs/work/jetty-0.0.0.0-8080-fs.war-_fs-any-/webapp/WEB-INF/lib java -cp libthrift-0.7.0.jar:cassandra-thrift-1.1.6.jar:commons-cli-1.1.jar:cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p 9192 ring sipfs …to see the following output returned tokens of nodes which are down: <fs-host-IP1> DC1 RAC1 Up Normal 1.17 MB100.00% 167086018864645871692761019448293152722 <fs-host-IP2> DC1 RAC2 Up Normal 1.29 MB 100.00% 26003787676682001822918611294472056316 <fs-host-IP3> DC2 RAC1 Down Normal 1.15 MB 100.00% 41007983964572150951275225962045789866 <fs-host-IP4> DC2 RAC2 Down Normal 1.16 MB 100.00% 53685600614278234503162023330018045221


The following nodetool commands remove <fs-host-IP3> and <fs-host-IP4> nodesfrom the ring: cd genesys/fs/work/jetty-0.0.0.0-8080-fs.war-_fs-any-/webapp/WEB-INF/lib java -cp libthrift-0.7.0.jar:cassandra-thrift-1.1.6.jar:commons-cli-1.1.jar:cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p 9192 removetoken 41007983964572150951275225962045789866 java -cp libthrift-0.7.0.jar:cassandra-thrift-1.1.6.jar:commons-cli-1.1.jar:cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p 9192 removetoken 53685600614278234503162023330018045221 After token removal, running nodetool ring should provide the following output: <fs-host-IP1> DC1 RAC1 Up Normal 1.17 MB 100.00% 167086018864645871692761019448293152722 <fs-host-IP2> DC1 RAC2 Up Normal 1.29 MB 100.00% 26003787676682001822918611294472056316


You should perform regular maintenance on your Feature Server Cassandra database, to be certain that the database and its backups are all in sync. When any of these events occur, run the corresponding procedure:

Run weekly on all Feature Server nodes

You can run this command line: 

https://intranet.genesys.com/pages/viewpage.action?title=SIPVM-3779+Cassandra+Maintenance+Procedure&spaceKey=RP

Run after a Feature Server restart

You can trigger a restart by adding or removing Feature Server nodes, in a new or existing data center. After the restart: 

cd /genesys/fs/work/jetty0.0.0.08080fs war_fsany/webapp/WEBINF/lib 
java cp libthrift0.7.0. jar:cassandrathrift1.1.6. jar:commonscli1.1. jar:cassandraall1.1.6. jar org.apache.cassandra.tools.NodeCmd h localhost p 9192 repair

Running Nodetool on a Feature Server Host

The nodetool is a command-line interface for managing a Cassandra cluster.

  1. Change working directory: cd <Feature Server Installation Directory>/work/jetty-<vms_host>-<http_port>-fs.war-_fs-any-/webapp/WEB-INF/lib
    where...
    <Feature Server Installation Directory> is where Feature Server is installed for your company.
    <vms_host> is the IP address of the vms host parameter that Feature Server started with, defined in launcher.xml or in the command line. The default is 0.0.0.0.
    <http_port> is the port number that Feature Server started with, defined in launcher.xml. The default is 8080.
  2. Run the nodetool utility.
    • In Linux: (.jar separated by “:”)
      java -cp libthrift-0.7.0.jar:cassandra-thrift-1.1.6.jar:commons-cli-1.1.jar:cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p <jmx_port> <nodetool command>
    • In Windows: (.jar separated by “;”)
      java -cp libthrift-0.7.0.jar;cassandra-thrift-1.1.6.jar;commons-cli-1.1.jar;cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p <jmx_port> <nodetool command>
      where...
      <jmx_port> is the JMX port number that Feature Server started with, defined in launcher.xml. Default=9192.
      <nodetool command> can be ring [keyspace name] OR removetoken <token> OR repair OR flush

For example, the following command line runs nodetool and sends the nodetool command ring to Feature Server running on the local host: 

cd genesys/fs/work/jetty-0.0.0.0-8080-fs.war-_fs-any-/webapp/WEB-INF/lib

java -cp libthrift-0.7.0.jar:cassandra-thrift-1.1.6.jar:commons-cli-1.1.jar:cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p 9192 ring sipfs

Removing a Feature Server Cassandra Cluster Node

When you remove a node from the ring, you must also remove the corresponding tokens. Use these software procedures:

  1. Run nodetool ring to obtain tokens. (See Running Nodetool on a Feature Server Cassandra Host
  2. Run nodetool removetoken to remove nodes from the ring.
  3. Run nodetool repair.
  4. Run nodetool ring to validate removal.

For example, running nodetool ring...
On Linux: 

java -cp libthrift-0.7.0.jar:cassandra-thrift-1.1.6.jar:commons-cli-1.1.jar:cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p 9192 ring</tt>

On Windows: 

java -cp libthrift-0.7.0.jar;cassandra-thrift-1.1.6.jar;commons-cli-1.1.jar;cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p 9192 ring

…to see the following output returned: 

10.51.29.117 usw1 RAC2 Up Normal 1.29 MB 100.00% 26003787676682001822918611294472056316
10.51.28.221 usw1 RAC1 Up Normal 1.17 MB 66.65% 167086018864645871692761019448293152722
10.52.89.189 aps2 RAC1 Down Normal 1.15 MB 25.90% 41007983964572150951275225962045789866
10.52.88.190 aps2 RAC2 Down Normal 1.16 MB 7.45% 53685600614278234503162023330018045221

The following nodetool commands remove <fs-host-IP3> and <fs-host-IP4> from the ring: 

cd genesys/fs/work/jetty-0.0.0.0-8080-fs.war-_fs-any-/webapp/WEB-INF/lib

java -cp libthrift-0.7.0.jar:cassandra-thrift-1.1.6.jar:commons-cli-1.1.jar:cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p 9192 removetoken41007983964572150951275225962045789866

java -cp libthrift-0.7.0.jar:cassandra-thrift-1.1.6.jar:commons-cli-1.1.jar:cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p 9192 removetoken53685600614278234503162023330018045221

After token removal, running nodetool ring should provide the following output:
<fs-host-IP1> DC1 RAC1 Up Normal 1.17 MB 100.00% 167086018864645871692761019448293152722
<fs-host-IP2> DC1 RAC2 Up Normal 1.29 MB 100.00% 26003787676682001822918611294472056316

Comments or questions about this documentation? Contact us for support!