removing a 'dead' node from cassandra cluster
We end up in ‘dead’ cassandra node situation if we add a node to the cluster and then delete this VM (for example, terminate during re-building second region) To recover from this problem, we need to do following:
1. Remove the ‘dead’ node. We do not have connection to cassandra on the node being deleted, so we can NOT use ‘decomission’ command. We need to use ‘removetoken’ instead. First, listing current nodes:
root@slbe-22-19-p lib]# java -cp libthrift-0.7.0.jar:cassandra-thrift-1.1.6.jar:commons-cli-1.1.jar:cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p 9192 ring Note: Ownership information does not include topology, please specify a keyspace. Address DC Rack Status State Load Owns Token
83794128212295607812427599225465974454
10.52.86.38 usw1 RAC1 Down Normal ? 59.09% 14189180876230654535300830053153397235 10.51.29.29 usw1 RAC2 Up Normal 547.84 KB 3.45% 20055304209618966354482420882580629466 10.51.28.170 usw1 RAC1 Up Normal 498.49 KB 37.46% 83794128212295607812427599225465974454
We see that node 10.52.86.38 is Down. Using its’ token for the next command: [root@slbe-22-19-p lib]# java -cp libthrift-0.7.0.jar:cassandra-thrift-1.1.6.jar:commons-cli-1.1.jar:cassandra-all-1.1.6.jar org.apache.cassandra.tools.NodeCmd -h localhost -p 9192 removetoken 14189180876230654535300830053153397235
This command takes a while to execute.
2. After that, I had to drop the ‘regional’ keyspace for apac (this keyspace has 0 nodes, so, data is gone anyway). So, this is sort of ‘cleanup’ process. If we have all nodes in a keyspace gone, we loose data in this keyspace.
3. After that, nodes in apac can be started (or, nodes can be redeployed)
Me must consider, that deleted keyspace is created ONLY by conf_sync node and ONLY in case all nodes are started. ‘Standard’ deployment workflow assumes, that primary node is started before backup, and in this case keyspace sipfs_ap never gets created. To resolve this, all nodes for keyspace sipfs_ap must be stopped and then either started nearly at the same, or conf_sync node restarted.