ConsistencyLevel in Hector and Cassandra

I started playing with failover in Casssandra the other day and rapidly found myself in a bit of a pickle. I had a 2 node development environment allowing me to play with Cassandra and start developing Hector programs. So I decided to turn one machine off and see if my program would carry on as normal.

It didn’t. Not much of a failover I thought. To make matters worse if attached via the Cassandra cli client, I could retrieve data from my single node. What was going on ?

Turns out this was all to do with the consistency of my cluster. Take a look at the consistency section of:

http://wiki.apache.org/cassandra/API

There are multiple levels of consistency available in a Cassandra cluster, and they can be different for read and write operations. What I hadn’t realized is that Hector defaults to a consistency of QUORUM whereas cli defults to a consistency of ONE (I believe). So with a two node cluster, Hector will fail on reads if one goes down.

Adding Nodes

One way to get round the problem is to add more nodes and up the replication level in the conf files to 4. I added two notes that where not part of the seeding process and used the Autoboostrap option in he conf file to join the cluster. This has kind of solved the problem. I can now read from one of the bootstrapped nodes if any of the other nodes goes down, but not from one of the seed nodes. I think more work on the configuration and layout of the cluster is needed by me.

Changing the consistency in Hector

You can of course change the consistency in Hector. The code here refers to Hector 0.6.15 and above (for now). The first thing we need to do is implement a consistencylevelpolicy. Here’s an example based on the default consistency implementation in Hector

import me.prettyprint.cassandra.model.*;
import org.apache.cassandra.thrift.ConsistencyLevel;

public final class MyConsistancyLevel implements ConsistencyLevelPolicy {

@Override
public  ConsistencyLevel get(OperationType op) {
   switch (op){
      case READ:return ConsistencyLevel.QUORUM;
      case WRITE: return ConsistencyLevel.ONE;
      default: return ConsistencyLevel.QUORUM; //Just in Case
   }
}
@Override
public ConsistencyLevel get(OperationType op, String cfName) {
   return ConsistencyLevel.QUORUM;
}
}

In this example we set the READ consistency to QUORUM and the WRITE consistency to ONE. (I’ve also included a default consistency just in case !). Once we’ve got this class we can set the consistency for the keyspace like this:

ConsistencyLevelPolicy mcl = new MyConsistancyLevel();
ko.setConsistencyLevelPolicy(mcl);

And that’s it !

Many thanks to Ran Tavory and Colin Vipurs for their help on this.

As ever comments and so on gratefully received.