Cassandra’s data model as records and lists

I have to admit I’ve never really been happy with Cassandra’s data model, or to be more precisely, I’ve never really been with my understanding of the model. However I’ve realized that if we think of two use cases for column families then things may become a bit clearer. For me, Column families can be used in one of two ways, either as a record or an ordered list.

Columns used as a record
If we place name value pairs under the column key that contain different attributes then we can consider the columns as classic database record. So if we are wanting to store the details of a user then the columns might be:

Name (key)

• Email: user@example.com
• Twiter: TwitterUser
• Phone: 01 000 345678

In this schema the order of the columns is not important because the names are not related. However unlike a relational database, there is no definition of the “fields” in the record, we define them at runtime in the application. This does give us the flexibility to add new fields providing our application can handle missing “fields”.

Columns used as a List
If each of the name value pairs are the same attributes then we can consider this as an ordered list . In this use case the ordering of the columns is important and the ordering type needs to be carefully thought out. For example if we want to store messages from a user, and we want to be able to get the most recent, then we will store them as:

Author (key)

• Timeuuid: Message
• Timeuuid: Message
• Timeuuid: Message

This ordered list can be thought of as an index of records. The records would be stored in another column family.

Supercolumns as a list of records
Even better, we can use supercolumns to create a combination of lists and records. Normally we would make the supercolumns the ordered list and the columns the record. In our messaging system, we want to get the latest messages from a user:

Author (Key)

• Timeuuid: (Supercolumn name)
• Message: Message Text
• Time: Time of message
• Picture: Binary picture data
• Timeuuid: (Supercolumn name)
• Message: Message Text
• Time: Time of message
• Picture: Binary picture data

The supercolumns are ordered by time, the columns under it are not ordered.

As ever I look forward to comments about this post.

Dundee Hackday 2011 begins.

So yesterday we kicked off this years Dundee Hackday with YDN and Mozilla. The timetable for this year is:

• Tuesday 15th, Video conference with Murray Rowan and Steve Marshall from YDN! and Christian Heilmann from Mozilla
• Wednesday 16th Start to assemble into teams and get your ideas together.
• Tuesday March 1st Post your groups and ideas to Entry form
• We will will then take a look at your idea and give you feedback.
• Start build your Hack.
• March 25: Show your Hack in the QMB Street to University staff, Yahoo! developers and Christian Heilmann
• March 25 The Best Hacks are announced, prizes are given. All retire to the student union for a well earned rest!

Christian Heilmann has published his thoughts on yesterday's video conference Introducing Mozilla technology and ideas to students for a hack day


TwitterTag #uhackdundee11
FlickrTag uhackdundee11

IIS and Glassfish together

This is really just a note to myself. However if you want to run IIS and glassfish on the same windows 2008 server it's not enough to config IIS to listen on one IP address. You need to tell windows that it is responsible for only one of the addresses. You can do this form the command line:


netsh http add iplisten ipaddress=xxx.xxx.xxx.xxx

Second Glassfish will need to be configured to listen to only one address. In the config.xml for the domain find the network -listeners and add an address to port 80:

Firefox and undefined array elements

So here's a Javascript question. Now I admit I'm not a Javascript expert so this ones got me a bit stumped.


In the following code snippet I'm trying to get a function to do something only when it is
first called. It can be called with a number of items (identified by an integer I), it
should call itself only the first time it is called with that arguement. I have an array (lsaCall)
with only the first item defined (and set to 0). If the array element is undefined (or 0) then the function
calls itsel on a timer.

var lsaCall = [0];
function loadSubscribedArticles(i,tag,Author){
 
 //alert("lsaCall "+lsaCall[i]);
 if ((lsaCall[i]==undefined) || (lsaCall[i] ==0)) {
  lsaCall[i]=1;
  //alert("Undefined"+lsaCall[i]);
  var int=self.setInterval("loadSubscribedArticles('"+i+"','"+tag+"','"+Author+"')",1000);
  
 }
}

This works fine and dandy but doesn't work at all in firefox. It complains that lsaCall
is undefined (well I know that !)
Any ideas on how to get this working in firefox ? Is there a better way of doing it ?

Moving from version 0.6 of Cassandra to version 0.7

In order to use the latest build of Hector you need to be running a cluster based on version 0.7 of Cassandra. For me this gave me some problems. For a start I’m running my cluster on windows based machines, and it’s been running fine. However beta 1 of Cassandra 0.7.0 does not include the necessary tools for windows to convert the config files and read the schema.

So to get my cluster updated I added a linux box to it and installed a version 0.6.0 of Cassandra and joined the cluster. I installed 0.7.0 on that and attempted to use config-convertor to convert storage-conf.xml to Cassandra.yaml. For some reason that got in a horrible mess so it was back to the original Cassandra.yaml and import the settings manually. For my simple configuration this wasn’t a pain.

Once that’s done I upgraded the windows machines to version 0.7.0.

The next step is to run schematool. Without running this, your cluster will not have any schema’s in the database. This needs to be done from the linux command line:

schematool 134.36.xx.yyy 8080 import

does the job.

See http://www.riptano.com/blog/live-schema-updates-cassandra-07 and http://wiki.apache.org/cassandra/LiveSchemaUpdates for more details.

ConsistencyLevel in Hector and Cassandra

I started playing with failover in Casssandra the other day and rapidly found myself in a bit of a pickle. I had a 2 node development environment allowing me to play with Cassandra and start developing Hector programs. So I decided to turn one machine off and see if my program would carry on as normal.

It didn’t. Not much of a failover I thought. To make matters worse if attached via the Cassandra cli client, I could retrieve data from my single node. What was going on ?

Turns out this was all to do with the consistency of my cluster. Take a look at the consistency section of:

http://wiki.apache.org/cassandra/API

There are multiple levels of consistency available in a Cassandra cluster, and they can be different for read and write operations. What I hadn’t realized is that Hector defaults to a consistency of QUORUM whereas cli defults to a consistency of ONE (I believe). So with a two node cluster, Hector will fail on reads if one goes down.

Adding Nodes

One way to get round the problem is to add more nodes and up the replication level in the conf files to 4. I added two notes that where not part of the seeding process and used the Autoboostrap option in he conf file to join the cluster. This has kind of solved the problem. I can now read from one of the bootstrapped nodes if any of the other nodes goes down, but not from one of the seed nodes. I think more work on the configuration and layout of the cluster is needed by me.

Changing the consistency in Hector

You can of course change the consistency in Hector. The code here refers to Hector 0.6.15 and above (for now). The first thing we need to do is implement a consistencylevelpolicy. Here’s an example based on the default consistency implementation in Hector

import me.prettyprint.cassandra.model.*;
import org.apache.cassandra.thrift.ConsistencyLevel;

public final class MyConsistancyLevel implements ConsistencyLevelPolicy {

@Override
public  ConsistencyLevel get(OperationType op) {
   switch (op){
      case READ:return ConsistencyLevel.QUORUM;
      case WRITE: return ConsistencyLevel.ONE;
      default: return ConsistencyLevel.QUORUM; //Just in Case
   }
}
@Override
public ConsistencyLevel get(OperationType op, String cfName) {
   return ConsistencyLevel.QUORUM;
}
}

In this example we set the READ consistency to QUORUM and the WRITE consistency to ONE. (I’ve also included a default consistency just in case !). Once we’ve got this class we can set the consistency for the keyspace like this:

ConsistencyLevelPolicy mcl = new MyConsistancyLevel();
ko.setConsistencyLevelPolicy(mcl);

And that’s it !

Many thanks to Ran Tavory and Colin Vipurs for their help on this.

As ever comments and so on gratefully received.

A brief note about using Clusters in Hector V2 API

Recently Hector, the java library for access to Cassandra dB was updated to version 2. Now it’s time to explore moving jBloggyAppy over to the new API. In this blog I’m briefly going to look at Hector V2’s clustering options which greatly improve on version 1. To create a cluster we use getOrCreateCluster from the Hector Factory (HFactory e.prettyprint.cassandra.model.HFactory.*; ) Ran Tavory recommends importing the static library to reduce typing !:

import static me.prettyprint.cassandra.model.HFactory.*;

Once we’ve done that we create the cluster

Cluster c = HFactory.getOrCreateCluster("MyCluster", "154.36.xx.yyy:9160");

And that’s it !

However, note that we are now connected to only the machine in the cluster that we have named. we can get a list of all machines in the cluster like this:

Set <String>hosts= c.getClusterHosts(true); 
Iterator it =hosts.iterator(); 
while (it.hasNext()) {  
   System.out.println(it.next()); 
} 

The problem here is that we haven't got the port number of each machine in the cluster, nor do can we find out about the topology. Thanks to the Hector mail list folks for pointing this out.

If we want to know the clusters name:

System.out.println(c.describeClusterName());

Finally should we want to use the cluster with a V1 pool:

CassandraClient client =c.borrowClient();

and release it in a finally clause:

} finally {
   c.releaseClient(client);
}