Snappy Compression fails for Apache Cassandra on a Raspberry Pi

Although I’ve managed to get Apache Casssandra running on a Raspberry Pi I’ve been struggling to make any use of it.    I’ve been using the latest build of Cassandra (1.1.0) and when ever I’ve tried creating a column family I’ve been getting the following error:

SnappyCompressor.create() threw an error: org.xerial.snappy.SnappyError [FAILED_TO_LOAD_NATIVE_LIBRARY] null

As you are aware compression on Column Families was introduced in  Cassandra 1 for space saving and increased disk IO (http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-compression).  By default the compression is Snappy using the Google Snappy Library (http://code.google.com/p/snappy-java/ ) which is a wrapper for Google Snappy (http://code.google.com/p/snappy/) written in C++.  There is no port for Google Snappy on the Pi and my initial attempts to build one have not been successful (autogen.sh is failing with undefined macro AC_DEFINE)

You should be able to create a column family with DeflateCompressor from  cqlsh as follows:

create TABLE users (KEY varchar Primary key, password varchar, gender varchar) WITH compression_parameters:sstable_compressor = 'DeflateCompressor';

However that’s failing with the same Snappy Compressor error.   Describing the Keyspace with Cassandra-cli gives the following:

Keyspace: test:
  Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
  Durable Writes: true
    Options: [replication_factor:1]
  Column Families:
    ColumnFamily: users
      Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type
      Default column value validator: org.apache.cassandra.db.marshal.UTF8Type
      Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
      GC grace seconds: 864000
      Compaction min/max thresholds: 4/32
      Read repair chance: 0.1
      DC Local Read repair chance: 0.0
      Replicate on write: true
      Caching: KEYS_ONLY
      Bloom Filter FP chance: default
      Built indexes: []
      Column Metadata:
        Column Name: password
          Validation Class: org.apache.cassandra.db.marshal.UTF8Type
        Column Name: gender
          Validation Class: org.apache.cassandra.db.marshal.UTF8Type
      Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
      Compression Options:
        sstable_compressor: DeflateCompressor
        sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor

Note the two compressors in action.  I’m not sure quite what’s going on, or if this is the correct result at the moment but I’m continuing to investigate (any answers gratefully received.    So for now it’s back to Apache Cassandra 0.8.10 which doesn’t have compression and seems to work on the Pi (once the env file has been changed).

Update

Thanks to Jonathan Ellis for pointing out the command should be:

Create TABLE users2 (KEY varchar Primary key, password varchar, gender varchar) WITH compression_parameters:sstable_compression = 'DeflateCompressor';

Starting Cassandra on Raspberry Pi restart

(may be useful for other Raspberry Pi services)

A couple of days ago I blogged about getting Cassandra running on a  Raspberry Pi,  a fairly straight forward procedure!    However the way I had it set up required Cassandra to be restarted manually each time the Pi was turned on which is not ideal.  We really want our Cassandra to start each time the Pi is started and to run as a service in the background.  Services on Debian (which I am using) are  defined in Init.d and are managed by the update-rc.d command.     So, we need a command script to put into /etc/init.d that will be run when the Pi restarts.  There is an example script at: http://www.jansipke.nl/centos-cassandra-init-start-stop-script/  but for the pi this has a number of problems.    


The first problem is that the script uses the  daemon command to run the Cassandra script.  This command does not exist under Debian instead we will need to use the start-stop-daemon command.  Instead of :


daemon $CASSANDRA_BIN -p $CASSANDRA_PID >> $CASSANDRA_LOG 2>&1


we will use:


start-stop-daemon --start  --pidfile $CASSANDRA_PID --startas $CASSANDRA_BIN -p $CASSANDRA_PID  >> $CASSANDRA_LOG 2>&1


The second problem is that usleep doesn’t exist under this Debian, use “sleep 0.500000” instead.  Finally we will need to add the java path and java_home to the start of the file:


PATH=$PATH:/usr/local/bin/java
JAVA_HOME=/usr/local/bin/java


The full file is available on github  here: https://github.com/acobley/CassandraStartup.  Remember you will need to change the variables that point to the locations of Cassandra and it’s PID file and lock file.   Once the file is copied to /etc/init.d (remember to change I so it’s executable “sudo chmod +x Cassandra”) you can use update-rc.d  to add it to all the /etc/rc?.d files:


update-rc.d  cassandra defaults


No doubt this procedure and the Cassandra file can be improved !


For more info on the update-rc.d file see: http://www.debuntu.org/how-to-manage-services-with-update-rc.d

Apache Cassandra on a Raspberry Pi

One of the  reason I got hold of a Raspberry Pi (the $35 arm based Linux

machine) was to play around with building a cluster of them  for handling

"Big Data".  This is a real exercise in tinkering, very much a “what if” scenario.  The first thing I wanted to play with was getting Apache Cassandra to run on  the Pi.  Of course Cassandra is built with Java, there is no java on the Pi out of the box.    Several people suggested building Openjdk (http://openjdk.java.net/) but I plumbed for Oracle’s java SE for  embedded available here:

http://www.oracle.com/technetwork/java/embedded/downloads/javase/index.html

Download Java SE for Embedded 7 (ARMv6/7 Linux - Headless )and install it on the PI.  Once done and with the path and java_home correctly set you should be able to run the java on your pi.

Next get hold of a version of Apache Cassandra (http://cassandra.apache.org/)  I was using version 1.1.0.  Install it as usual.    If yo u try and run Cassandra from the bin directory it will fail to start with the serror:

“Invalid initial eden size: -Xmn0M”

The problem is that cassandra-env.sh is trying to work out the heap size by getting the  amount of memory on the machine and multiplying it it by the number of processors (line 69)

max_sensible_yg_in_mb=`expr $max_sensible_yg_per_core_in_mb "*" $system_cpu_cores`

The number of system cpu cores (line 22 or there abouts):

system_cpu_cores=`egrep -c 'processor([[:space:]]+):.*' /proc/cpuinfo`

is failing on the Pi (there is no processor line in /proc/cpuinfo).  For now I’ve manually altered cassanda-env.sh  to report only on core. 

system_cpu_cores=1

Cassandra now runs on the Pi.     There must be a better way of doing this( my Linux programming is failing me for the moment)  but for now I’m just tinkering so it will do.

Next up, try and get some performance figures for the Pi running Cassanadra.

Update

The correct way to fix this is to add change the Linux section of cassandra-env.sh to

           system_memory_in_mb=`free -m | awk '/Mem:/ {print $2}'`

            system_cpu_cores=`egrep -c 'processor([[:space:]]+):.*' /proc/cpuinfo`

            if [ "$system_cpu_cores" -lt "1" ]

            then

               system_cpu_cores="1"

            fi

            echo "Linux"

            echo "memory" $system_memory_in_mb

            echo "cores" $system_cpu_cores

Generate rails from a ERD ?

This isn't much of a blog post, it's a question !  I love ERDs (Entity Relationship Diagrams)  for database design and have been known to produce them before the database or the code .  Recently I've been doing a lot of work in Ruby on Rails which generates the database and model code for you, hiding the database under a ORM layer.  This is all very well and good  but you can see my problem.  No longer can I build  a ERD, check it's normalisation  nd optimise it with a bit of de-normalisation.

So, the question is:

Is there a tool out there that will take a ERD and generate the rails commands to build a app from the ERD.  I mean create the models and generate the migrations at least  ?

Suppose you had an ERD like this:

When I want to generate rails this would do something like the following

rails generate scaffold Person name:string
rails generate scaffold Town name:string<
rake db:migrate

It would then go in and alter the model for Person to set up the relationship to town.


Answers on a postcard !

Setting a home directory link in Rails 3

If you need to set up a path to the home directory in Rails 3  you need to change your routes.rb  file to create a named path.  In your routes.rb file you create the following mapping


match '/' => "home#index", :as => :home

(

this assumes your rails app has been created with 

rails generate controller home index

)

you can then create a link using linked_to

<%= link_to "Home", home_path %>

The point here is we've created the named link as "home"  but the path is referred to home_path.

Fun with the Independent's URL's

So:

This blog points out that you can alter URL's at "The Independent" to say anything you wish.  For instance @girlgeeks produced this one

http://www.independent.co.uk/environment/Cameron-destroys-environmental-laws-inUK-26-2270274.html

And

http://www.independent.co.uk/news/business/news/rupert-murdochs-is-a-PROPAGANDA-peddling-bampot-2270217.html

The question is, will they find their way into google search engine ? Lets wait a day or to and see !

General Advice for commissioning a web service (web site)

Contract
Ensure you are clear what you are going to get for the money ! Make a tick list of what will be delivered and produce a Gantt chart to track the progress. Ask what development methodology the programmers are going to follow, agile or waterfall. Make sure they explain their methodology so you are clear when you are going to meet with them and how you can track progress.

Browser support
Define which Browsers should be supported. At a very minimum this should include:

I.E9, I.E8
Firefox 3.6 and 4
Chrome 9 and 10
Safari 4

HTML version
Define the version of HTMl you want, make sure your website validates correctly to that version. If you plumb for HTML 5 make sure it displays correctly on the supported browsers. If you use HTML 5, use roles correctly.

CSS version
Again, check the supported browsers will support the version of CSS you choice. If you use CSS3, use only features supported by all browsers.
Interface

Ensure the interface is restful. Ensure return codes are meaningful. This will help when you want to create a mobile verson of the site or create mobile apps that use the site as a service.

A restful interface should help with jQuery support as well. Make sure your interface supports returning data as JSON at a the very least and a RSS feed as an option
Security

Escape all input to avoid XSS attacks . For PHP use htmlentities, all other languages should have something similar

All input and queries to the database should either be Paramatised or better still stored procedures should be used.

Any uploaded content should be limited to known file types. The content of the file should be verified and the file should be stored outside the webserver directories. Download should be anonymous so the file system is obscured.

Version Control
What version control system will they use. If they use a public one are they aware of any copyright problems that it may bring

Testing
How are they going to test the site ? Do they propose to use unit testing, if so will it be automated ?