NoSQLite4j - Really light NoSQL database

Introducing NoSQLite4j

NoSQLite4j offers a really lite JSON based database. It is really easy to bundle with various projects. Has only dependency of java-json.jar.

It can be found here

It is really easy to implement.

Scanning the table;

import java.io.IOException;

import org.nosqlite4j.core.dml.Scan;

public class TestScan {
    public static void main(String [] args) throws IOException, NullPointerException
    {
        try{
        Scan table = new Scan("Shop");
        System.out.println(table.scan());
        System.out.println(table.getRowCount());
        }
        catch(Exception e)
        {
            e.printStackTrace();
        }
      
    }
}

CRUD operations

//put

import java.io.IOException;

import org.nosqlite4j.core.dml.Put;

public class TestPut {
    public static void main(String [] args) throws IOException, NullPointerException
    {
        try{
        Put table = new Put("Shop","user2","shopped1","date1","article5");
        table.add();
        }
        catch(Exception e)
        {
            e.printStackTrace();
        }
      
    }

}

//get

import java.io.IOException;

import org.nosqlite4j.core.dml.Get;

public class TestGet {
    public static void main(String [] args) throws IOException, NullPointerException
    {
        try{
       
        Get table = new Get("Shop6","user2","shopped10","date1");
        System.out.println(table.get());
        Get table2 = new Get("shop2","user2","shopped");
        System.out.println(table2.get());
   
        Get table3 = new Get("Shop3","user2");
        System.out.println(table3.get());
        }
        catch(Exception e)
        {
            e.printStackTrace();
        }
       
    }
}

Hadoop and HBase at production

Well I am back.
I have done hell lot of hadoop implementation for various fortune 500 clients. But i still believe TTSL(Tata Tele Services Limted) was one of the biggest revelation and a milestone in my life. They use case was little off-beat as generally the use cases that I came across was mostly data intensive batch based processing jobs in hadoop. For example, an ETL implementation at an Investment bank of US (where Morgan always chases :p) or may be the one at the biggest leading retail enterprise. Even for HBase implementation is was a weird use case  It had all the facet that you can see in a proper production environment. But then it wasn't the use case that was the barrier but then it was the bureaucracy that made this project much more interesting and memorable.

So what is the fuzz about. The use case was pretty simple. Given a CDRs, you need to find call records up to a year either by phone number, IMEI number, Cell tower number or network switch number. The data size was around 20 TB (now 25TB). So TTSL was using Oracle databases that took (according to them) in and around 41 sec for finding call records of by a given phone number on 4TB which was in and around 3-4 months. At first look, it appears to be simple, well it is simple enough. I had to come up with a simple plan to execute this project with given time frame of a month. If you had to use oracle you would create indexes and query on them. Well, there is no auto Secondary indexing on HBase but it is really quick with key based jumps. So we utilized this heavily in our design and therefore finally it was a really good working model to go ahead with.

Well, No one told me that going forward with an open source product in an enterprise is really big challenge. Firstly, Sale guys from other database companies are really good in selling their stuff. Before we could implement our solution, we already had benchmarks for same use case from different vendors. Another challenge is the opposition from the infrastructure teams. Well, I am not sure you would face this in any other enterprise we surely had initially. In an enterprise, there is always bureaucracy and with bureaucracy there is always delay in delivery. We faced the same. Not only that, when the vision of the stake holder is different it is very difficult to change it. But then at end, everything comes down to the operational cost.

Anyways, So we implemented our solution on a SAN with RAID level 5(which i believe is not ideal for hadoop). If you are implementing it on SAN, it is necessary to make sure that you have physical dedicated disk array and not logical ones. So, overall we had around 6 machines machine hbase on hadoop cluster with 16 cores each connected with 4 * 1Gbps lines bonded together. If you are a newbie, you will use directly upload data into the table in hbase. The problem is it will explode the data as HBase for each key value stores in and around 15 more parameters like Row length,key offset etc. So how can we stop HBase from exploding data? Well the solution is simple. Firstly we need to keep our column names as small as possible. Secondly we need to apply some kind compression on HBase. Possible options are GZ, LZO and snappy. I believe snappy is the best among them but then we used GZ which is kind of good enough. Another issue you would sure face is loading data in HBase. There are two ways you can bulk load data into HBase- using tableoutput format or Hfileoutput format. One of the biggest challenge is devising a strategy of using them. Hfileoutput format is addictive but then it has a really big challenge. Number of reducers started by HFileoutput format is equivalent to number of regions in HBase now the problem is when your number of regions is small, it works like charms (well it may not work like charm in some cases where all the data goes into one region) but the moment your regions you will hit the bottle neck. We used split and force merge strategy to overcome this bottleneck.

Now here comes the fun part. When we were implementing the solution, everything was pretty smooth. We got benchmarks for 5GB of data and then 10GB of data. Then hadoop started crying, no one knew why it was cranking up. CPU utilization was not going beyond 40% and most of he mappers started dying. Even we had the moment of "shit hitting the fan" . We couldn't figure out the problem. We tried everything so everyone started blaming open sourceness of hadoop as the problem. Then day, by chance we discovered the problem. The issue was that the machines that we were using were 64 bit whereas the Java installed on them was 32 bit. And from there we took off and finished the implementation. Even the infrastructure people started believing that it can work and it worked. In the final test, We tested around 20TB of CDR and the average response time was around 500 milliseconds.

CloudMine is here

Some would consider it as difficult challenge

I think was the last minute changes in the final report that might save us from getting a fail grade.

But then it was fun.

So here I am .. Introducing Project CloudMine, an open source, auto-scaling (Ah! Rather Elastic) and a FUN application implemented on Open Nebula.

The challenge was to built a automated,elastic,load balancing application with a monitoring service on cloud services. We implemented  on Open Nebula on DAS4 systems at TU delft.

Installation steps are pretty simple. If you have Open Nebula cloud installed at your end. Then simply download it from Github and then

export PROJECT_CLOUD=<dir-to-ProjectCloud>

export PATH=$PATH:$PROJECT_CLOUD/bin

now there are two aspect left before it automatically starts acquiring Virtual Machines

in $PROJECT_CLOUD/conf    you will two configuration files - cloud_env and image_conf

In image_conf you need to paste the .one file which consists of you image details

whereas in cloud_env you need put your

user:cloudpass
RPC-XML service
POOL_SIZE

So the CloudMine reads this file for Open Nebula especially Pool size it automatically scales up or scales down based in this size.


Once you have set your configurations, you need to start the automation script.

auto user1 password

user1 and password are default user and credentials for CloudMine (yes we did implement ACL). It starts a service  so in the mean while if you chance the pool size, this will pick up the change and start or stop VMs.

The Load balancing on the other hand is kept simple and is implemented in a Round Robin fashion.
If you want to submit a task.Do as follows

submit user1 password <file> <word-to-be-mined>

We have a benchmark script that runs multiple tasks and collects the output and vm stats.

There are two kinds of monitoring implemented in CloudMine. Well one evolved from another. The one which is implmented by default is as follows. The moment a VM is acquired it starts a monitoring service in that VM. We have scripts that pull these logs and display.

Overall it is a fun application. We have identified we shortcoming and will improve it (i.e if we get an opportunity).

Keep following us for more exciting stuff.

Initiation inc

Well i have finally started my blog describing all open source projects i have been trying out these days.With the advent of acceptance of open source products in enterprise has opened by the infinite combinations of opens source products.What can you expect from this blog?? Well, lets explore each and every open source project in this world!!