Well I am back.
I have done hell lot of hadoop implementation for various fortune 500 clients. But i still believe TTSL(Tata Tele Services Limted) was one of the biggest revelation and a milestone in my life. They use case was little off-beat as generally the use cases that I came across was mostly data intensive batch based processing jobs in hadoop. For example, an ETL implementation at an Investment bank of US (where Morgan always chases :p) or may be the one at the biggest leading retail enterprise. Even for HBase implementation is was a weird use case It had all the facet that you can see in a proper production environment. But then it wasn't the use case that was the barrier but then it was the bureaucracy that made this project much more interesting and memorable.
So what is the fuzz about. The use case was pretty simple. Given a CDRs, you need to find call records up to a year either by phone number, IMEI number, Cell tower number or network switch number. The data size was around 20 TB (now 25TB). So TTSL was using Oracle databases that took (according to them) in and around 41 sec for finding call records of by a given phone number on 4TB which was in and around 3-4 months. At first look, it appears to be simple, well it is simple enough. I had to come up with a simple plan to execute this project with given time frame of a month. If you had to use oracle you would create indexes and query on them. Well, there is no auto Secondary indexing on HBase but it is really quick with key based jumps. So we utilized this heavily in our design and therefore finally it was a really good working model to go ahead with.
Well, No one told me that going forward with an open source product in an enterprise is really big challenge. Firstly, Sale guys from other database companies are really good in selling their stuff. Before we could implement our solution, we already had benchmarks for same use case from different vendors. Another challenge is the opposition from the infrastructure teams. Well, I am not sure you would face this in any other enterprise we surely had initially. In an enterprise, there is always bureaucracy and with bureaucracy there is always delay in delivery. We faced the same. Not only that, when the vision of the stake holder is different it is very difficult to change it. But then at end, everything comes down to the operational cost.
Anyways, So we implemented our solution on a SAN with RAID level 5(which i believe is not ideal for hadoop). If you are implementing it on SAN, it is necessary to make sure that you have physical dedicated disk array and not logical ones. So, overall we had around 6 machines machine hbase on hadoop cluster with 16 cores each connected with 4 * 1Gbps lines bonded together. If you are a newbie, you will use directly upload data into the table in hbase. The problem is it will explode the data as HBase for each key value stores in and around 15 more parameters like Row length,key offset etc. So how can we stop HBase from exploding data? Well the solution is simple. Firstly we need to keep our column names as small as possible. Secondly we need to apply some kind compression on HBase. Possible options are GZ, LZO and snappy. I believe snappy is the best among them but then we used GZ which is kind of good enough. Another issue you would sure face is loading data in HBase. There are two ways you can bulk load data into HBase- using tableoutput format or Hfileoutput format. One of the biggest challenge is devising a strategy of using them. Hfileoutput format is addictive but then it has a really big challenge. Number of reducers started by HFileoutput format is equivalent to number of regions in HBase now the problem is when your number of regions is small, it works like charms (well it may not work like charm in some cases where all the data goes into one region) but the moment your regions you will hit the bottle neck. We used split and force merge strategy to overcome this bottleneck.
Now here comes the fun part. When we were implementing the solution, everything was pretty smooth. We got benchmarks for 5GB of data and then 10GB of data. Then hadoop started crying, no one knew why it was cranking up. CPU utilization was not going beyond 40% and most of he mappers started dying. Even we had the moment of "shit hitting the fan" . We couldn't figure out the problem. We tried everything so everyone started blaming open sourceness of hadoop as the problem. Then day, by chance we discovered the problem. The issue was that the machines that we were using were 64 bit whereas the Java installed on them was 32 bit. And from there we took off and finished the implementation. Even the infrastructure people started believing that it can work and it worked. In the final test, We tested around 20TB of CDR and the average response time was around 500 milliseconds.
I have done hell lot of hadoop implementation for various fortune 500 clients. But i still believe TTSL(Tata Tele Services Limted) was one of the biggest revelation and a milestone in my life. They use case was little off-beat as generally the use cases that I came across was mostly data intensive batch based processing jobs in hadoop. For example, an ETL implementation at an Investment bank of US (where Morgan always chases :p) or may be the one at the biggest leading retail enterprise. Even for HBase implementation is was a weird use case It had all the facet that you can see in a proper production environment. But then it wasn't the use case that was the barrier but then it was the bureaucracy that made this project much more interesting and memorable.
So what is the fuzz about. The use case was pretty simple. Given a CDRs, you need to find call records up to a year either by phone number, IMEI number, Cell tower number or network switch number. The data size was around 20 TB (now 25TB). So TTSL was using Oracle databases that took (according to them) in and around 41 sec for finding call records of by a given phone number on 4TB which was in and around 3-4 months. At first look, it appears to be simple, well it is simple enough. I had to come up with a simple plan to execute this project with given time frame of a month. If you had to use oracle you would create indexes and query on them. Well, there is no auto Secondary indexing on HBase but it is really quick with key based jumps. So we utilized this heavily in our design and therefore finally it was a really good working model to go ahead with.
Well, No one told me that going forward with an open source product in an enterprise is really big challenge. Firstly, Sale guys from other database companies are really good in selling their stuff. Before we could implement our solution, we already had benchmarks for same use case from different vendors. Another challenge is the opposition from the infrastructure teams. Well, I am not sure you would face this in any other enterprise we surely had initially. In an enterprise, there is always bureaucracy and with bureaucracy there is always delay in delivery. We faced the same. Not only that, when the vision of the stake holder is different it is very difficult to change it. But then at end, everything comes down to the operational cost.
Anyways, So we implemented our solution on a SAN with RAID level 5(which i believe is not ideal for hadoop). If you are implementing it on SAN, it is necessary to make sure that you have physical dedicated disk array and not logical ones. So, overall we had around 6 machines machine hbase on hadoop cluster with 16 cores each connected with 4 * 1Gbps lines bonded together. If you are a newbie, you will use directly upload data into the table in hbase. The problem is it will explode the data as HBase for each key value stores in and around 15 more parameters like Row length,key offset etc. So how can we stop HBase from exploding data? Well the solution is simple. Firstly we need to keep our column names as small as possible. Secondly we need to apply some kind compression on HBase. Possible options are GZ, LZO and snappy. I believe snappy is the best among them but then we used GZ which is kind of good enough. Another issue you would sure face is loading data in HBase. There are two ways you can bulk load data into HBase- using tableoutput format or Hfileoutput format. One of the biggest challenge is devising a strategy of using them. Hfileoutput format is addictive but then it has a really big challenge. Number of reducers started by HFileoutput format is equivalent to number of regions in HBase now the problem is when your number of regions is small, it works like charms (well it may not work like charm in some cases where all the data goes into one region) but the moment your regions you will hit the bottle neck. We used split and force merge strategy to overcome this bottleneck.
Now here comes the fun part. When we were implementing the solution, everything was pretty smooth. We got benchmarks for 5GB of data and then 10GB of data. Then hadoop started crying, no one knew why it was cranking up. CPU utilization was not going beyond 40% and most of he mappers started dying. Even we had the moment of "shit hitting the fan" . We couldn't figure out the problem. We tried everything so everyone started blaming open sourceness of hadoop as the problem. Then day, by chance we discovered the problem. The issue was that the machines that we were using were 64 bit whereas the Java installed on them was 32 bit. And from there we took off and finished the implementation. Even the infrastructure people started believing that it can work and it worked. In the final test, We tested around 20TB of CDR and the average response time was around 500 milliseconds.
Wow mate!! superb...
ReplyDeleteCan you please tell me why you chose HBase and not just use HIVE? I have a problem differentiating scenarios where to use which?
The reason we chose HBase was because they were low latency query with SLA under 3 seconds
DeleteHow did you compose rowkey like? I was thinking about phone number and timestamp but would like to hear opinion from your side as you have done tests. What about IMEI?
ReplyDeleteWell, I would do something like
ReplyDeletebucketId#IMEI#Phno#timestamp
But also create a lookup or reverse index using phone number on this master table for faster lookup
If I design row key like you suggested and search criteria is only phone number am I still able to search only by row key even if I don't have IMEI number?
ReplyDeleteAnd if you use bucketId for distribution, can I use scan operation with start row and stop row because I only have partial row key?