Chapter : 1 Hbase introduction

Hbase Introduction

Hbase is one of the component of hadoop built on top of HDFS and is ment for Real time random read/write. In against to the sequential file accessing of HDFS.

Hbase is opensource , distributed, scalable , fault tolerance , non -relational, multi-dimensional versioned, column oriented database which is built on top of BIG TABLES (BUILT ON THE ORIGIN OF GFS).

Google big tables ment for structured data only.

All the database will be organized by the means of tables only and whenever we inserting values in Hbase table for each and every values there will be a unique timestamp which will be generated by HBASE TABLES automatically.

HBASE is not a [part of default installatipon of hadoop and hence therefore we need to install it on top of hadoop only.

IN SVN when we insert data and from other location B modified it it creates version.

SVN

Input.log

Input_1.log

A has saved it =Input.log

Some modification done = SVN Framework generated another copy Input_1.log

Timestamp :- its a row and column orientesaction point where data is stored, multiple data can be stored on same timestamp.

APACHE PIG BUILT IN TRANSFORMATION

COGROUP 

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
B = load 'data5' using PigStorage(' ') as (a3:int,a4:int);
C =  COGROUP A by a1,B by a4;
Dump C;

Group BY

A = load 'data3' using PigStorage(' ') as (a1:chararray,a2:int,a3:int);
B = Group A by a2;
C = foreach B generate group,COUNT(A);
Dump C;

CROSS 

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
B = load 'data5' using PigStorage(' ') as (a3:int,a4:int);
C = CROSS A,B;
Dump C;

DISTINCT 

A = load 'data7' using PigStorage(' ') as (a1:int,a2:int,a3:int);
B = DISTINCT A;
Dump B;

Filter 

A = load 'data7' using PigStorage(' ') as (a1:int,a2:int,a3:int);
B = Filter A by a2==5;
Dump B;

COGROUP 

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
B = load 'data5' using PigStorage(' ') as (a3:int,a4:int);
C =  COGROUP A by a1,B by a4;
D = Foreach C generate group,FLATTEN (A);
Dump D;

A = load 'data3' using PigStorage(' ') as (a1:chararray,a2:int,a3:int);
B = Group A by a2;
  Describe B;
Dump B;
C = foreach B generate  group,Count(A);
Dump C;

JOIN 

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
B = load 'data5' using PigStorage(' ') as (a3:int,a4:int);
C =  JOIN A by a1,B by a4;
Dump C;

LIMIT 

A = load 'data6' using PigStorage(' ') as (a1:int,a2:int,a3:int,a4:int,a5:int);
B = LIMIT A 3;
Dump B;
 

Order by

A = load 'data6' using PigStorage(' ') as (a1:int,a2:int,a3:int,a4:int,a5:int);
B = Order A by $1;
Dump B;
 

SPLIT 

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
SPLIT A into B if a1==2, C if a1==3;
Dump B;
Dump C;
 

UNION 

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
B = load 'data5' using PigStorage(' ') as (a3:int,a4:int);
C = UNION A,B;
Dump C;
 

Managed Vs ExternalTables

-A Hive table that's not external is called a managed table.

-One of the main differences between an external and a managed table in Hive is that when an external table is dropped,

 the data associated with it  doesn't get deleted, only the metadata (number of columns, type of columns, terminators,

 etc.) gets dropped from the Hive metastore. When a managed table gets dropped, both the metadata and data get dropped.

 I have so far always preferred making tables external because if the schema of my Hive table changes,

 I can just drop the external table and re-create another external table over the same HDFS data with the new schema.

 However, most (if not all) of the changes to schema can now be made through ALTER TABLE or similar commands so my

 recommendation/preference to use external tables over managed ones might be more of a legacy concern than a contemporary

 one.

PARTITIONING vs BUCKTING-IN-HIVE

Basically both Partitioning and Bucketing slice the data for executing the query much more efficiently than on the non-sliced data. The major difference is that the number of slices will keep on changing in the case of partitioning as data is modified, but with bucketing the number of slices are fixed which are specified while creating the table.

Bucketing happen by using a Hash algorithm and then a modulo on the number of buckets. So, a row might get inserted into any of the bucket. Bucketing can be used for sampling of data, as well also for joining two data sets much more effectively and much more.

Hive Performance tuning

1)      Where Time  = int

When the time field within the where clause is specified query parser  will automatically detect partition to be processed.

Please not it work with integer type not with float.

Syntax

[GOOD]: SELECT field1, field2, field3 FROM tbl WHERE time > 1349393020

[GOOD]: SELECT field1, field2, field3 FROM tbl WHERE time > 1349393020 + 3600

[GOOD]: SELECT field1, field2, field3 FROM tbl WHERE time > 1349393020 – 3600

2)      Don not Use Distinct with Count in hive basically it do process this kind of query Only one reducer is used.

Syntax :  select count(Distinct Field1) from tablename;

Go for this

Select Count(1)

From ( Select Distinct Field1 from tablename ) t;