Welcome to Apache Pig? Many Calls As hadoop Pig

As We all Know that Map Reduce is a low level language and require lots of java code to perform a small operation.

In Map Reduce Keeping Key , Value becomes a hectic for programmer.

To Solve these Problem Apache Pig has been Introduced : -

Apache Pig is one of the component of hadoop built on top of HDFS which is ment for processing the huge amount of data over mapreduce.

Apache Pig is High level and abstract language over mapreduce to Process the data .

Apache pig Will take the data summerization with respect to processing.

Processing in Apache Pig Will happen by the means of Transformation only or built on operation.f Transformation .Data Will flow through This Transformation only to get the desired output.
Hence We can call it as Data Flow or Transformational Language.


Pig Latin introduced by yahoo laboratory by there researcher and afterward it adopted by Apache Software Foundation .


Pig Installation

Note : -Apache pig was not bunding with default installation of hadoop ,hence we need to install Pig on Hadoop boxes.

Note : - Even though processing in Pig will happen with respect to multiple transformation each  & every transforamation is internally converted into map reduce job.


Click to Continue : Modes of Pig Execution

 

Download Data For Hadoop Practice

I am providing some links below to to download large amount of data for practice  :-GROUPLENS

• http://grouplens.org/datasets/movielens/

LABORA

• http://labrosa.ee.columbia.edu/millionsong/

• 
  

• INFOCHIMPS

• http://www.infochimps.com/collections/million-songs

 Stanford network data collectionhttp://snap.stanford.edu/data/index.html

• 
  

 Open FlightsCrowd sourced flight data http://openflights.org/

• 
  

 Flight arrival datahttp://stat-computing.org/dataexpo/2009/the-data.html Wikipedia datawikipedia data

• 
  

 OpenStreetMap.orgOpenStreetMap is a free worldwide map, created by people users. The geo and map data is available for download.
openstreet.org

• 
  

 Natural Earth Datahttp://www.naturalearthdata.com/downloads/

• 
  

 Geocommhttp://data.geocomm.com/drg/index.html

• 
  

 Geonames datahttp://www.geonames.org/

• 
  

 US GIS DataAvailable from http://libremap.org/

• 
  

Web data Wikipedia datawikipedia data

• 
  

 Google N-gram datagoogle ngram

• 
  

 Public terabyte dataWeb data crawl data linky

• 
  

 Freebase datavariety of data available from http://www.freebase.com/

• 
  

 Stack OverFlow datahttp://blog.stackoverflow.com/category/cc-wiki-dump/

• 
  

 UCI KDD datahttp://kdd.ics.uci.edu/
proceedings from Statistical machine Translation

• data.gov

•  data.gov.uk

• www.google.com/googlebooks/uspto.html

• 
  

 World Bank Datahttp://datacatalog.worldbank.org/

• 
  

 Public Health Data setshttp://phpartners.org/health_stats.html

• 
  

 National Institute of Healthhttp://projectreporter.nih.gov/reporter.cfm

• 
  

 Aid informationhttp://www.aidinfo.org/data

• 
  

 UN Datahttp://data.un.org/Explorer.aspx

• • RefineDemoData.zip

WordCount MapReduce Programme

import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.examples.WordCount.IntSumReducer;
import org.apache.hadoop.examples.WordCount.TokenizerMapper;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;




public class WordCountNew {


 // mapper class
 public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>
 {
  private static final IntWritable one=new IntWritable(1);
  private Text word=new Text();
  public void map(Object key,Text value,Context cont) throws IOException, InterruptedException
  {
   StringTokenizer itr=new StringTokenizer(value.toString());
   while(itr.hasMoreTokens())
   {
    word.set(itr.nextToken());
    cont.write(word, one);
   }
  }
 }

 //Reducer Class
 public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable>
 {
 private IntWritable result=new IntWritable();
 public void reduce(Text key,Iterable<IntWritable> values,Context cont) throws IOException, InterruptedException
   {
    int sum=0;
    for(IntWritable val: values)
    {
     sum+=val.get();
    }
    result.set(sum);
    cont.write(key, result);
   }
 }


 public static void main(String[] args) throws Exception
 {

  //Configuration for Job,jar

  Configuration conf=new Configuration();
  Job j=new Job(conf,"Batch48S-Wordcount");
  j.setJarByClass(WordCountNew.class);

  // mapper,reducer,combiner class

  j.setMapperClass(TokenizerMapper.class);
  j.setCombinerClass(IntSumReducer.class);
  j.setReducerClass(IntSumReducer.class);

  //Output key, Value, data type details
  j.setOutputKeyClass(Text.class);
  j.setOutputValueClass(IntWritable.class);

  // Hdfs input and output path

  FileInputFormat.addInputPath(j,new Path(args[0]));
  FileOutputFormat.setOutputPath(j,new Path(args[1]));
  //execution

  System.exit(j.waitForCompletion(true)?0:1);
 }

}

BIG DATA

We are Living in Data world so every day lots of data is generation by

internet usage such uploading pic,comment like on facebook, gmail etc

Everyday so single usage create min 100 log on gmail...


To Store and Process the So huge amount of data Hadoop come into Picture...

Please Refer Below blog :-

http://hdfs-bigdata-hadoop.blogspot.in

Bundle Knockout Data-Bind

In one my projects I ran into a scenario where I need to specify multiple bindings for a textbox. Here is the code I used to this multiple bindings

<input type="text" data-bind="value: model.Allergies.Value,

css: {focusctrl: model.Allergies.ColourType() === 1,

correctctrl: model.Allergies.ColourType() === 2},

event: { focus: function() {$root.resetColour(model.Allergies)} }" />

The problem here is have several of these text boxes (about 50, I guess not a good design, but is what the user wants). I don’t want to repeat this binding several times (DRY). I want to bundle these bindings (like a function) so that I can avoid duplication and keep the code clean.

Luckily Knockout provides the functionality I am looking. It provides “ko.applyBindingsToNode” to dynamically do the databinding. So I defined a custom binding in which I am in turning doing the databinding I want. Here is my custom binding

ko.bindingHandlers.notesvalue = {

    init: function (element, valueAccessor, allBindingsAccessor, data) {

        var field = valueAccessor();

        field.focusctrl = ko.computed(function () { return field.ColourType() === 1 });

        field.correctctrl = ko.computed(function () { return field.ColourType() === 2 });

        ko.applyBindingsToNode(element, {

            value: field.Value,

            css: { focusctrl: field.focusctrl, correctctrl: field.correctctrl },

            event: {

                focus: function() {

                    if(field.ColourType() === 1){

                        field.ColourType(0);

                    }

                }

            }

        });

    }

};

In the above code, I am just using the initializer and doing the bundling of the value, css and event data-binding. As you see, in the above code I am also defining computed attributes. That’s neat.

With my bundling code my databinding is reduced to as below

<input type="text" data-bind="notesvalue: model.Allergies" />

This is something I can live with. J.