USER DEFINED FUNCTION IN PIG HADOOP

Embedded Mode :

 When we are not getting the desired functionality through built in transformation operator in Pig then we ahead with UDF.

Steps For Developing Pig UDF(USER DEFINED FUNCTION)

1. Write a Class that will extend the base class of EVAL FUNCTION syntax :Eval Function<string>
   
2. In order to write business logic we need to overwrite method called execute which takes tuple input 
   exec(tuple input)
3. We need to dependent compilation Jar File for compilation purpose we have to create our own jar file to be deployed in hadoop environment.
4. Add the same Jar file Main Script Using REGISTER Keyword.
   Register is the First Line of any script.

Built in transformation or operator in PIG

Load -

Foreach

Generate

Filter

Dump

Store 

Split

Describe

Illustrate

Explain

Order By

Group By

Cogroup 

Join

Union

Cross

LIMIT

Tokenize

Flatten

AggFunction such as max,min,avg,count etc

Distinct

Next Chapter : User Defind Function in Apache Pig

Mapreduce VS Apache Pig

Mapreduce                                                                                     Apache Pig

Need to write entire logic                                         Built in function are available
for join ,group,Filter etc

No of lines of code is too                                           10 lines of pig latin =200
much for simple fuctionality                                      lines of java


Time of effort in coding is                                         what took 4 hours to write in
high.                                                                           it takes 15 min in pig

Less Productive                                                          High Productive


next chapter : Built in transformation

Diffrent Flavour of Pig Execution

Diffrent Flavour of Pig Execution

Grunt Shell

This is the default  mode of Pig execution which  interact to shell irrespective of whether the output is sucess or failure.

Syntax 

Local Mode                                             HDFS Mode

Pig -x local (then enter)                               Pig (then enter)
grunt>                                                                   or
                                                              Pig -x mapred
                                                              grunt>

-------------------------------------------------------------------------------
Script Mode 

Instead of executing transformation at the grunt shell one after another we are going to inscribe bunch of pig statement in a single file which generally we are going to trigger for execution.

Syntax

Local Mode                                                                  HDFS Mode

pig -x local <<scriptname>>                                  pig <<scriptname>>

-x stand for execution mode

--------------------------------------------------------------------------------------------

Embedded Mode

If we are not acheiving the desired functionality using the pre-defined function then we can generally go ahead with the User defined function (UDF) i.e we developing the external piece of code and embedding in the main script.

Next Chapter : Mapreduce Vs Apache Pig

Diffrent modes of Pig Execution

In order to achieve the desired processing in Pig there are two modes available : -

1. Local Mode
2. HDFS (Hadoop Distributed File System) Mode Or MR Mode Or Clustered Mode

 Local Mode

In this Mode the Apache Pig Will take the input from LFS (Local File System) and the Output will also produced on the same i.e LFS only.

Note : -There is no intervantion of HDFS 

HDFS Mode Or MR Mode Or Clustered Mode

In this mode Apache pig will take the input from HDFS and the output will also produced on HDFS only.

Note : -There is no intervantion of LFS here


Continue to next Chapter :- Different Flavour of Pig Execution