Introduction to Oozie

Hadoop, Pig, Hive and many other projects provide the foundation for storing and processing large amounts of data in an efficient way. Most of the time, it is not possible to perform all required processing with a single job. This lead to the need for a general-purpose system to run multistage Hadoop jobs with the following requirements:

• Should use an adequate and well understood programming model
• Be easy to troubleshoot and recover jobs when something goes wrong
• Should scale to support several thousand concurrent jobs
• Jobs should run in a server to increase reliability
• Should be a multi-tenant service to reduce cost of operation

Oozie was then created, able to run multistage jobs consisting of MapR Pig and SSh,