Apache Oozie

Mohammad Kamrul Islam is currently working at Uber in data engineering team as a Staff Software Engineer. Previously, he worked at Linkedin for more than two years as Staff Software Engineer in the Hadoop development team. Before that, he worked at Yahoo for nearly five years as an Oozie architect/technical lead. His fingerprints can befound all over Oozie and is a respected voice in the Oozie community. He has been intimately involved with the Apache Hadoop ecosystem since 2009. Mohammad has a Ph.D. in Computer Science with a specialization in parallel job scheduling from Ohio State University. He received his MSCS degree from Wright State University, Ohio andBSCS from Bangladesh University of Engineering and Technology (BUET). He is a Project Management Committee (PMC) member of both Apache Oozie and Apache TEZ and frequently contributes to Apache YARN/MapReduce and Apache Hive. He was elected as the PMC chair and Vice-President of Oozie as part of the Apache Software Foundation from 2013 through 2015.

Aravind Srinivasan has been involved with Hadoop in general and Oozie in particular since 2008. He is currently a Lead Application Architect at Altiscale, a Hadoop-as-a-service company, where he helps customers with Hadoop application design and architecture. His association with Big Data and Hadoop started during his time at Yahoo, where he spent almost six years working on various data pipelines for advertising systems. He has extensive experience building complicated, low latency data pipelines and also in porting legacy pipelines to Oozie. He drove a lot of Oozie’s requirements as a customer in its early days of adoption inside Yahoo and later spent some time as a Product Manager in Yahoo’s Hadoop team where he contributed further to Oozie’s roadmap. He also spent a year after Yahoo at Think Big Analytics, a Hadoop consulting firm, where he got to consult on some interesting and challenging Big Data integration projects at Facebook. He has a Masters in Computer Science from Arizona State and lives in Silicon Valley.

Chapter 1Introduction to Oozie
Big Data Processing
Chapter 2Oozie Concepts
Oozie Applications
Parameters, Variables, and Functions
Application Deployment Model
Oozie Architecture
Chapter 3Setting Up Oozie
Oozie Deployment
Basic Installations
Advanced Oozie Installations
Chapter 4Oozie Workflow Actions
Workflow
Actions
Action Types
Synchronous Versus Asynchronous Actions
Chapter 5Workflow Applications
Outline of a Basic Workflow
Control Nodes
Job Configuration
Parameterization
The job.properties File
Configuration and Parameterization Examples
Lifecycle of a Workflow
Chapter 6Oozie Coordinator
Coordinator Concept
Triggering Mechanism
Coordinator Application and Job
Coordinator Job Lifecycle
Coordinator Action Lifecycle
Parameterization of the Coordinator
Execution Controls
An Improved Coordinator
Chapter 7Data Trigger Coordinator
Expressing Data Dependency
Example: Rollup
Parameterization of Dataset Instances
Parameter Passing to Workflow
A Complete Coordinator Application
Chapter 8Oozie Bundles
Bundle Basics
Bundle Specification
Bundle State Transitions
Chapter 9Advanced Topics
Managing Libraries in Oozie
Oozie Security
Supporting New API in MapReduce Action
Supporting Uber JAR
Cron Scheduling
Emulate Asynchronous Data Processing
HCatalog-Based Data Dependency
Chapter 10Developer Topics
Developing Custom EL Functions
Supporting Custom Action Types
Overriding an Asynchronous Action Type
Creating a New Asynchronous Action
Chapter 11Oozie Operations
Oozie CLI Tool
Oozie REST API
Oozie Java Client
The oozie-site.xml File
The Oozie Purge Service
Job Monitoring
Oozie Instrumentation and Metrics
Reprocessing
Server Tuning
Oozie High Availability
Debugging in Oozie
MiniOozie and LocalOozie
The Competition