The LinkedIn Gradle Plugin for Apache Hadoop (which we shall refer to as simply the "Hadoop Plugin" for brevity) will help you more effectively build, test and deploy Hadoop applications.
In particular, the Plugin will help you easily work with Hadoop applications like Apache Pig and build workflows for Hadoop workflow schedulers like Azkaban and Apache Oozie.
The Plugin includes the LinkedIn Gradle DSL for Apache Hadoop (which we shall refer to as simply the "Hadoop DSL" for brevity), a language for specifying jobs and workflows for Hadoop workflow schedulers like Azkaban and Apache Oozie.
The Hadoop Plugin User Guide is available at [User Guide] (https://github.com/linkedin/linkedin-gradle-plugin-for-apache-hadoop/wiki/User-Guide).
The Hadoop DSL Language Reference is available at [Hadoop DSL Language Reference] (https://github.com/linkedin/linkedin-gradle-plugin-for-apache-hadoop/wiki/Hadoop-DSL-Language-Reference).
The project structure is setup as follows:
hadoop-plugin
: Code for the various plugins that comprise the Hadoop Pluginhadoop-plugin-test
: Test cases for the Hadoop Pluginli-hadoop-plugin
: LinkedIn-specific extensions to the Hadoop Plugin
Although the li-hadoop-plugin
code is generally specific to LinkedIn, it is included in the
project to show you how to use subclassing to extend the core functionality of the Hadoop Plugin.
To build the Plugin and run the test cases, run ./gradlew build
from the top-level project directory.
To see all the test tasks, run ./gradlew tasks
from the top-level project directory. You can run
an individual test with ./gradlew test_testName
. You can also run multiple tests by running
./gradlew test_testName1 ... test_testNameN
.
August 2015
Initial pull requests for Oozie versioned deployments and the Oozie Hadoop DSL compiler have been mergedAugust 2015
The Hadoop Plugin and Hadoop DSL were released on Github! See the LinkedIn Engineering Blog post for the announcement!July 2015
See our talk at the Gradle Summit!