luckyp71/hadoop-hbase-phoenix-zookeeper-integration

Hadoop, HBase, Phoenix, and Zookeeper Integration

Hadoop, HBase, Phoenix, and Zookeeper Integration

This repository is nothing but an example on how to integrate Apache Hadoop, Apache Zookeeper, Apache HBase, and Apache Phoenix on debian 8 (jessie).

Since the HBase's configuration on this example is in pseudo distributed mode, not the standalone one, so that we need Hadoop installed in our machine.

Prerequisites

1. JDK 1.8

2. Apache Hadoop-3.1.0

3. Apache Zookeeper-3.4.12

4. Apache HBase-2.0.0

5. Apache Phoenix-5.0.0-alpha-HBase

Configuration Steps

Hadoop Installation

1. Install and set java environment

2. Unpack and copy hadoop-3.1.0 package to particular path e.g. /home/usr/programs/hadoop-3.1.0 (tar -zxvf -C /home/usr/programs)

3. Go to hadoop-3.1.0 root directory

4. Edit and set JAVA_HOME in hadoop-env.sh (vi ./etc/hadoop/hadoop-env.sh and export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0.171)

5. Edit core-site.xml and set hadoop.tmp.dir as well as fs.default.name properties

6. Create data directory then create namenode as well as datanode directories inside it

7. Change the data directory and its subdirectories permissions for readable, writable and executable by everyone (chmod 7777 -R ./data/*)

8. Edit hdfs-site.xml and set dfs.replication, dfs.namenode.name.dir, and dfs.datanode.data.dir properties

9. Edit mapred-site.xml and set mapred.job.tracker as well as mapreduce.framework.name properties

10. Edit yarn-site.xml and set yarn.nodemanager.aux-services property

11. Format namenode by executing ./bin/hdfs namenode -format command

12. Set HDFS_DATANODE_USER, HADOOP_SECURE_DN_USER, HDFS_NAMENODE_USER, HDFS_SECONDARYNAMENODE_USER in start-dfs.sh and stop-dfs.sh (vi ./sbin/s*dfs.sh and set the values for all mentioned properties with root for example)

13. Set YARN_RESOURCEMANAGER_USER and YARN_NODEMANAGER_USER in start-yarn.sh and stop-yarn.sh (vi ./sbin/s*yarn.sh and set the values for both properties with root for example)

Zookeeper Installation

Unpack zookeeper-3.4.12 package to particular path e.g. /home/usr/programs/zookeeper-3.4.12 (tar -zxvf -C /home/usr/programs)

HBase Installation

1. Unpack hbase-2.0.0 package to particular path e.g. /home/usr/programs/hbase-2.0.0 (tar -zxvf -C /home/usr/programs)

2. Go to hbase-2.0.0 root directory

3. Edit and set JAVA_HOME in hbase-env.sh (vi ./etc/hbase/hbase-env.sh and export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0.171) as well as set HBASE_MANAGES_ZK=false

4. Edit hbase-site.xml and set the properties needed

Phoenix Installation

Unpack phoenix-5.0.0-alpha-hbase package to particular path e.g. /home/usr/programs/phoenix-5.0.0-alpha-hbase (tar -zxvf -C /home/usr/programs)

Run Hadoop

Go to hadoop-3.1.0 root directory and execute ./sbin/start-dfs.sh and ./sbin/start-yarn.sh commands sequentially

Run Zookeeper

Go to zookeeper-3.4.12 root directory and execute ./bin/zkServer.sh start

Run HBase

Go to hbase-2.0.0 root directory and execute ./bin/start-hbase.sh

Run Phoenix

1. Go to phoenix-5.0.0-alpha-hbase root directory

2. Start the server by executing ./bin/queryserver.py start command

3. Execute ./bin/sqline-thin.py -a BASIC --auth-user=root --auth-password=root localhost:8765 command to enter phoenix shell