balabalayi/ambari-redis-service

Ambari plus Redis 5.0 integration.

Ambari-Redis-Service

Project

Redis 5.0.x

Feature	Description
Version	5.0.0
Service	Redis
Component-Master	Redis-Master
Component-Slave	Redis-Slave

Prerequisite

1. Internet Access (need to yum install ruby etc);
2. Ambari 2.x (test on 2.6.2.0) fully installed.

Features

1. Support redis single instance deployment: 1 instance;
2. Support redis cluster mode deployment: 3+ master instance and 3+ slave instance.

Installation

1. Download the source code.
2. Copy the project folder to ambari 's lib folder. Such as:

cp -r ./REDIS-5.0.x /var/lib/ambari-server/resources/stacks/HDP/2.6/services

3. Restart ambari-server:

ambari-server restart

4. Log into web then add service as usual:

5. Configure the service installation:

For single instance deployment

1). choose 1 node be the master:

2). choose 0 node be the slave:

3). configure down load url:

4). configure cluster disabled:

For cluster mode deployment

1). choose at least 3 nodes be the master:

2). choose at least 3 nodes be the slave:

3). configure down load url:

4). configure cluster enabled:

5). configure network:

6. Then continue the deployment and enjoy urself.^^

Background Maintaince

Here is some manually maintaince method specially for cluster operations (All the {} pamas can be found in Redis Configs in Ambari Web):

Check cluster:

{redis.base.dir}/src/redis-cli --cluster check {host_ip}:{port}

Create cluster:

{redis.base.dir}/src/redis-cli --cluster create host1_ip:{master-port} host2_ip:{master-port} host3_ip:{master-port} ...

Add master node to cluster:

{redis.base.dir}/src/redis-cli --cluster add-node {new_host_ip}:{master-port} {exist_cluster_host_ip}:{master-port}
{redis.base.dir}/src/redis-cli --cluster reshard {new_host_ip}:{master-port}

Add slave node to cluster:

{redis.base.dir}/src/redis-cli --cluster add-node {new_host_ip}:{master-port} {exist_cluster_host_ip}:{slave-port} --cluster-slave

Remove node from cluster:

# get node_id by check or info 
{redis.base.dir}/src/redis-cli --cluster check {host_ip}:{master-port}

# then remove
{redis.base.dir}/src/redis-cli --cluster del-node {host_ip}:{port} {node_id}

Clean up node:

# if node is already removed from cluster and stopped.
rm -fr {redis.base.dir} && rm -fr {log.file.dir} && rm -fr {pid.file.dir} && rm -fr {dir}

Re add a removed node back to cluster:

# remove the nodes.conf in dir
rm -fr {dir}/nodes.conf

# start redis (can be done on web)
{redis.base.dir}/src/redis-server {redis.base.dir}/redis.conf

# cleanup db
{redis.base.dir}/src/redis-cli
> flushdb

# use method above-mentioned add node to master or slave

Notice

1. Only support creating master node cluster for the first time.
2. Master node scaling need to be done mannually.
3. Support salve node incremental scaling.
4. Need more test if using on production.

ambari-redis-service's Issues

HDP 3.1 installation issues and fixes

Hello,
Bellow you can see some issues that I have encountered on HDP 3.1.
I present my quick fixes (I don't know if this is the best approach but maybe will help someone :) )

1. Error:
   File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users
   groups = params.user_to_groups_dict[user],
   KeyError: u'redis'

Fix:
https://community.hortonworks.com/questions/212871/add-custom-service-via-ambari-error.html
I have choose 'python configs.py' solution.

2. Error:
   ImportError: cannot import name format_hdp_stack_version

Fix:
In file REDIS-5.0.0/package/scripts/params.py
replace 'format_hdp_stack_version' with 'format_stack_version'

3. Error:
   File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/REDIS-5.0.0/package/scripts/redisSlave.py", line 30, in install
   ignore_failures=True
   TypeError: new() takes at least 2 arguments (1 given)

Fix:
Remove 'username' from 'User' object creation in both redisSlave.py and redisMaster.py.
E.g.
User(params.redis_user,
gid=params.redis_group,
groups=[params.redis_group],
ignore_failures=True
)

hdp_3.1_fix.txt