sequenceiq/docker-ambari

Docker image with Ambari

Ambari on Docker

This projects aim is to help you to get started with Ambari.

Please note that this project is not officially supported by Hortonworks and may not be suitable for production use. It can be used to experiment with Hadoop on Docker but for a complete and supported solution please check out Cloudbreak.

Install Docker

Follow the description at the docker getting started page for your appropriate OS: (Linux, Mac, Windows)

OSX

Ambari containers started by ambari-function are using bridge networking. This means that you will not be able to communicate with containers directly from host unless you specify the route to containers. You can do this with:

# Getting the IP of docker-machine or boot2docker
docker-machine ip <name-of-docker-vm>
# or
boot2docker ip

# Setting up the
sudo route add -net 172.17.0.0/16 <docker-machine or boot2docker>
# e.g:
sudo route add -net 172.17.0.0/16 192.168.99.100

Note: the above mentioned route command will not survive a reboot and you need to execute again after reboot of your machine.

Starting containers

This will start (and download if you never used it before) an image based on Centos 7 with pre-installed Ambari 2.2.0 ready to install HDP 2.3.

This git repository also contains an ambari-functions script which will launch all the necessary containers to create a fully functional cluster. Download the file and source it:

. ambari-functions or source ambari-functions

Now you can issue commands with amb-prefix like:

amb-settings

To start a 3 node cluster:

amb-start-cluster 3

It will launch containers like this (1 Ambari server 2 agents 1 consul server):

CONTAINER ID        IMAGE                          COMMAND                  STATUS              NAMES
52b563756d26        hortonworks/ambari-agent       "/usr/sbin/init syste"   Up 9 seconds        amb2
ddfc8f00d30a        hortonworks/ambari-agent       "/usr/sbin/init syste"   Up 10 seconds       amb1
ca87a0fb6306        hortonworks/ambari-server      "/usr/sbin/init syste"   Up 12 seconds       amb-server
7d18cc35a6b0        sequenceiq/consul:v0.5.0-v6   "/bin/start -server -"    Up 17 seconds       amb-consul

Now you can reach the Ambari UI on the amb-server container's 8080 port. Type the amb-settings for IP:

amb-settings
...
AMBARI_SERVER_IP=172.17.0.17

Cluster deployment via blueprint

Once the container is running, you can deploy a cluster. Instead of going to the webui, we can use ambari-shell, which can interact with ambari via cli, or perform automated provisioning. We will use the automated way, and of course there is a docker image, with prepared ambari-shell in it:

amb-shell

Ambari-shell uses Ambari's Blueprints capability. It posts a cluster definition JSON to the ambari REST api, and 1 more json for cluster creation, where you specify which hosts go to which hostgroup.

Ambari shell will show the progress in the upper right corner.

Multi-node Hadoop cluster

For the multi node Hadoop cluster instructions please take a look at Cloudbreak.

If you don't want to check out the project from github, then just dowload the ambari-fuctions script, source it and deploy a an Ambari cluster:

curl -Lo .amb j.mp/docker-ambari && source .amb && amb-deploy-cluster

docker-ambari's Issues

ambari in container cannot use DNS server

I am working with ambari 1.7.0 and using ambari functions to start containers.

root@zqin:/home/zqin/Desktop/ambari-1.7.0# source ambari-functions
root@zqin:/home/zqin/Desktop/ambari-1.7.0# amb-deploy-cluster 3
starting an ambari cluster with: 3 nodes
[DEBUG] docker run -d --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb0 -h amb0.mycorp.kom sequenceiq/ambari:1.7.0 --tag ambari-server=true
WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.
4c549361f6a69f1330efcd8c924fa553a22c9fea7dbc78f81f6ac8fd8a71f14b
[DEBUG] docker run -d -e SERF_JOIN_IP=172.17.0.1 --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb1 -h amb1.mycorp.kom sequenceiq/ambari:1.7.0 --log-level debug

and I find a warning like:WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.

I use ubuntu in VM virtual box. When the installation completes, I can visit 172.17.0.1:8080 and ambari web UI can be shown.

However, I go into container amb0 and find ambari-server is not running. And the container cannot resolve any host like below:
bash-4.1# ping google.com
ping: unknown host google.com

Any when I check ambari server , it tells me it is not running like this:
bash-4.1# ambari-server status
Using python /usr/bin/python2.6
Ambari-server status
Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pid

Is the dns problem or other issues?

keychain.io is insecure and not always available

Piping to bash is not secure, and in addition the keychain.io service is not always available (currently getting python errors when confirming key upload). Providing another way to install image would be appreciated.

Edit: I contacted one of the developers and he said the site has not worked for some time now.

add volume support

is it possible to have volume support?

ambari server can not start

amb-deploy-cluster hung on "waits for ambari server". Because /var/lib/pgsql/data/postmaster.pid is exist, postgres start failed, ambari server can not start.
sequenceiq/ambari:1.7.0 sequenceiq/ambari:2.0.0 and sequenceiq/ambari:latest all have this issue.

Ambari cluster is created but nodes have connectivity issues

I have tried your one-step script for cluster creation on multiple computers on multiple networks. Although cluster creation works great, when looking at the nodes in the cluster from the ambari manager page, the nodes appear to have issues connecting to their own services. Is there a fix for this?

Problem adding new blueprint

I am trying to add a new blueprint that I got from querying a running cluster via the api. I saved the file in the current directory as blueprint.json and the ran the following to get the Ambari-shell;

docker run -it --rm -v $(pwd):/tmp -w /tmp sequenceiq/ambari-shell --ambari.host=$AMBARI_IP
Error: Unable to access jarfile ambari-shell.jar

And I have tried to get it from a S3 bucket and I get a error;

ambari-shell>blueprint add --url https://s3-us-west-1.amazonaws.com/ambari-blueprint/hdp.json
No blueprint specified

What is the proper way of adding blueprints form the ambari-shell?

Update:
I was however able to add the blueprint by using ADD in the Dockerfile to add it during the build.
I am running into a issue with a customized blueprint where we added Zeppelin to our stack.

ambari-shell>blueprint add --file blueprint.json
Cannot add blueprint: {
  "status" : 400,
  "message" : "The component 'ZEPPELIN_MASTER' in host group 'host_group_1' is not valid for the specified stack"
}

docker run not working with given command

Hi,
I'm trying to use the command in the README to run the ambari-server:

docker run -d -P -h amb0.mycorp.kom -e KEYCHAIN=<keychain@email> --name amb0  sequenceiq/ambari --tag ambari-server=true

But I'm getting this error:
Error response from daemon: Cannot start container xyz: [8] System error: exec: "--tag": executable file not found in $PATH

Output of docker ps -a:

CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS                     PORTS               NAMES
xyz                 sequenceiq/ambari         "--tag ambari-server="   5 seconds ago       Created                                        amb0

Docker version:
Docker version 1.8.2, build 0a8c2e3
OS: Ubuntu 14.04

Hint for creating a Hadoop 1.X or HDP 1.X based system

I am new to both Hadoop and Docker, however I got the 3 node cluster running on my Mac. I have accessed the Ambari UI by adding a -p switch to the amber server startup.

Do you have a hint on setting up Ambari with the older version of MapReduce So I can use Job Tracker and Task Tracker. I have some legacy Hadoop systems I am trying to mirror.

Time Drift

I have the latest install running but I have time drift between the amb0 and amb1 and amb2. It appears to the time between when I started the amb0 and amb1/amb2.

I then tried installing ntpd to try and correct the problem but I ran into the following issue:

date: cannot set date: Operation not permitted
http://serverfault.com/questions/508922/date-cannot-set-date-operation-not-permitted/508928#508928

I am looking at the Dockerfile to see if I can resolve the problem.

blueprint with pig and single-node-hdfs-yarn

HI

I want to use pig and single node cluster.
Is there any blueprint that i can use?

Ambari2.0.0 cluster creation fails

When using the command
curl -Lo .amb j.mp/docker-ambari-200 && . .amb && amb-deploy-cluster
The cluster deploys fine, the clients connect to the master, and the ambari shell runs.
The deployment fails with an error message: "Installation: FAILED"
This happens at about 10% configuration after the following is run:
cluster create --exitOnFinish true
Is there any way to debug this/has anyone else gotten it to work?

Some questions : start, commit, restart...

Hello,

I explain all my tasks.
I installed your docker-ambari on my computer (MacOs) and on a VM Centos with your script : amb-deploy-cluster. (I just add an option " -p 32777:8080 " on the ambari-server command to have access to the UI Ambari).
Everythink is ok, then I installed storm and others services and now I'm blocked.
Because when I stopped containers and tried to restart using : "docker start amb-consul amb-server amb1 amb2", Ambari don't see any agents (heartbeat lost) because containers changed IP addresses.
What are your recommandations about that ?
And how can I do backup my actions (install and configuration of storm for example) and start all the components after a stop the containers ? If I commit the image, what command I need to use after to start my 4 containers ?

Sorry I'm a little lost now.
Thank you.

Ambari-Server 1.7.0-ea issues

I'm following the steps declared in the blog for Ambari 1.7.0 early access, found here: http://blog.sequenceiq.com/blog/2014/09/05/apache-ambari-1-7-0-ea/

I've tried these steps on a couple of machines:
Mac OSX - Mavericks
CentOS 6.5 (AMI on AWS)

All images fail when following the outlined steps at the aforementioned page.

On all machines, the only outlined issue comes from Ambari-Shell progress monitor bar. The ambari-server seems to start fine, and the 'clustering' seems to happen successfully. However, once Ambari-Shell is launched and the default blueprint is set, along with the command "cluster build --blueprint multi-node-hdfs-yarn" is activated, I see immediately the report of:

Welcome to Ambari Shell. For command and param completion press TAB, for assistance type 'hint'.
ambari-shell>blueprint defaults
Default blueprints added
ambari-shell>cluster build --blueprint multi-node-hdfs-yarn
HOSTNAME STATE

amb1.mycorp.kom HEALTHY
amb0.mycorp.kom UNKNOWN
amb3.mycorp.kom HEALTHY
amb2.mycorp.kom HEALTHY"

AMB0 is always showing as unhealthy. This is occurring on CentOS6.5.
The next command that seemingly shows success is "cluster autoAssign", which declares amb0 as master (but yet is currently showing unknown status).

After that command, we move onto "cluster create --exitOnFinish true". This is when the "Installation" progress bar begins being displayed. After a short time, the progress bar immediately shows "FAILED" status. I'm not exactly sure why, either:

This is also the case only colleagues machine (MacOSX - Yosemite):

Any assistance on this is greatly appreciated! I'm doing nothing outside of what's defined in the blog as well.

ambari-server doesn't work with ambari-functions

I'm trying to run ambari-functions with a docker container built from your ambari-server folder's Dockerfile but it appears that that image is lacking the serf utilities required by this process. Am I looking in the wrong place? Is there somewhere for me to find a Dockerfile that will work with the ambari-functions file? I have to build it myself as I need to change the base OS for testing purposes. Thanks for any help you can provide.

ambari-functions do not work well on ZSH

Hi

I was running the following tutorial:
http://blog.sequenceiq.com/blog/2014/09/19/apache-tez-cluster/

First, when sourcing the .amb file, in ZSH I needed to pass the full pass.
So, this does not work:

$ . .amb

But this work:

$ . ~/.amb

After when running the following command:

$ amb-deploy-cluster 4

I get the error:

starting an ambari cluster with: 4 nodes
run-command:2: = not found

In bash, this does not happens.

Thanks.

public-hostname.sh no longer works in non-cloud environment.

Commit 61f8cc7 removed the logic that tests which cloud platform the container is running on. Hostnames are incorrect on locally run containers. This causes the UI links to point to strange locations.

How to use ambari-functions ?

I have no idea about code below:

docker@boot2docker:~/git/docker-ambari$ curl -Lo .amb j.mp/docker-ambari && . .amb && amb-deploy-cluster
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (52) Empty reply from server

Give me some keywords to google,please.
Thanks.

HDFS block corruption when running a new container

I am systematically facing HDFS / HBase block corruption when running a new container from an image of a healthy HDP cluster (single-node).

Steps followed:

• Creation of HDP cluster using sequenceiq/ambari Dockerfile. HDFS filesystem was healthy.
• Stop all services in Ambari web
• docker commit
• docker run new container from image
• Restart all services in Ambari,
• HDFS systematically reports corrupt and missing block issues.

Example of corruption
-bash-4.1# HADOOP_USER_NAME=hdfs hdfs fsck /
Connecting to namenode via http://og.mycorp.com:50070
FSCK started by hdfs (auth:SIMPLE) from /172.17.0.2 for path / at Mon Jun 08 11:33:06 EDT 2015
.
/app-logs/ambari-qa/logs/application_1433328045348_0001/og.mycorp.com_45454: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741858

/app-logs/ambari-qa/logs/application_1433328045348_0001/og.mycorp.com_45454: MISSING 1 blocks of total size 7080 B..
/app-logs/ambari-qa/logs/application_1433502507205_0001/og.mycorp.com_45454: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741976_1154. Target Replicas is 3 but found 1 replica(s).
.
/app-logs/ambari-qa/logs/application_1433502507205_0002/og.mycorp.com_45454: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741988_1166. Target Replicas is 3 but found 1 replica(s).
.
/apps/hbase/data/data/default/ambarismoketest/.tabledesc/.tableinfo.0000000001: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741841_1017. Target Replicas is 3 but found 1 replica(s).
.
/apps/hbase/data/data/default/ambarismoketest/ac82f75a8636f78f9629dd4b480106d2/.regioninfo: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741842

/apps/hbase/data/data/default/ambarismoketest/ac82f75a8636f78f9629dd4b480106d2/.regioninfo: MISSING 1 blocks of total size 50 B..
/apps/hbase/data/data/default/ambarismoketest/ac82f75a8636f78f9629dd4b480106d2/family/0ade395e2a9b49b8a6ce711d482788d8: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741863_1039. Target Replicas is 3 but found 1 replica(s).
..
/apps/hbase/data/data/hbase/meta/.tabledesc/.tableinfo.0000000001: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741828

/apps/hbase/data/data/hbase/meta/.tabledesc/.tableinfo.0000000001: MISSING 1 blocks of total size 372 B..
/apps/hbase/data/data/hbase/meta/1588230740/.regioninfo: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741827_1003. Target Replicas is 3 but found 1 replica(s).
.
/apps/hbase/data/data/hbase/meta/1588230740/info/8420cae8bce94280995695060a910546: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073742145_1325. Target Replicas is 3 but found 1 replica(s).
..
/apps/hbase/data/data/hbase/namespace/.tabledesc/.tableinfo.0000000001: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741834

/apps/hbase/data/data/hbase/namespace/.tabledesc/.tableinfo.0000000001: MISSING 1 blocks of total size 286 B..
/apps/hbase/data/data/hbase/namespace/14115c2297e3486d8f3f4ebf785fd11d/.regioninfo: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741835_1011. Target Replicas is 3 but found 1 replica(s).
.
/apps/hbase/data/data/hbase/namespace/14115c2297e3486d8f3f4ebf785fd11d/info/418efc3186ad4896978913edf793cec4: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741861_1037. Target Replicas is 3 but found 1 replica(s).
..
/apps/hbase/data/hbase.id: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741826

/apps/hbase/data/hbase.id: MISSING 1 blocks of total size 42 B..
/apps/hbase/data/hbase.version: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741825_1001. Target Replicas is 3 but found 1 replica(s).
.
/apps/hbase/data/oldWALs/og.mycorp.com%2C60020%2C1433773404945.1433773424585: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073742140

/apps/hbase/data/oldWALs/og.mycorp.com%2C60020%2C1433773404945.1433773424585: MISSING 1 blocks of total size 655 B..
/apps/hbase/data/oldWALs/og.mycorp.com%2C60020%2C1433773404945.1433773750783.meta: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073742144

/apps/hbase/data/oldWALs/og.mycorp.com%2C60020%2C1433773404945.1433773750783.meta: MISSING 1 blocks of total size 541 B..
/hdp/apps/2.2.4.2-2/hive/hive.tar.gz: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741989

/hdp/apps/2.2.4.2-2/hive/hive.tar.gz: MISSING 1 blocks of total size 83000677 B..
/hdp/apps/2.2.4.2-2/mapreduce/hadoop-streaming.jar: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741991

/hdp/apps/2.2.4.2-2/mapreduce/hadoop-streaming.jar: MISSING 1 blocks of total size 104996 B..
/hdp/apps/2.2.4.2-2/mapreduce/mapreduce.tar.gz: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741829_1005. Target Replicas is 3 but found 1 replica(s).

/hdp/apps/2.2.4.2-2/mapreduce/mapreduce.tar.gz: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741830

/hdp/apps/2.2.4.2-2/mapreduce/mapreduce.tar.gz: MISSING 1 blocks of total size 58479639 B..
/hdp/apps/2.2.4.2-2/pig/pig.tar.gz: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741990_1168. Target Replicas is 3 but found 1 replica(s).
.
/hdp/apps/2.2.4.2-2/tez/tez.tar.gz: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741960_1138. Target Replicas is 3 but found 1 replica(s).
.
/mr-history/done/2015/06/03/000000/job_1433328045348_0001-1433328283077-ambari%2Dqa-word+count-1433328323621-1-1-SUCCEEDED-default-1433328302419.jhist: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741856

/mr-history/done/2015/06/03/000000/job_1433328045348_0001-1433328283077-ambari%2Dqa-word+count-1433328323621-1-1-SUCCEEDED-default-1433328302419.jhist: MISSING 1 blocks of total size 33669 B..
/mr-history/done/2015/06/03/000000/job_1433328045348_0001_conf.xml: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741857_1033. Target Replicas is 3 but found 1 replica(s).
.
/mr-history/done/2015/06/05/000000/job_1433502507205_0001-1433503933474-ambari%2Dqa-PigLatin%3ApigSmoke.sh-1433503964156-1-0-SUCCEEDED-default-1433503952122.jhist: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741974_1152. Target Replicas is 3 but found 1 replica(s).
.
/mr-history/done/2015/06/05/000000/job_1433502507205_0001_conf.xml: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741975

/mr-history/done/2015/06/05/000000/job_1433502507205_0001_conf.xml: MISSING 1 blocks of total size 227572 B..
/tmp/id11ac4100_date410315: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741840

/tmp/id11ac4100_date410315: MISSING 1 blocks of total size 1393 B..
/user/ambari-qa/mapredsmokeinput: Under replicated BP-108620518-172.17.0.65-1433327686475:blk_1073741847_1023. Target Replicas is 3 but found 1 replica(s).
..
/user/ambari-qa/mapredsmokeoutput/part-r-00000: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741854

/user/ambari-qa/mapredsmokeoutput/part-r-00000: MISSING 1 blocks of total size 1475 B..
/user/ambari-qa/passwd: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741977

/user/ambari-qa/passwd: MISSING 1 blocks of total size 1521 B...
/user/ambari-qa/pigsmoke.out/part-v000-o000-r-00000: CORRUPT blockpool BP-108620518-172.17.0.65-1433327686475 block blk_1073741987

/user/ambari-qa/pigsmoke.out/part-v000-o000-r-00000: MISSING 1 blocks of total size 207 B.Status: CORRUPT
Total size: 414441608 B
Total dirs: 8591
Total files: 35
Total symlinks: 0
Total blocks (validated): 31 (avg. block size 13369084 B)

---

CORRUPT FILES: 16
MISSING BLOCKS: 16
MISSING SIZE: 141860175 B
CORRUPT BLOCKS: 16

---

Minimally replicated blocks: 15 (48.387096 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 15 (48.387096 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 0.48387095
Corrupt blocks: 16
Missing replicas: 30 (32.258064 %)
Number of data-nodes: 1
Number of racks: 1
FSCK ended at Mon Jun 08 11:33:06 EDT 2015 in 605 milliseconds

The filesystem under path '/' is CORRUPT
-bash-4.1#

Hbase Client install failed with OSError: [Errno 20] Not a directory: '/etc/resolv.conf/hadoop'

I have setup successfully a Single-node cluster using sequenceiq/ambari v1.7.0 by following instructions on https://registry.hub.docker.com/u/sequenceiq/ambari/.

But when adding Hbase service through Ambari web console, Hbase Client install failed with the message :

OSError: [Errno 20] Not a directory: '/etc/resolv.conf/hadoop'

stderr: Traceback (most recent call last): File “/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HBASE/package/scripts/hbaseclient.py”, line 43, in HbaseClient().execute() File “/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py”, line 123, in execute method(env) File “/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HBASE/package/scripts/hbase_client.py”, line 30, in install self.configure(env) File “/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HBASE/package/scripts/hbase_client.py”, line 36, in configure hbase(name='client') File “/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HBASE/package/scripts/hbase.py”, line 37, in hbase recursive = True File “/usr/lib/python2.6/site-packages/resource_management/core/base.py”, line 148, in _init self.env.run() File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 149, in run self.run_action(resource, action) File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 115, in run_action provider_action() File “/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py”, line 148, in action_create os.makedirs(path, self.resource.mode or 0755) File “/usr/lib64/python2.6/os.py”, line 150, in makedirs makedirs(head, mode) File “/usr/lib64/python2.6/os.py”, line 157, in makedirs mkdir(name, mode) OSError: [Errno 20] Not a directory: '/etc/resolv.conf/hadoop'

stdout: 2015-05-15 11:55:56,285 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x “” –retry 10 http://amb0.mycorp.kom:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment': …, 'notif': 'test -e /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']} 2015-05-15 11:55:56,303 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x “” –retry 10 http://amb0.mycorp.kom:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] due to not_if 2015-05-15 11:55:56,303 - Group['hadoop'] {'ignore_failures': False} 2015-05-15 11:55:56,305 - Modifying group hadoop 2015-05-15 11:55:56,775 - Group['users'] {'ignore_failures': False} 2015-05-15 11:55:56,776 - Modifying group users 2015-05-15 11:55:57,209 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-15 11:55:57,210 - Modifying user mapred 2015-05-15 11:55:57,231 - User['hbase'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-15 11:55:57,231 - Modifying user hbase 2015-05-15 11:55:57,560 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']} 2015-05-15 11:55:57,561 - Modifying user ambari-qa 2015-05-15 11:55:57,581 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-15 11:55:57,582 - Modifying user zookeeper 2015-05-15 11:55:57,602 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-15 11:55:57,603 - Modifying user hdfs 2015-05-15 11:55:57,623 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']} 2015-05-15 11:55:57,624 - Modifying user yarn 2015-05-15 11:55:57,645 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2015-05-15 11:55:57,648 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'} 2015-05-15 11:55:57,667 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] due to not_if 2015-05-15 11:55:57,668 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2015-05-15 11:55:57,670 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/etc/resolv.conf/hadoop/hbase 2>/dev/null'] {'not_if': 'test $(id -u hbase) -gt 1000'} 2015-05-15 11:55:58,202 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root', 'group': 'root', 'recursive': True} 2015-05-15 11:55:58,204 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'} 2015-05-15 11:55:58,223 - Skipping Link['/etc/hadoop/conf'] due to not_if 2015-05-15 11:55:58,255 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(…), 'owner': 'hdfs'} 2015-05-15 11:55:58,279 - Repository['HDP-2.2'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.2.4.2', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP', 'mirror_list': None} 2015-05-15 11:55:58,297 - File['/etc/yum.repos.d/HDP.repo'] {'content': Template('repo_suse_rhel.j2')} 2015-05-15 11:55:58,300 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None} 2015-05-15 11:55:58,308 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': Template('repo_suse_rhel.j2')} 2015-05-15 11:55:58,309 - Package['unzip'] {} 2015-05-15 11:55:59,018 - Skipping installing existent package unzip 2015-05-15 11:55:59,018 - Package['curl'] {} 2015-05-15 11:55:59,726 - Skipping installing existent package curl 2015-05-15 11:55:59,726 - Package['hdp-select'] {} 2015-05-15 11:56:00,435 - Skipping installing existent package hdp-select 2015-05-15 11:56:00,438 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x “” –retry 10 http://amb0.mycorp.kom:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] {'environment': …, 'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']} 2015-05-15 11:56:00,455 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x “” –retry 10 http://amb0.mycorp.kom:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] due to not_if 2015-05-15 11:56:00,457 - Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] {'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']} 2015-05-15 11:56:00,474 - Skipping Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] due to not_if 2015-05-15 11:56:00,738 - Package['hbase_2_2'] {} 2015-05-15 11:56:01,447 - Skipping installing existent package hbase2_2 2015-05-15 11:56:01,480 - Directory['/etc/hbase/conf'] {'owner': 'hbase', 'group': 'hadoop', 'recursive': True} 2015-05-15 11:56:01,482 - Changing owner for /etc/hbase/conf from 0 to hbase 2015-05-15 11:56:01,482 - Changing group for /etc/hbase/conf from 0 to hadoop 2015-05-15 11:56:01,483 - Directory['/etc/resolv.conf/hadoop/hbase'] {'owner': 'hbase', 'recursive': True} 2015-05-15 11:56:01,483 - Creating directory Directory['/etc/resolv.conf/hadoop/hbase']

docker version Client version: 1.5.0 Client API version: 1.17 Go version (client): go1.3.3 Git commit (client): a8a31ef/1.5.0 OS/Arch (client): linux/amd64 Server version: 1.5.0 Server API version: 1.17 Go version (server): go1.3.3 Git commit (server): a8a31ef/1.5.0

mutli-node cluster builds fail because of missing nagios dependency

The 1.7.0 version of the Dockerfile sets the script
/tmp/download-warmup-deps.sh
to be executable, but never runs the script. Because the script isn't run, nagios is not available
to be installed when building multi-node clusters using default blueprints
multi-node-hdfs-yarn
hdp-multinode-default
and these cluster builds fail.

Port 53 is often not available

Hi

I am new to this, but I have noticed that amb-start-cluster tries to publish port 53 to the host.
Often this fails because dnsmasq is often enabled with libvirt.

Should there be a change to the script so that it would more likely work out of the box and not conflict with popular services?

Fix Path

One of the downloaded packages in [docker-ambari/ambari-server/download-warmup-deps.sh] changes the path.

Cannot install

Hi

I have been following this blog post http://blog.sequenceiq.com/blog/2014/09/05/apache-ambari-1-7-0-ea/ trying to install docker with ambari.

I am running the following commands:

docker pull sequenceiq/ambari:1.7.0-ea
curl -Lo .amb j.mp/docker-ambari-170ea && . .amb
amb-deploy-cluster 4

I am using Docker version Docker version 1.2.0, build fa7b24f, Ubuntu Desktop 12.04 x64

docker ps shows this:

The error message that I get is installation failed

If I go to the ipaddress of amb0:8080 this is what I see:

---

Here are the logs:
amb0
Executing scripts from /usr/local/init
Execute: /usr/local/init/00_init.sh
Basic init has been executed
==> Starting Serf agent...
==> Starting Serf agent RPC...
==> Serf agent running!
Node name: 'amb0.mycorp.kom'
Bind addr: '172.17.0.14:7946'
RPC addr: '0.0.0.0:7373'
Encrypted: false
Snapshot: false
Profile: lan

==> Log data will now stream in as it occurs:

2014/09/30 09:51:23 [INFO] agent: Serf agent starting
2014/09/30 09:51:23 [INFO] serf: EventMemberJoin: amb0.mycorp.kom 172.17.0.14
2014/09/30 09:51:23 [INFO] serf: EventMemberJoin: amb1.mycorp.kom 172.17.0.15
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb2.mycorp.kom 172.17.0.16
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb3.mycorp.kom 172.17.0.17
2014/09/30 09:51:25 [INFO] agent: Received event: member-join
2014/09/30 09:51:26 [WARN] agent: Script '/usr/local/serf/handlers/ambari-bootstrap' slow, execution exceeding 1s
2014/09/30 09:51:47 [INFO] agent.ipc: Accepted client: 127.0.0.1:35540

lee@lee-W150HNM-W170HN:~$

amb1
lee@lee-W150HNM-W170HN:~$ docker logs amb1
Executing scripts from /usr/local/init
Execute: /usr/local/init/00_init.sh
Basic init has been executed
==> Starting Serf agent...
==> Starting Serf agent RPC...
==> Serf agent running!
Node name: 'amb1.mycorp.kom'
Bind addr: '172.17.0.15:7946'
RPC addr: '0.0.0.0:7373'
Encrypted: false
Snapshot: false
Profile: lan

==> Log data will now stream in as it occurs:

2014/09/30 09:51:23 [INFO] agent: Serf agent starting
2014/09/30 09:51:23 [INFO] serf: EventMemberJoin: amb1.mycorp.kom 172.17.0.15
2014/09/30 09:51:23 [INFO] agent: Joining cluster...(replay: false)
2014/09/30 09:51:23 [INFO] agent: joining: [172.17.0.14] replay: false
2014/09/30 09:51:23 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
2014/09/30 09:51:23 [INFO] serf: EventMemberJoin: amb0.mycorp.kom 172.17.0.14
2014/09/30 09:51:23 [INFO] agent: joined: 1 nodes
2014/09/30 09:51:23 [INFO] agent: Join completed. Synced with 1 initial agents
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb1.mycorp.kom
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb2.mycorp.kom 172.17.0.16
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb1.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb1.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb2.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb2.mycorp.kom
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb3.mycorp.kom 172.17.0.17
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb2.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:25 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:25 [INFO] agent: Received event: member-join
2014/09/30 09:51:26 [WARN] agent: Script '/usr/local/serf/handlers/ambari-bootstrap' slow, execution exceeding 1s
2014/09/30 09:51:27 [DEBUG] agent: Event 'member-join' script output: [DEBUG] sed -i.bak /^hostname/ s/.*/hostname=amb0.mycorp.kom/ /etc/ambari-agent/conf/ambari-agent.ini

[DEBUG] ambari-agent start
Verifying Python version compatibility...
Using python /usr/bin/python2.6
Checking for previously running Ambari Agent...
Starting ambari-agent
Verifying ambari-agent process status...
tput: No value for $TERM and no -T specified
Ambari Agent successfully started
tput: No value for $TERM and no -T specified
Agent PID at: /var/run/ambari-agent/ambari-agent.pid
Agent out at: /var/log/ambari-agent/ambari-agent.out
Agent log at: /var/log/ambari-agent/ambari-agent.log
2014/09/30 09:51:27 [INFO] agent.ipc: Accepted client: 127.0.0.1:35457
2014/09/30 09:51:27 [DEBUG] agent: Event 'member-join' script output:
Starting dnsmasq: [ OK ]

2014/09/30 09:51:27 [DEBUG] agent: Event 'member-join' script output:

SERF_EVENT=member-join
SERF_HOME=/usr/local/serf
SERF_JOIN_IP=172.17.0.14
SERF_SELF_NAME=amb1.mycorp.kom
SERF_SELF_ROLE=

SERF_TAG_AMBARI_AGENT=true

amb1.mycorp.kom 172.17.0.15 ambari-agent=true
amb0.mycorp.kom 172.17.0.14 ambari-agent=true,ambari-server=true
amb2.mycorp.kom 172.17.0.16 ambari-agent=true
amb3.mycorp.kom 172.17.0.17 ambari-agent=true
=== DEBUG-BODY ===
amb1.mycorp.kom^I172.17.0.15^I^Iambari-agent=true$
amb0.mycorp.kom^I172.17.0.14^I^Iambari-agent=true,ambari-server=true$
amb2.mycorp.kom^I172.17.0.16^I^Iambari-agent=true$
amb3.mycorp.kom^I172.17.0.17^I^Iambari-agent=true$
2014/09/30 09:51:28 [DEBUG] agent: Event 'member-join' script output: + LOGFILE=/tmp/ssh.log
++ date +%Y%m%d-%H%M

• EVENT_LOG=/tmp/serf-ssh-20140930-0951.log
• cat
• [[ member-join == \m\e\m\b\e\r-\j\o\i\n ]]
• grep amb1.mycorp.kom /tmp/serf-ssh-20140930-0951.log
  amb1.mycorp.kom 172.17.0.15 ambari-agent=true
• /etc/init.d/sshd start
  Generating SSH1 RSA host key: [ OK ]
  Starting sshd: [ OK ]
• '[' -n '' ']'
  2014/09/30 09:52:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 09:52:19 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:42282
  2014/09/30 09:52:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.17:7946
  2014/09/30 09:52:44 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:41876
  2014/09/30 09:53:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 09:53:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:38311
  2014/09/30 09:53:14 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:41989
  2014/09/30 09:53:19 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:42560
  2014/09/30 09:53:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:53:44 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:42111
  2014/09/30 09:54:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.17:7946
  2014/09/30 09:54:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:38531
  2014/09/30 09:54:14 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:42209
  2014/09/30 09:54:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.17:7946
  2014/09/30 09:54:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:38632
  2014/09/30 09:54:44 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:42319
  2014/09/30 09:55:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:55:14 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:42437
  2014/09/30 09:55:19 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:42996
  2014/09/30 09:55:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:56:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 09:56:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:56:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:39126
  2014/09/30 09:57:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:57:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:39230
  2014/09/30 09:57:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 09:58:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 09:58:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 09:58:44 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:43215
  2014/09/30 09:59:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.17:7946
  2014/09/30 09:59:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:59:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:39731
  2014/09/30 09:59:44 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:43416
  2014/09/30 09:59:49 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:43969
  2014/09/30 10:00:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 10:00:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:39835
  2014/09/30 10:00:14 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:43512
  2014/09/30 10:00:19 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:44065
  2014/09/30 10:00:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.17:7946

amb2
lee@lee-W150HNM-W170HN:~$ docker logs amb2
Executing scripts from /usr/local/init
Execute: /usr/local/init/00_init.sh
Basic init has been executed
==> Starting Serf agent...
==> Starting Serf agent RPC...
==> Serf agent running!
Node name: 'amb2.mycorp.kom'
Bind addr: '172.17.0.16:7946'
RPC addr: '0.0.0.0:7373'
Encrypted: false
Snapshot: false
Profile: lan

==> Log data will now stream in as it occurs:

2014/09/30 09:51:24 [INFO] agent: Serf agent starting
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb2.mycorp.kom 172.17.0.16
2014/09/30 09:51:24 [INFO] agent: Joining cluster...(replay: false)
2014/09/30 09:51:24 [INFO] agent: joining: [172.17.0.14] replay: false
2014/09/30 09:51:24 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb1.mycorp.kom 172.17.0.15
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb0.mycorp.kom 172.17.0.14
2014/09/30 09:51:24 [INFO] agent: joined: 1 nodes
2014/09/30 09:51:24 [INFO] agent: Join completed. Synced with 1 initial agents
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb1.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb1.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb2.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb2.mycorp.kom
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb3.mycorp.kom 172.17.0.17
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb2.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:25 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:25 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:25 [INFO] agent: Received event: member-join
2014/09/30 09:51:26 [WARN] agent: Script '/usr/local/serf/handlers/ambari-bootstrap' slow, execution exceeding 1s
2014/09/30 09:51:27 [DEBUG] agent: Event 'member-join' script output: [DEBUG] sed -i.bak /^hostname/ s/.*/hostname=amb0.mycorp.kom/ /etc/ambari-agent/conf/ambari-agent.ini

[DEBUG] ambari-agent start
Verifying Python version compatibility...
Using python /usr/bin/python2.6
Checking for previously running Ambari Agent...
Starting ambari-agent
Verifying ambari-agent process status...
tput: No value for $TERM and no -T specified
Ambari Agent successfully started
tput: No value for $TERM and no -T specified
Agent PID at: /var/run/ambari-agent/ambari-agent.pid
Agent out at: /var/log/ambari-agent/ambari-agent.out
Agent log at: /var/log/ambari-agent/ambari-agent.log
2014/09/30 09:51:27 [INFO] agent.ipc: Accepted client: 127.0.0.1:35456
2014/09/30 09:51:27 [DEBUG] agent: Event 'member-join' script output:
Starting dnsmasq: [ OK ]

2014/09/30 09:51:27 [DEBUG] agent: Event 'member-join' script output:

SERF_EVENT=member-join
SERF_HOME=/usr/local/serf
SERF_JOIN_IP=172.17.0.14
SERF_SELF_NAME=amb2.mycorp.kom
SERF_SELF_ROLE=

SERF_TAG_AMBARI_AGENT=true

amb2.mycorp.kom 172.17.0.16 ambari-agent=true
amb1.mycorp.kom 172.17.0.15 ambari-agent=true
amb0.mycorp.kom 172.17.0.14 ambari-agent=true,ambari-server=true
amb3.mycorp.kom 172.17.0.17 ambari-agent=true
=== DEBUG-BODY ===
amb2.mycorp.kom^I172.17.0.16^I^Iambari-agent=true$
amb1.mycorp.kom^I172.17.0.15^I^Iambari-agent=true$
amb0.mycorp.kom^I172.17.0.14^I^Iambari-agent=true,ambari-server=true$
amb3.mycorp.kom^I172.17.0.17^I^Iambari-agent=true$
2014/09/30 09:51:28 [DEBUG] agent: Event 'member-join' script output: + LOGFILE=/tmp/ssh.log
++ date +%Y%m%d-%H%M

• EVENT_LOG=/tmp/serf-ssh-20140930-0951.log
• cat
• [[ member-join == \m\e\m\b\e\r-\j\o\i\n ]]
• grep amb2.mycorp.kom /tmp/serf-ssh-20140930-0951.log
  amb2.mycorp.kom 172.17.0.16 ambari-agent=true
• /etc/init.d/sshd start
  Generating SSH1 RSA host key: [ OK ]
  Starting sshd: [ OK ]
• '[' -n '' ']'
  2014/09/30 09:52:14 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 09:52:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:42132
  2014/09/30 09:52:44 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:53:14 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:53:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:48453
  2014/09/30 09:53:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:42386
  2014/09/30 09:53:44 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:54:14 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:54:19 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:35902
  2014/09/30 09:54:44 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:54:49 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:36010
  2014/09/30 09:55:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:48776
  2014/09/30 09:55:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:42709
  2014/09/30 09:55:14 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:55:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:48933
  2014/09/30 09:55:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:42866
  2014/09/30 09:55:44 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.17:7946
  2014/09/30 09:55:49 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:36281
  2014/09/30 09:56:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:42977
  2014/09/30 09:56:14 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 09:56:19 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:36385
  2014/09/30 09:56:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:49157
  2014/09/30 09:56:44 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.17:7946
  2014/09/30 09:56:49 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:36502
  2014/09/30 09:57:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:49261
  2014/09/30 09:57:14 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.17:7946
  2014/09/30 09:57:19 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:36598
  2014/09/30 09:57:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:43290
  2014/09/30 09:57:44 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.17:7946
  2014/09/30 09:58:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:43394
  2014/09/30 09:58:14 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.17:7946
  2014/09/30 09:58:44 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:59:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.17:43598
  2014/09/30 09:59:14 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 09:59:19 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:36997
  2014/09/30 09:59:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:49762
  2014/09/30 09:59:44 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 10:00:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:49866
  2014/09/30 10:00:14 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 10:00:44 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.17:7946
  2014/09/30 10:00:49 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:37302
  2014/09/30 10:01:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:50066

amb3
^[[Alee@lee-W150HNM-W170HN:~$ docker logs amb3
Executing scripts from /usr/local/init
Execute: /usr/local/init/00_init.sh
Basic init has been executed
==> Starting Serf agent...
==> Starting Serf agent RPC...
==> Serf agent running!
Node name: 'amb3.mycorp.kom'
Bind addr: '172.17.0.17:7946'
RPC addr: '0.0.0.0:7373'
Encrypted: false
Snapshot: false
Profile: lan

==> Log data will now stream in as it occurs:

2014/09/30 09:51:24 [INFO] agent: Serf agent starting
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb3.mycorp.kom 172.17.0.17
2014/09/30 09:51:24 [INFO] agent: Joining cluster...(replay: false)
2014/09/30 09:51:24 [INFO] agent: joining: [172.17.0.14] replay: false
2014/09/30 09:51:24 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb1.mycorp.kom 172.17.0.15
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb2.mycorp.kom 172.17.0.16
2014/09/30 09:51:24 [INFO] serf: EventMemberJoin: amb0.mycorp.kom 172.17.0.14
2014/09/30 09:51:24 [INFO] agent: joined: 1 nodes
2014/09/30 09:51:24 [INFO] agent: Join completed. Synced with 1 initial agents
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb2.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb2.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:24 [DEBUG] serf: messageJoinType: amb3.mycorp.kom
2014/09/30 09:51:25 [INFO] agent: Received event: member-join
2014/09/30 09:51:26 [WARN] agent: Script '/usr/local/serf/handlers/ambari-bootstrap' slow, execution exceeding 1s
2014/09/30 09:51:27 [DEBUG] agent: Event 'member-join' script output: [DEBUG] sed -i.bak /^hostname/ s/.*/hostname=amb0.mycorp.kom/ /etc/ambari-agent/conf/ambari-agent.ini

[DEBUG] ambari-agent start
Verifying Python version compatibility...
Using python /usr/bin/python2.6
Checking for previously running Ambari Agent...
Starting ambari-agent
Verifying ambari-agent process status...
tput: No value for $TERM and no -T specified
Ambari Agent successfully started
tput: No value for $TERM and no -T specified
Agent PID at: /var/run/ambari-agent/ambari-agent.pid
Agent out at: /var/log/ambari-agent/ambari-agent.out
Agent log at: /var/log/ambari-agent/ambari-agent.log
2014/09/30 09:51:27 [INFO] agent.ipc: Accepted client: 127.0.0.1:35455
2014/09/30 09:51:27 [DEBUG] agent: Event 'member-join' script output:
Starting dnsmasq: [ OK ]

2014/09/30 09:51:27 [DEBUG] agent: Event 'member-join' script output:

SERF_EVENT=member-join
SERF_HOME=/usr/local/serf
SERF_JOIN_IP=172.17.0.14
SERF_SELF_NAME=amb3.mycorp.kom
SERF_SELF_ROLE=

SERF_TAG_AMBARI_AGENT=true

amb3.mycorp.kom 172.17.0.17 ambari-agent=true
amb1.mycorp.kom 172.17.0.15 ambari-agent=true
amb2.mycorp.kom 172.17.0.16 ambari-agent=true
amb0.mycorp.kom 172.17.0.14 ambari-agent=true,ambari-server=true
=== DEBUG-BODY ===
amb3.mycorp.kom^I172.17.0.17^I^Iambari-agent=true$
amb1.mycorp.kom^I172.17.0.15^I^Iambari-agent=true$
amb2.mycorp.kom^I172.17.0.16^I^Iambari-agent=true$
amb0.mycorp.kom^I172.17.0.14^I^Iambari-agent=true,ambari-server=true$
2014/09/30 09:51:27 [DEBUG] agent: Event 'member-join' script output: + LOGFILE=/tmp/ssh.log
++ date +%Y%m%d-%H%M

• EVENT_LOG=/tmp/serf-ssh-20140930-0951.log
• cat
• [[ member-join == \m\e\m\b\e\r-\j\o\i\n ]]
• grep amb3.mycorp.kom /tmp/serf-ssh-20140930-0951.log
  amb3.mycorp.kom 172.17.0.17 ambari-agent=true
• /etc/init.d/sshd start
  Generating SSH1 RSA host key: [ OK ]
  Starting sshd: [ OK ]
• '[' -n '' ']'
  2014/09/30 09:52:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 09:52:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:53854
  2014/09/30 09:52:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:52:49 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:44271
  2014/09/30 09:53:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:53:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:53:49 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:44496
  2014/09/30 09:54:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:54217
  2014/09/30 09:54:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:54:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:54318
  2014/09/30 09:54:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:55:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:55:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:55:44 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:58179
  2014/09/30 09:56:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:56:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:56:44 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:58403
  2014/09/30 09:57:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 09:57:14 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:58499
  2014/09/30 09:57:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:57:44 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:58603
  2014/09/30 09:57:49 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:45396
  2014/09/30 09:58:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:58:14 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:58699
  2014/09/30 09:58:19 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:45492
  2014/09/30 09:58:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 09:58:49 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.14:45600
  2014/09/30 09:59:08 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:55320
  2014/09/30 09:59:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.16:7946
  2014/09/30 09:59:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 10:00:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.15:7946
  2014/09/30 10:00:38 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.15:55617
  2014/09/30 10:00:38 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 10:00:44 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:59208
  2014/09/30 10:01:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.17.0.14:7946
  2014/09/30 10:01:14 [DEBUG] memberlist: Responding to push/pull sync with: 172.17.0.16:59304

SSH

What's the login and password for SSH access to the docker machines in the Ambari cluster?

sequenceiq/ambari:1.7.0 connect client with server

We are running sequenceiq/ambari:1.7.0 in different nodes. The following commands works on one node. But the communication between two nodes(two instance running on was) is not working

server node command

docker run -d --dns 127.0.0.1 -p 8080:8080 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb0 -h amb0.mycorp.kom sequenceiq/ambari:1.7.0 --tag ambari-server=true

client node command

docker run -d -e SERF_JOIN_IP=54.208.22.28 -p 7373:7373 -p 7346:7346 --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb3 -h amb3.mycorp.kom sequenceiq/ambari:1.7.0 --log-level debug

The error message on server node is

==> Starting Serf agent...
==> Starting Serf agent RPC...
==> Serf agent running!
         Node name: 'amb0.mycorp.kom'
         Bind addr: '172.17.0.4:7946'
          RPC addr: '0.0.0.0:7373'
         Encrypted: false
          Snapshot: false
           Profile: lan

==> Log data will now stream in as it occurs:

    2015/10/08 15:19:48 [INFO] agent: Serf agent starting
    2015/10/08 15:19:48 [INFO] serf: EventMemberJoin: amb0.mycorp.kom 172.17.0.4
    2015/10/08 15:19:49 [INFO] agent: Received event: member-join
    2015/10/08 15:19:50 [WARN] agent: Script '/usr/local/serf/handlers/ambari-bootstrap' slow, execution exceeding 1s
    2015/10/08 15:20:54 [INFO] serf: EventMemberJoin: amb3.mycorp.kom 172.17.0.5
    2015/10/08 15:21:01 [INFO] serf: EventMemberFailed: amb3.mycorp.kom 172.17.0.5
    2015/10/08 15:21:18 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:21:51 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:22:24 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:22:57 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:23:30 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:24:03 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:24:36 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:25:09 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:25:42 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:26:15 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:26:48 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:27:21 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:27:54 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946
    2015/10/08 15:28:27 [INFO] serf: attempting reconnect to amb3.mycorp.kom 172.17.0.5:7946

The error message on client side

Generating SSH1 RSA host key: [  OK  ]
Starting sshd: [  OK  ]
+ '[' -n '' ']'
    2015/10/08 15:21:01 [INFO] serf: EventMemberFailed: amb0.mycorp.kom 172.17.0.4
    2015/10/08 15:21:02 [INFO] agent: Received event: member-failed
    2015/10/08 15:21:02 [INFO] agent.ipc: Accepted client: 127.0.0.1:45370
    2015/10/08 15:21:02 [DEBUG] agent: Event 'member-failed' script output: Shutting down dnsmasq: [  OK  ]
Starting dnsmasq: [  OK  ]
    2015/10/08 15:21:02 [DEBUG] agent: Event 'member-failed' script output: 
==================
SERF_EVENT=member-failed
SERF_HOME=/usr/local/serf
SERF_JOIN_IP=54.208.22.28
SERF_SELF_NAME=amb3.mycorp.kom
SERF_SELF_ROLE=
SERF_TAG_AMBARI_AGENT=true
==================
amb0.mycorp.kom 172.17.0.4      ambari-agent=true,ambari-server=true
=== DEBUG-BODY ===
amb0.mycorp.kom^I172.17.0.4^I^Iambari-agent=true,ambari-server=true$
    2015/10/08 15:21:24 [INFO] serf: attempting reconnect to amb0.mycorp.kom 172.17.0.4:7946
    2015/10/08 15:21:57 [INFO] serf: attempting reconnect to amb0.mycorp.kom 172.17.0.4:7946
    2015/10/08 15:22:30 [INFO] serf: attempting reconnect to amb0.mycorp.kom 172.17.0.4:7946
    2015/10/08 15:23:03 [INFO] serf: attempting reconnect to amb0.mycorp.kom 172.17.0.4:7946
    2015/10/08 15:23:36 [INFO] serf: attempting reconnect to amb0.mycorp.kom 172.17.0.4:7946
    2015/10/08 15:24:09 [INFO] serf: attempting reconnect to amb0.mycorp.kom 172.17.0.4:7946

Something wrong with the ip configuration?

Documentation reports HDP 2.2 is used, when it is not

Leveraging branch 1.7.0, documentation reports that Ambari 1.7.0 is leveraged, which it is. However, documentation also mentions that HDP 2.2 is leveraged, it is not:

Note: Attempting to use the single line execution noted under documentation section "Multinode Hadoop Cluster" which references the following command "curl -Lo .amb j.mp/docker-ambari && . .amb && amb-deploy-cluster"

Note2: Yes - I am referencing the right docker-ambari script which calls out ambari:1.7.0 docker image and not the default one seen in the URL (which calls out 1.6)

When the cluster comes up, a user can go into the Ambari UI Admin section and determine full stack deployment versions. In the ambari UI, it clearly states that HDP 2.1 is present. Examination of the Dockerfile that is leveraged shows several issues:

1. hdp.repo = This file is incorrect for HDP 2.2 (Ref: http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDP2/HDP-2-trunk/bk_installing_manually_book/content/upgrade-2X-repo.html)
2. Addition of hdp.repo to yum repository during docker image build is commented out

This is misleading to users, whom expect improvements done in HDP 2.2 to exist. Unfortunately they do not, since HDP 2.1 is what is leveraged.

However, when changing hdp.repo to full-fill the proper URLs, and uncommenting out lines 21 and 22 do not seem to help. (Ref: https://github.com/sequenceiq/docker-ambari/blob/1.7.0/ambari-server/Dockerfile)

Unfortunately I'm not quite sure how to get all this working at the moment, hence the issue being opened. Any response is appreciated!

Docker swarm support?

Hi,

Do you plan to support creating clusters running on Docker Swarm? I tried the current version but the network doesn't seem to work with multiple nodes in a swarm cluster. I think it might be possible to add support for network overlays in the ambari-functions script.

using host network not working

both 1.6.0 and 1.7.0-ea fail to start up with --net=host. ambari-server will fail with exit code 255 on 1.7.0-ea and exit code 1 on 1.6.0. I don't know if this is supported yet, with docker network it works as expected.

Cannot connect to Ambari http://172.17.0.2:8080/api/v1/

I keep getting this error this couple of days, which previously I have no issues on. Looks like the ambari server was not started automatically. Was there any changes to the Dockerfile?

Thanks in advance.

Cannot pulling sequenceiq/ambari

Hi Guys,
i got such error When i'm trying to docker pull sequenceiq/ambari

After install, container server is not able to connect - Mac

This is a really neat project and I'm looking forward to seeing it running, unfortunately I've hit a snag testing it on Mavericks, and I cannot connect to the ambari server.

Here are the steps I am doing:

1. installing boot2docker and using the basic ubuntu tests to validate setup
2. using the single curl script to install the two docker containers for ambari
   a) I see the Ambari Shell in green, and in the top-right corner of my terminal an "Installing ..." sequence
   b) once the Installing gets to 100% it immediately dumps me back to my host terminal.
   c) I cannot connect to Ambari through the host or container IP at port 8080. When I try to connect, the socket waits (possibly indicating the container binding works, but is unable to connect to a service on the container, or the container service is failing?).... and times out to the host.

Also a few docker command outputs for your reference:

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

6e5fa0735dfd sequenceiq/ambari:latest "/usr/local/serf/bin 39 minutes ago Up 17 minutes 7373/tcp, 7946/tcp, 0.0.0.0:49153->8080/tcp ambari-singlenode

docker attach 6e5fa0735dfd

this hangs and never connects

HDP repository is not available and Ambari can't pull the RPMs (ubuntu 14.04/ docker(1.3.x and 1.4.0)/ ambari 1.7.0)

Under Ubuntu version 14.04, i'm getting error in installation: Failed in the ambari shell after reaching 4.5%.

I see in /var/logs/ambari-server/ambari-server.log there errors :

1:59,109 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (public-repo-1.hortonworks.com)
19:11:59,109 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json19:11:59,110 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (publ--More--ic-repo-1.hortonworks.com)
19:11:59,110 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
19:11:59,110 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (public-repo-1.hortonworks.com)
19:11:59,111 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json
19:11:59,111 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json (s3.amazonaws.com)
19:11:59,111 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
19:11:59,112 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (public-repo-1.hortonworks.com)
19:11:59,115 INFO [main] ActionDefinitionManager:124 - Added custom action definition for nagios_update_ignore
19:11:59,116 INFO [main] ActionDefinitionManager:124 - Added custom action definition for check_host
19:11:59,117 INFO [main] ActionDefinitionManager:124 - Added custom action definition for validate_configs
19:11:59,119 INFO [main] CertificateManager:69 - Initialization of root certificate
19:11:59,119 INFO [main] CertificateManager:71 - Certificate exists:false
19:11:59,119 INFO [main] CertificateManager:138 - Generation of server certificate
19:11:59,493 INFO [main] ShellCommandUtil:44 - Command openssl genrsa -des3 -passout pass:* -out /var/lib/ambari-server/keys/ca.key 4096 was finished with exit code: 0 - the operation was completely successfully.
19:11:59,513 INFO [main] ShellCommandUtil:44 - Command openssl req -passin pass:* -new -key /var/lib/ambari-server/keys/ca.key -out /var/lib/ambari-server/keys/ca.csr -batch was finished with exit code: 0 - the operation was completely successfully.
19:11:59,544 INFO [main] ShellCommandUtil:44 - Command openssl ca -create_serial -out /var/lib/ambari-server/keys/ca.crt -days 365 -keyfile /var/lib/ambari-server/keys/ca.key -key * -selfsign -extensions jdk7_ca -config /var/lib/ambari-server/keys/ca.config -batch -infiles /var/lib/ambari-server/keys/ca.csr was finished with exit code: 0 - the operation was completely successfully.
19:11:59,559 INFO [main] ShellCommandUtil:44 - Command openssl pkcs12 -export -in /var/lib/ambari-server/keys/ca.crt -inkey /var/lib/ambari-server/keys/ca.key -certfile /var/lib/ambari-server/keys/ca.crt -out /var/lib/ambari-server/keys/keystore.p12 -password pass:* -passin pass:****
was finished with exit code: 0 - the operation was completely successfully.
19:11:59,858 INFO [main] Configuration:487 - Credential provider creation failed. Reason: Master key initialization failed.
19:12:00,169 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
19:12:00,171 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (public-repo-1.hortonworks.com)
19:12:00,172 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
19:12:00,172 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (public-repo-1.hortonworks.com)
19:12:00,173 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
19:12:00,173 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (public-repo-1.hortonworks.com)
19:12:00,173 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json
19:12:00,174 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json (s3.amazonaws.com)
19:12:00,174 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
19:12:00,174 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (public-repo-1.hortonworks.com)
19:12:00,177 WARN [main] ActionDefinitionManager:117 - Ignoring action definition as a different definition by that name already exists. ActionDefinition: actionName: nagios_update_ignore actionType: SYSTEM inputs: [nagios_ignore] description: Used to create an alert blackout targetService: NAGIOS targetComponent: NAGIOS_SERVER defaultTimeout: 60 targetType: ANY
19:12:00,177 WARN [main] ActionDefinitionManager:117 - Ignoring action definition as a different definition by that name already exists. ActionDefinition: actionName: check_host actionType: SYSTEM inputs: description: General check for host targetService: targetComponent: defaultTimeout: 60 targetType: ANY
19:12:00,177 WARN [main] ActionDefinitionManager:117 - Ignoring action definition as a different definition by that name already exists. ActionDefinition: actionName: validate_configs actionType: SYSTEM inputs: description: Validate if provided service config can be applied to specified hosts targetService: targetComponent: defaultTimeout: 60 targetType: ALL

can't work correctly in cloudbreak

Hi,

When we created the cluster with cloudbreak, the ambari can't work correclty. Could you give some hints on how to fix it? Do we need to configure something for docker engine or OS?

docker inspect:

[
{
    "Id": "df8926b8290e80645b01fe92c434ef78e59419fd35080c6f995e3b4aaef86947",
    "Created": "2016-01-21T08:14:57.258655233Z",
    "Path": "/start-server",
    "Args": [],
    "State": {
        "Status": "running",
        "Running": true,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 3141,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2016-01-21T08:15:03.51735823Z",
        "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "70f5dbe71f4a2baaffd67f1d44b562b317cf272382a1a88740f98a1f57c3c15a",
    "ResolvConfPath": "/var/lib/docker/containers/df8926b8290e80645b01fe92c434ef78e59419fd35080c6f995e3b4aaef86947/resolv.conf",
    "HostnamePath": "/var/lib/docker/containers/df8926b8290e80645b01fe92c434ef78e59419fd35080c6f995e3b4aaef86947/hostname",
    "HostsPath": "/var/lib/docker/containers/df8926b8290e80645b01fe92c434ef78e59419fd35080c6f995e3b4aaef86947/hosts",
    "LogPath": "/var/lib/docker/containers/df8926b8290e80645b01fe92c434ef78e59419fd35080c6f995e3b4aaef86947/df8926b8290e80645b01fe92c434ef78e59419fd35080c6f995e3b4aaef86947-json.log",
    "Name": "/ambari-server",
    "RestartCount": 0,
    "Driver": "devicemapper",
    "ExecDriver": "native-0.2",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": [
            "/hadoopfs/fs1/logs:/var/log:rw",
            "/etc/krb5.conf:/etc/krb5.conf:rw"
        ],
        "ContainerIDFile": "",
        "LxcConf": null,
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "KernelMemory": 0,
        "CpuShares": 0,
        "CpuPeriod": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "CpuQuota": 0,
        "BlkioWeight": 0,
        "OomKillDisable": false,
        "MemorySwappiness": null,
        "Privileged": true,
        "PortBindings": {
            "8080/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "8080"
                }
            ]
        },
        "Links": null,
        "PublishAllPorts": false,
        "Dns": null,
        "DnsOptions": null,
        "DnsSearch": null,
        "ExtraHosts": null,
        "VolumesFrom": null,
        "Devices": null,
        "NetworkMode": "host",
        "IpcMode": "",
        "PidMode": "",
        "UTSMode": "",
        "CapAdd": null,
        "CapDrop": null,
        "GroupAdd": null,
        "RestartPolicy": {
            "Name": "always",
            "MaximumRetryCount": 0
        },
        "SecurityOpt": null,
        "ReadonlyRootfs": false,
        "Ulimits": null,
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "CgroupParent": "",
        "ConsoleSize": [
            0,
            0
        ],
        "VolumeDriver": ""
    },
    "GraphDriver": {
        "Name": "devicemapper",
        "Data": {
            "DeviceId": "271",
            "DeviceName": "docker-253:0-17855111-df8926b8290e80645b01fe92c434ef78e59419fd35080c6f995e3b4aaef86947",
            "DeviceSize": "107374182400"
        }
    },
    "Mounts": [
        {
            "Source": "/hadoopfs/fs1/logs",
            "Destination": "/var/log",
            "Mode": "rw",
            "RW": true
        },
        {
            "Source": "/etc/krb5.conf",
            "Destination": "/etc/krb5.conf",
            "Mode": "rw",
            "RW": true
        }
    ],
    "Config": {
        "Hostname": "***",
        "Domainname": "abc.com",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": {
            "8080/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "POSTGRES_DB=localhost",
            "CLOUD_PLATFORM=***",
            "SERVICE_NAME=ambari-8080",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/jdk64/jdk1.7.0_67/bin:/usr/local/maven/bin",
            "JAVA_HOME=/usr/jdk64/jdk1.7.0_67",
            "AMBARI_JVM_ARGS=-XX:MaxPermSize=512m",
            "PLUGIN_PATH=/plugins",
            "HADOOP_CLASSPATH=/usr/lib/hadoop/lib/*",
            "M2_HOME=/usr/local/maven",
            "PS1=[\\u@docker-ambari \\W]# "
        ],
        "Cmd": [
            "/start-server"
        ],
        "Image": "sequenceiq/ambari:2.1.2-v8",
        "Volumes": {
            "/var/log": {}
        },
        "WorkingDir": "/tmp",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": {
            "com.docker.swarm.constraints": "[\"node==wdc-vhadp-pub1-dhcp44.eng.vmware.com\"]",
            "com.docker.swarm.id": "d756b74215f5ee2df3f15bb8fcf71db43f19ca41c07169f8fb309a8220c6405a"
        }
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "1b4d3e7601208df75c8306abc74e6a476d3f63c7d2510b7ae97d40e68e871cfa",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {},
        "SandboxKey": "/var/run/docker/netns/default",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "host": {
                "EndpointID": "b577381157d2ef3375a48ebbdd365561bb8e3b79b6fd5c9b493dca27db67b622",
                "Gateway": "",
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": ""
            }
        }
    }
}
]

docker logs:

sing python  /usr/bin/python2.6
Starting ambari-server
WARNING: The hostname was not found in the reverse DNS lookup. This may result in incorrect behavior. Please check the DNS setup and fix the issue.
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.........
ERROR: Exiting with exit code -1.
REASON: Ambari Server java process died with exitcode 1. Check /var/log/ambari-server/ambari-server.out for more information.
Using python  /usr/bin/python2.6
Starting ambari-server
WARNING: The hostname was not found in the reverse DNS lookup. This may result in incorrect behavior. Please check the DNS setup and fix the issue.
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start...................
Ambari Server 'start' completed successfully.
Using python  /usr/bin/python2.6
Starting ambari-server
ERROR: Exiting with exit code 1.
REASON: Ambari Server is already running.

HDP 2.3 Repo URL

I installed sequence/ambari docker successfully and started Ambari UI.

But following repo's seems to be unreachable ? Can some one help with correct URL's

HDP 2.3
http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.0.0

HDP-UTILS-1.1.0.20
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6

hdp-2.1.2.1 repository not available anymore. Please upgrade to 2.1.5.0.

KylinOLAP docker image is not usable due this issue.

cannot view web UI after start ambari:1.7.0 container

I use ambari-functions to start container. -->the commnd is :amb-deploy-cluster 3

and get :
root@zqin:/home/zqin/Desktop/ambari-1.7.0# amb-deploy-cluster 3
starting an ambari cluster with: 3 nodes
[DEBUG] docker run -d --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb0 -h amb0.mycorp.kom sequenceiq/ambari:1.7.0 --tag ambari-server=true
WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.
e890f2b92a5e2216760e085905ee49cbbbfb1c73e22eb8f5c3d2ac652ba1abcc
[DEBUG] docker run -d -e SERF_JOIN_IP=172.17.0.2 --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb1 -h amb1.mycorp.kom sequenceiq/ambari:1.7.0 --log-level debug
WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.
25349b53564f908d357305e8355ae672dc422e89e996613246f95cac280946cb
[DEBUG] docker run -d -e SERF_JOIN_IP=172.17.0.2 --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb2 -h amb2.mycorp.kom sequenceiq/ambari:1.7.0 --log-level debug
WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.
c025eb09e1a44314e45cdb0002ec1a1ffa516b1fc78e392eab488c4d628974f6
[DEBUG] docker run -it --rm -e EXPECTED_HOST_COUNT=3 -e BLUEPRINT=multi-node-hdfs-yarn --link amb0:ambariserver --entrypoint /bin/sh sequenceiq/ambari:1.7.0 -c /tmp/install-cluster.sh
AMBARI_HOST=172.17.0.2
[DEBUG] waits for ambari server: 172.17.0.2 RUNNING ...
..............
[DEBUG] waits until 3 hosts connected to server ...
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 1
[DEBUG] connected hosts: 3
_ _ _ ____ _ _ _
/ \ _ __ ___ | |__ __ _ _ __ ()/ | | | ___ | || |
/ _ \ | ' _ \ | '_ \ / _ || '**|| |** \ | '_ \ / _ | || |
/ ___ \ | | | | | || |) || (| || | | | ) || | | || __/| || |
// _|| || |||./ __,||| |||_/ || || _|||||

Welcome to Ambari Shell. For command and param completion press TAB, for assistance type 'hint'.
ambari-shell>blueprint defaults
Default blueprints added
ambari-shell>cluster build --blueprint multi-node-hdfs-yarn
HOSTNAME STATE

---

amb2.mycorp.kom amb2.mycorp.kom
amb1.mycorp.kom amb1.mycorp.kom
amb0.mycorp.kom amb0.mycorp.kom

HOSTGROUP COMPONENT

---

master NAMENODE
master APP_TIMELINE_SERVER
master HISTORYSERVER
master ZOOKEEPER_SERVER Installation: COMPLETEroot@zqin:/home/zqin/Desktop/ambari-1.7.0# ^C
root@zqin:/home/zqin/Desktop/ambari-1.7.0#
master NAGIOS_SERVER
master HDFS_CLIENT
master MAPREDUCE2_CLIENT
master YARN_CLIENT
slave_1 YARN_CLIENT
slave_1 NODEMANAGER
slave_1 HDFS_CLIENT
slave_1 ZOOKEEPER_CLIENT
slave_1 DATANODE
slave_1 MAPREDUCE2_CLIENT

CLUSTER_BUILD:multi-node-hdfs-yarn>cluster autoAssign
HOSTGROUP HOST

---

master amb0.mycorp.kom
slave_1 amb1.mycorp.kom
slave_1 amb2.mycorp.kom

CLUSTER_BUILD:multi-node-hdfs-yarn>cluster create --exitOnFinish true
Successfully created the cluster
CLUSTER:multi-node-hdfs-yarn>

when I visit http://172.17.0.2:8080/, it shows error.
and I use docker exec -it amb0 bash to see what happen in container.

bash-4.1# ambari-server status
Using python /usr/bin/python2.6
Ambari-server statusCE2_CLIENT
Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pidve

the ambari-server is not running.

I check log and it show some error:
22:02:43,354 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (public-repo-1.hortonworks.com)
22:02:43,354 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
22:02:43,354 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (public-repo-1.hortonworks.com)
22:02:43,354 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
22:02:43,355 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (public-repo-1.hortonworks.com)
22:02:43,355 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
22:02:43,355 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json (public-repo-1.hortonworks.com)
22:02:43,355 INFO [Stack Version Loading Thread] LatestRepoCallable:72 - Loading latest URL info from http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json
22:02:43,355 ERROR [Stack Version Loading Thread] LatestRepoCallable:89 - Could not load the URI http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json (s3.amazonaws.com)

and I check those URI in the browser and it can show the data normally . I restart the ambari-server it still not work.
bash-4.1# ambari-server start
Using python /usr/bin/python2.6
Starting ambari-server
Ambari Server running with 'root' privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.........
ERROR: Exiting with exit code -1.
REASON: Ambari Server java process died with exitcode 143. Check /var/log/ambari-server/ambari-server.out for more information.
bash-4.1#

anyone knew that is the reason?

How to properly restart containers?

I installed sequenceiq/ambari:1.7.0 on Vagrant with Ubuntu and run cluster with 3 nodes. They were working fine and all services too. IPs of these nodes are 172.17.0.2...4
But I stopped containers and tried to restart using docker start amb0 amb1 amb2. Now Ambari is not see any of nodes because they are having other IPs. I think that would be better if cluster will work with any IPs of nodes

How to restart nodes properly and where is config for that hardcoded IP

Unwanted HBase regionserver

I noticed that the HBase region server installation gets messed up when doing a manual amb-shell installation using a json gist.

source ambari-functions
amb-start-cluster 3
amb-shell

blueprint add --url https://gist.githubusercontent.com/krisskross/901ed8223c1ed1db80e3/raw/869327be9ad15e6a9f099a7591323244cd245357/ambari-hdp2.3
cluster build --blueprint hdp-2.3
cluster assign --hostGroup master --host amb1.service.consul
cluster assign --hostGroup slave_1 --host amb2.service.consul
cluster create

host list
amb1.service.consul [ALERT] 172.17.0.79 centos6:x86_64
amb2.service.consul [ALERT] 172.17.0.80 centos6:x86_64

First, one extra container 12311dd1655c gets created that i'm not sure is needed?

$ sudo docker ps

CONTAINER ID        IMAGE                         COMMAND                CREATED             STATUS              PORTS                                                              NAMES
12311dd1655c        sequenceiq/ambari:2.1.2-v1    "/bin/sh -c /tmp/amb   42 minutes ago      Up 42 minutes       8080/tcp                                                           loving_stallman     
ff51cc267878        sequenceiq/ambari:2.1.2-v1    "/start-agent"         43 minutes ago      Up 43 minutes       8080/tcp                                                           amb2                
b50f6b429c61        sequenceiq/ambari:2.1.2-v1    "/start-agent"         43 minutes ago      Up 43 minutes       8080/tcp                                                           amb1                
78b6c91713ae        sequenceiq/ambari:2.1.2-v1    "/start-server"        43 minutes ago      Up 43 minutes       8080/tcp                                                           amb-server          
1169a087ce4a        sequenceiq/consul:v0.5.0-v6   "/bin/start -server    43 minutes ago      Up 43 minutes       53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 8500/tcp   amb-consul          

The HBase installation creates two instead of one region servers on the slave. This messes with regions in transition and the servers generally unstable.

amb2.node.dc1.consul
amb2.service.consul

I noticed that "node.dc1" comes from the start-agent and start-server scripts, but i'm not sure they are to blame. Any way, the amb2.node.dc1.consul region server must go.

Cannot start container. System error: exec: "--tag": executable file not found in $PATH

I am unable to start the Ambari container following your documentation.

I have installed Docker on a fresh, stock CentOS 7 VM (versions below) and ran the following command to start the docker container: docker run -d -P -h amb0.mycorp.kom -e KEYCHAIN=<keychain@email> --name amb0 sequenceiq/ambari --tag ambari-server=true.

I received the error:

Error response from daemon: Cannot start container 8e08ccf3dc76: [8] System error: exec: "--tag": executable file not found in $PATH
Error: failed to start containers: [8e08ccf3dc76]

Subsequent attempts to start the container with docker start yielded the same result. Possibly related to something in the Dockerfile?

Versions:

CentOS Linux release 7.1.1503 (Core)
Docker version 1.8.2, build 0a8c2e3

Adding custom files into the containers

Hello,

We are working on building a sandbox version that has a couple of extra files for services that we need baked in.

Is there any way we could add those file to the sequenceiq/ambari image provided here: http://blog.sequenceiq.com/blog/2014/12/04/multinode-ambari-1-7-0/ ?

Cannot run pig via a blueprint

Hi

I created a blueprint https://gist.githubusercontent.com/amitadpsl/33ff30ced085b1a63bf6/raw/e469f0694a79b32ab3de7ec68f877fa8d163ba56/multi-node-hdfs-yarn1

Followed the following steps to create a cluster

1)curl -Lo .amb j.mp/docker-ambari-170 && . .amb
2)amb-start-cluster 3
3) amb-shell
4) In the Ambari Shell:

blueprint add --url https://gist.githubusercontent.com/amitadpsl/33ff30ced085b1a63bf6/raw/fe602c64c2a51aa9a926ca82d2fe7e024c5f9175/multi-node-hdfs-yarn1
cluster build --blueprint multi-node-hdfs-yarn1
cluster autoAssign
cluster create

Next i used the datanode container to run a simple copyFromLocal command

I get the following error:

hadoop fs -copyFromLocal t.txt /user/.
15/06/12 16:24:08 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/t.txt.COPYING could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3203)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)

at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy15.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:te a pig scr1533)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1350)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:589)

copyFromLocal: File /user/t.txt.COPYING could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.

Same is true when i try to execute pig script

PLEASE HELP.

cannot start container in windows

I install docker in my window computer and it runs hello-world successfully. When I pull sequence/ambari image, and start container using both ambari-function and command line, it not works, the error is shown:

WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.
2c6ae14af473d17e4b43b666ba7a086437364dcff20a5745b3c52c21492a5019
Error response from daemon: Cannot start container 2c6ae14af473d17e4b43b666ba7a0
86437364dcff20a5745b3c52c21492a5019: [8] System error: exec: "C:/Program Files/G
it/usr/local/serf/bin/start-serf-agent.sh": stat C:/Program Files/Git/usr/local/
serf/bin/start-serf-agent.sh: no such file or directory

zqin@L-SHC-00436827 MINGW64 ~

Anyone knows what wrong?

How to boot ambari:1.7.0-consul without scripts?

Hi,
Can I boot ambari:1.7.0-consul without amb-scripts?
I usually use this command to boot my ambari:

# server
docker run -d -P -h <hostname> -p 100.80.140.74:53:53/udp --name <hostname>.url  sequenceiq/ambari:1.6.0 --tag ambari-server=true
# client
docker run -d -e SERF_JOIN_IP=$SERVER_IP --name ${NODE_PREFIX}$NUMBER -h <hostname> sequenceiq/ambari:1.6.0 --log-level debug

But the consul version doesn't let me do that!

$ docker run -d -h amb1 --name amb1 sequenceiq/ambari:1.7.0-consul --tag ambari-server=true

12906821a516db5a9fd40b573e93427c6684d65d5253dc769eefa81af2f86402
FATA[0002] Error response from daemon: Cannot start container 12906821a516db5a9fd40b573e93427c6684d65d5253dc769eefa81af2f86402: exec: "--tag": executable file not found in $PATH 

Any plans to support Ambari 2.0+? Options for HDP 2.3 locally?

I want to deploy HDP 2.3 to my local workstation or to a single EC2 instance in a manner that is consistent with using CloudBreak to deploy larger HDP clusters to the cloud.

Essentially this is to keep some level of consistency between development => staging => production, where our developers mirror our production environment at a smaller scale on their local workstations.

This is especially important because we always have issues hopping between major versions of HDP. So getting that properly deployed for HDP 2.3 is essential to smoothing the transition.

I came here from https://github.com/sequenceiq/cloudbreak/issues/444, where it appears CloudBreak will not be supporting either a local or single-node version of HDP. But it does not appear this project has been updated for Ambari 2.0 or 2.1, and is generally lagging behind.

Is there any plan to catch up? Can anybody at SequenceIQ or Hortonworks point me towards the best way to accomplish that goal?

submit jobs

Hi,
How to submit jobs either using ambari web or by logging in inside the container??

ambari agent failed to start

Hi,

I met the issue that ambari agent couldn't be started. It reported the error:

WARNING 2016-01-26 23:01:47,144 NetUtil.py:105 - Server at https://ambari-8080.service.consul:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-01-26 23:01:57,144 NetUtil.py:59 - Connecting to https://ambari-8080.service.consul:8440/ca
WARNING 2016-01-26 23:02:17,162 NetUtil.py:82 - Failed to connect to https://ambari-8080.service.consul:8440/ca due to [Errno -3] Temporary failure in name resolution

Connecting to Ambari HDFS Client container

I was able to run the cluster w/o issues. I installed HDFS client on all nodes and I simply do not know how to properly connect to one of the existing nodes for leveraging "hadoop" and other commands..

So I tried docker run -dit -name "c000" -host "amb1" /bin/bash followed by "docker connect c000". I was able to connect but I do not see any of HDFS clients.. I thought above will connect me to same container I launched HDFS client to .. but seems it is different

any suggestions

Ambari 1.7.0 fails to install cluster at ~5 - 6%

As per my discussion on http://blog.sequenceiq.com/blog/2014/12/04/multinode-ambari-1-7-0/

This is running in Ubuntu server 14.04.1 LTS

Docker version info:

Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

Image: sequenceiq/ambari tag: 1.7.0

Ambari 1.6.0 does work though.

For zipped log files from the 3 containers please let me know where I can send them.

Data persistance

Hi

Does this docker image allow data persistence? i.e. consider the following

 1. I perform docker pull, initialize my HDP services
 2. Upload some data files in HDFS and Hive etc
 3. Shut down docker 
 4. Restart docker image

Will my data persist between restarts, and do I have to setup services like Hive, Pig etc every time I restart docker and upload my files in HDFS?

Cannot install hdp-multinode-default

Hi

I have been using your nifty shell functions sucessfully with your default settings when creating a cluster. E.g amb-deploy-cluster 4

Now I am trying to use thehdp-multinode-default blueprint, but installation fails at around 12%.

Here is the command that I am running: amb-deploy-cluster 6 hdp-multinode-default

This is what Ambari shows:

Here is the detailed message from stdout:

2014-10-09 04:53:31,154 - Package['unzip'] {}
2014-10-09 04:53:32,234 - Skipping installing existent package unzip
2014-10-09 04:53:32,234 - Package['curl'] {}
2014-10-09 04:53:32,257 - Skipping installing existent package curl
2014-10-09 04:53:32,257 - Package['net-snmp-utils'] {}
2014-10-09 04:53:32,279 - Skipping installing existent package net-snmp-utils
2014-10-09 04:53:32,279 - Package['net-snmp'] {}
2014-10-09 04:53:32,309 - Skipping installing existent package net-snmp
2014-10-09 04:53:32,310 - Execute['mkdir -p /tmp/HDP-artifacts/ ;   curl -kf   --retry 10 http://amb0.mycorp.kom:8080/resources//jdk-7u45-linux-x64.tar.gz -o /tmp/HDP-artifacts//jdk-7u45-linux-x64.tar.gz'] {'environment': ..., 'not_if': 'test -e /usr/jdk64/jdk1.7.0_45/bin/java', 'path': ['/bin', '/usr/bin/']}
2014-10-09 04:53:32,326 - Skipping Execute['mkdir -p /tmp/HDP-artifacts/ ;   curl -kf   --retry 10 http://amb0.mycorp.kom:8080/resources//jdk-7u45-linux-x64.tar.gz -o /tmp/HDP-artifacts//jdk-7u45-linux-x64.tar.gz'] due to not_if
2014-10-09 04:53:32,327 - Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /tmp/HDP-artifacts//jdk-7u45-linux-x64.tar.gz > /dev/null 2>&1'] {'not_if': 'test -e /usr/jdk64/jdk1.7.0_45/bin/java', 'path': ['/bin', '/usr/bin/']}
2014-10-09 04:53:32,344 - Skipping Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /tmp/HDP-artifacts//jdk-7u45-linux-x64.tar.gz > /dev/null 2>&1'] due to not_if
2014-10-09 04:53:32,345 - Execute['mkdir -p /tmp/HDP-artifacts/;     curl -kf --retry 10     http://amb0.mycorp.kom:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /tmp/HDP-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment': ..., 'not_if': 'test -e /tmp/HDP-artifacts//UnlimitedJCEPolicyJDK7.zip', 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']}
2014-10-09 04:53:32,448 - Group['hadoop'] {}
2014-10-09 04:53:32,451 - Modifying group hadoop
2014-10-09 04:53:32,824 - Group['users'] {}
2014-10-09 04:53:32,825 - Modifying group users
2014-10-09 04:53:33,114 - Group['users'] {}
2014-10-09 04:53:33,115 - Modifying group users
2014-10-09 04:53:33,445 - User['ambari-qa'] {'gid': 'hadoop', 'groups': [u'users']}
2014-10-09 04:53:33,445 - Adding user User['ambari-qa']
2014-10-09 04:53:36,186 - File['/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-10-09 04:53:36,288 - Writing File['/tmp/changeUid.sh'] because it doesn't exist
2014-10-09 04:53:36,309 - Changing permission for /tmp/changeUid.sh from 644 to 555
2014-10-09 04:53:36,311 - Execute['/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
2014-10-09 04:53:36,872 - User['hbase'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2014-10-09 04:53:36,872 - Adding user User['hbase']
2014-10-09 04:53:38,064 - File['/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-10-09 04:53:38,066 - Execute['/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/hadoop/hbase 2>/dev/null'] {'not_if': 'test $(id -u hbase) -gt 1000'}
2014-10-09 04:53:38,084 - Skipping Execute['/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/hadoop/hbase 2>/dev/null'] due to not_if
2014-10-09 04:53:38,084 - Group['nagios'] {}
2014-10-09 04:53:38,085 - Adding group Group['nagios']
2014-10-09 04:53:38,759 - User['nagios'] {'gid': 'nagios'}
2014-10-09 04:53:38,760 - Adding user User['nagios']
2014-10-09 04:53:39,367 - User['oozie'] {'gid': 'hadoop'}
2014-10-09 04:53:39,368 - Adding user User['oozie']
2014-10-09 04:53:40,064 - User['hcat'] {'gid': 'hadoop'}
2014-10-09 04:53:40,064 - Adding user User['hcat']
2014-10-09 04:53:40,641 - User['hcat'] {'gid': 'hadoop'}
2014-10-09 04:53:40,641 - Modifying user hcat
2014-10-09 04:53:40,655 - User['hive'] {'gid': 'hadoop'}
2014-10-09 04:53:40,655 - Adding user User['hive']
2014-10-09 04:53:41,369 - User['yarn'] {'gid': 'hadoop'}
2014-10-09 04:53:41,369 - Modifying user yarn
2014-10-09 04:53:41,663 - Group['nobody'] {}
2014-10-09 04:53:41,664 - Modifying group nobody
2014-10-09 04:53:41,998 - Group['nobody'] {}
2014-10-09 04:53:41,998 - Modifying group nobody
2014-10-09 04:53:42,307 - User['nobody'] {'gid': 'hadoop', 'groups': [u'nobody']}
2014-10-09 04:53:42,308 - Modifying user nobody
2014-10-09 04:53:43,027 - User['nobody'] {'gid': 'hadoop', 'groups': [u'nobody']}
2014-10-09 04:53:43,027 - Modifying user nobody
2014-10-09 04:53:43,044 - User['hdfs'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2014-10-09 04:53:43,044 - Modifying user hdfs
2014-10-09 04:53:43,278 - User['mapred'] {'gid': 'hadoop', 'groups': [u'hadoop']}
2014-10-09 04:53:43,279 - Modifying user mapred
2014-10-09 04:53:43,515 - User['zookeeper'] {'gid': 'hadoop'}
2014-10-09 04:53:43,515 - Modifying user zookeeper
2014-10-09 04:53:43,840 - Repository['HDP-2.1'] {'action': ['create'], 'mirror_list': None, 'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.1.2.0/', 'components': ['HDP', 'main'], 'repo_file_name': 'HDP'}
2014-10-09 04:53:43,854 - File['/etc/yum.repos.d/HDP.repo'] {'content': InlineTemplate(...)}
2014-10-09 04:53:43,856 - Writing File['/etc/yum.repos.d/HDP.repo'] because it doesn't exist
2014-10-09 04:53:43,858 - Package['libganglia-3.5.0-99'] {}
2014-10-09 04:53:43,889 - Installing package libganglia-3.5.0-99 ('/usr/bin/yum -d 0 -e 0 -y install libganglia-3.5.0-99')
2014-10-09 04:55:28,428 - Package['ganglia-devel-3.5.0-99'] {}
2014-10-09 04:55:28,450 - Installing package ganglia-devel-3.5.0-99 ('/usr/bin/yum -d 0 -e 0 -y install ganglia-devel-3.5.0-99')
2014-10-09 04:55:39,627 - Package['ganglia-gmetad-3.5.0-99'] {}
2014-10-09 04:55:39,656 - Installing package ganglia-gmetad-3.5.0-99 ('/usr/bin/yum -d 0 -e 0 -y install ganglia-gmetad-3.5.0-99')
2014-10-09 04:56:36,408 - Package['ganglia-web-3.5.7-99.noarch'] {}
2014-10-09 04:56:36,438 - Installing package ganglia-web-3.5.7-99.noarch ('/usr/bin/yum -d 0 -e 0 -y install ganglia-web-3.5.7-99.noarch')

I put the shell output below:

[root@lee lee]# amb-deploy-cluster 6 hdp-multinode-default
starting an ambari cluster with: 6 nodes
[DEBUG] docker run -d --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb0 -h amb0.mycorp.kom sequenceiq/ambari:1.6.0 --tag ambari-server=true
19b4ed2a9363a1203302358b0f6bc3e89da937d15d6cb68643d405dee71a4a2e
[DEBUG] docker run -d -e SERF_JOIN_IP=172.17.0.8 --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb1 -h amb1.mycorp.kom sequenceiq/ambari:1.6.0 --log-level debug
a9ba0f6e5622d1a37dfb43a7cfb9e84326d70ca3b8fb8701cee841508babd138
[DEBUG] docker run -d -e SERF_JOIN_IP=172.17.0.8 --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb2 -h amb2.mycorp.kom sequenceiq/ambari:1.6.0 --log-level debug
881b1402ed65db3ec21bb358132e0f7e1cca602a3f256d383503ffc2a2771976
[DEBUG] docker run -d -e SERF_JOIN_IP=172.17.0.8 --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb3 -h amb3.mycorp.kom sequenceiq/ambari:1.6.0 --log-level debug
f90a012ba61d532adf7d181ac4ad536088fd414d73f4a2b59028fc5f0a8aaafa
[DEBUG] docker run -d -e SERF_JOIN_IP=172.17.0.8 --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb4 -h amb4.mycorp.kom sequenceiq/ambari:1.6.0 --log-level debug
9ac53b4a2f337797ab9dc83db80006e40a0e03571b53728b891526f4cdf3f678
[DEBUG] docker run -d -e SERF_JOIN_IP=172.17.0.8 --dns 127.0.0.1 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= --name amb5 -h amb5.mycorp.kom sequenceiq/ambari:1.6.0 --log-level debug
6cd606f7b201ef35bcccef16836bd20ea3ace50d9cb6af42f8ed8a316041885d
[DEBUG] docker run -it --rm -e EXPECTED_HOST_COUNT=6 -e BLUEPRINT=hdp-multinode-default --link amb0:ambariserver --entrypoint /bin/sh sequenceiq/ambari:1.6.0 -c /tmp/install-cluster.sh
AMBARI_HOST=172.17.0.8
[DEBUG] waits for ambari server: 172.17.0.8 RUNNING ...
...........
[DEBUG] waits until 6 hosts connected to server ...
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 0
[DEBUG] connected hosts: 6
    _                _                   _  ____   _            _  _ 
   / \    _ __ ___  | |__    __ _  _ __ (_)/ ___| | |__    ___ | || |
  / _ \  | '_ ` _ \ | '_ \  / _` || '__|| |\___ \ | '_ \  / _ \| || |
 / ___ \ | | | | | || |_) || (_| || |   | | ___) || | | ||  __/| || |
/_/   \_\|_| |_| |_||_.__/  \__,_||_|   |_||____/ |_| |_| \___||_||_|


Welcome to Ambari Shell. For command and param completion press TAB, for assistance type 'hint'.
ambari-shell>blueprint defaults
Default blueprints added
ambari-shell>cluster build --blueprint hdp-multinode-default
  HOSTNAME         STATE
  ---------------  -------
  amb1.mycorp.kom  UNKNOWN
  amb0.mycorp.kom  UNKNOWN
  amb4.mycorp.kom  UNKNOWN
  amb3.mycorp.kom  UNKNOWN
  amb2.mycorp.kom  UNKNOWN
  amb5.mycorp.kom  UNKNOWN

  HOSTGROUP  COMPONENT
  ---------  ------------------
  master_4   OOZIE_SERVER
  master_4   ZOOKEEPER_SERVER
  master_4   GANGLIA_MONITOR
  slave_1    NODEMANAGER
  slave_1    HBASE_REGIONSERVER
  slave_1    GANGLIA_MONITOR
  slave_1    DATANODE
  gateway    YARN_CLIENT
  gateway    HIVE_CLIENT
  gateway    HDFS_CLIENT
  gateway    SQOOP
  gateway    GANGLIA_SERVER
  gateway    HBASE_CLIENT
  gateway    OOZIE_CLIENT
  gateway    AMBARI_SERVER
  gateway    PIG
  gateway    ZOOKEEPER_CLIENT
  gateway    GANGLIA_MONITOR
  gateway    MAPREDUCE2_CLIENT
  gateway    NAGIOS_SERVER
  gateway    HCAT
  master_2   YARN_CLIENT
  master_2   HIVE_CLIENT
  master_2   HDFS_CLIENT
  master_2   HIVE_SERVER
  master_2   HIVE_METASTORE
  master_2   HISTORYSERVER
  master_2   ZOOKEEPER_CLIENT
  master_2   WEBHCAT_SERVER
  master_2   GANGLIA_MONITOR
  master_2   MYSQL_SERVER
  master_2   SECONDARY_NAMENODE
  master_3   ZOOKEEPER_SERVER
  master_3   RESOURCEMANAGER                                Installation: FAILED[root@lee lee]# GLIA_MONITOR
  master_1   YARN_CLIENT
  master_1   HDFS_CLIENT
  master_1   NAMENODE
  master_1   GANGLIA_SERVER
  master_1   HBASE_MASTER
  master_1   ZOOKEEPER_SERVER
  master_1   GANGLIA_MONITOR
  master_1   HCAT

CLUSTER_BUILD:hdp-multinode-default>cluster autoAssign
  HOSTGROUP  HOST
  ---------  ---------------
  master_4   amb0.mycorp.kom
  gateway    amb1.mycorp.kom
  master_2   amb2.mycorp.kom
  master_3   amb3.mycorp.kom
  master_1   amb4.mycorp.kom
  slave_1    amb5.mycorp.kom

CLUSTER_BUILD:hdp-multinode-default>cluster create --exitOnFinish true
Successfully created the cluster