jp/ambari

Chef cookbook to install Ambari server and Ambari agent, then deploy a Hadoop cluster using the Blueprints API

Ambari Cookbook

Install ambari-server and ambari-agent in order to provision a cluster with the Hortonworks Data Platform. This cookbook also supports provisioning supported services through the Ambari API via Ambari Blueprints. For more on blueprints, please refer to the Apache project's documentation here.

Requirements

Supports all OS distributions and versions supported by Ambari. Supports Ambari versions 1.7 through 2.4.

• RHEL, CentOS, and Oracle Linux 5, 6, and 7.
• Suse 11
• Ubuntu 12 and 14.
• Debian 7

Usage

ambari::server

Include ambari::server in your node's run_list:

{
  "name":"my_node",
  "run_list": [
    "recipe[ambari::server]"
  ]
}

ambari::agent

Include ambari::agent in your node's run_list (all the nodes of your cluster):

The Ambari Server will be automatically detected, but you can override this detection using the attribute node['ambari']['server_fqdn'].

{
  "name":"my_node",
  "run_list": [
    "recipe[ambari::agent]"
  ]
}

ambari::blueprints

Include ambari::blueprints in your node's run_list. Then configure the blueprints attributes listed below.

Main Attributes

Key	Description	Default
node['ambari']['rhel_5_repo']	URL for the RHEL 5.x repository	http://public-repo-1.hortonworks.com/ambari/centos5/2.x/updates/2.0.0/ambari.repo
node['ambari']['rhel_6_repo']	URL for the RHEL 6.x repository	http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.0.0/ambari.repo
node['ambari']['suse_11_repo']	URL for the Suse 11.x repository	http://public-repo-1.hortonworks.com/ambari/suse11/2.x/updates/2.0.0/ambari.repo
node['ambari']['server_fdqn']	FQDN for the Ambari server	Found by searching a node using the recipe ambari::server in the same environment

Blueprints Attributes

Key	Description
node['ambari']['blueprints']['blueprint_name']	Name of the cluster's blueprint
node['ambari']['blueprints']['blueprint_json']	JSON description of the Blueprint. See a single-node example here : https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-ExampleBlueprint
node['ambari']['blueprints']['cluster_name']	Cluster's name
node['ambari']['blueprints']['cluster_json']	JSON template for the Cluster. See a single-node example here: https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-ExampleClusterCreationTemplate

Contributing

If you wish to contribute on this cookbook:

1. Fork the repository on Github
2. Create a named feature branch (like add_component_x)
3. Write your change
4. Write tests for your change (if applicable)
5. Run the tests, ensuring they all pass
6. Submit a Pull Request using Github

License and Authors

Author: Julien Pellet

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

ambari's Issues

Any plans to configure ambari.properties?

Hi,

Ambari Server's configuration is stored in /etc/ambari-server/conf/ambari.properties. For example, to enable ldap authentication requires changing numerous settings in this file. Would be great for ambari.properties to be configurable by the server.rb recipe. Many thanks.

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

Hi,

Thanks for this cookbook, but on Ubuntu 15.04,
I'm having this error on the agent:

ERROR 2015-10-12 14:39:49,408 NetUtil.py:78 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
ERROR 2015-10-12 14:39:49,408 NetUtil.py:79 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

When I curl the server port 8440, I'm getting:

curl: (60) SSL certificate problem: self signed certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Any suggestion ?

Please add a license to the repo

Some of the files do have a comment at the top stating the license is Apache 2.0, but it would be much clearer if there were a LICENSE file at the root level of the repo with the entire Apache 2.0 license, and if the README specified the license as well. Right now, the README has a section called "License and Authors" but doesn't specify a license; it only lists an author.

Thanks!

[Maintenance] Add support for Ambari 2.2.2

Ambari 2.2.2

No resource or method named `not_if' for `Chef::Recipe "setup_package_manager"'

Hey there!

I ran into error No resource or method namednot_if' for Chef::Recipe "setup_package_manager when attempting to use this cookbook with the ChefDK (0.4.0) and Chef (12.2.1).

================================================================================
10.0.1.78 Recipe Compile Error in /var/chef/cache/cookbooks/hdp/recipes/ambari_server.rb
10.0.1.78 ================================================================================
10.0.1.78
10.0.1.78 NoMethodError
10.0.1.78 -------------
10.0.1.78 No resource or method named `not_if' for `Chef::Recipe "setup_package_manager"'
10.0.1.78
10.0.1.78 Cookbook Trace:
10.0.1.78 ---------------
10.0.1.78   /var/chef/cache/cookbooks/ambari/recipes/setup_package_manager.rb:45:in `from_file'
10.0.1.78   /var/chef/cache/cookbooks/ambari/recipes/server.rb:20:in `from_file'
10.0.1.78   /var/chef/cache/cookbooks/hdp/recipes/ambari_server.rb:1:in `from_file'
10.0.1.78
10.0.1.78 Relevant File Content:
10.0.1.78 ----------------------
10.0.1.78 /var/chef/cache/cookbooks/ambari/recipes/setup_package_manager.rb:
10.0.1.78
10.0.1.78  38:      yum_repo = node['ambari']['rhel_5_repo']
10.0.1.78  39:    when 6
10.0.1.78  40:      yum_repo = node['ambari']['rhel_6_repo']
10.0.1.78  41:    end
10.0.1.78  42:    remote_file "/etc/yum.repos.d/ambari.repo" do
10.0.1.78  43:      source yum_repo
10.0.1.78  44:    end
10.0.1.78  45>>   not_if do
10.0.1.78  46:      ::File.exists?("/etc/yum.repos.d/ambari.repo")
10.0.1.78  47:    end
10.0.1.78  48:  when "suse"
10.0.1.78  49:    remote_file "/etc/zypp/repos.d/ambari.repo" do
10.0.1.78  50:      source node['ambari']['suse_11_repo']
10.0.1.78  51:    end
10.0.1.78  52:    not_if do
10.0.1.78  53:      ::File.exists?("/etc/zypp/repos.d/ambari.repo")
10.0.1.78  54:    end

Versions:

chef (12.2.1, 12.0.3)
chef-dk (0.4.0)
chef-provisioning (1.0.1, 0.18)
chef-provisioning-aws (1.0.4, 0.2.1)
chef-provisioning-azure (0.1)
chef-provisioning-fog (0.12)
chef-provisioning-vagrant (0.8.1)
chef-vault (2.4.0)
chef-zero (4.2.1, 3.2.1, 1.5.6)
cheffish (1.1.2, 0.9.2)
chefspec (4.2.0)

Conflicting version of snappy 1.1.0; ambari requires 1.0.5 for rhel7

Julien; while trying ambari to deploy applications throughout the cluster we encountered an issue with the version of snappy; it's a known problem for rhel7/hdp, see http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_releasenotes_ambari_2.1.0.0/content/ambari_relnotes-2.1.0.0-known-issues.html

resource_management.core.exceptions.
Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install snappy-devel' returned 1.
Error: Package: snappy-devel-1.0.5-1.el6.x86_64 (HDP-UTILS-1.1.0.20)
Requires: snappy(x86-64) = 1.0.5-1.el6
Installed: snappy-1.1.0-3.el7.x86_64 (@anaconda/7.1)
snappy(x86-64) = 1.1.0-3.el7
Available: snappy-1.0.5-1.el6.x86_64 (HDP-UTILS-1.1.0.20)
snappy(x86-64) = 1.0.5-1.el6

A fix can be to add following to the agent recipe:

yum_package 'remove snappy 1.1.0-3.el7' do
package_name 'snappy'
version '1.1.0-3.el7'
action :remove
end

I realize that this may not be the best solution as the version is hardcoded and did not test other platforms or ambari versions.

Paul

Supermarket release is quite outdated

It seems like supermarket release is lagging by almost a year. By any chance you could update it?

Thanks for maintaining this!

Support Ambari 2.0.0

Ran into an issue today, using this cookbook to expand an old cluster that was provisioned with Ambari 2.0.0.

The repos under the case switch https://github.com/jp/ambari/blob/master/attributes/default.rb#L17 for Ambari 2.0.0 had been switched to the repos for 2.0.1 in #4ed33fb.

This meant any node provisioned with that code and added to an existing cluster would have an inconsistent version on the Ambari Server (2.0.0) and the newly-provisioned Ambari Agent (2.0.1) and fail to register.

INFO 2015-10-29 18:56:40,612 NetUtil.py:60 - Connecting to https://ip-10-0-1-168.ec2.internal:8440/connection_info
INFO 2015-10-29 18:56:40,724 security.py:93 - SSL Connect being called.. connecting to the server
INFO 2015-10-29 18:56:40,816 security.py:55 - SSL connection established. Two-way SSL authentication is turned off on the server.
ERROR 2015-10-29 18:56:40,860 Controller.py:143 - Cannot register host with non compatible agent version, hostname=ip-10-0-1-161.ec2.internal, agentVersion=2.0.1, serverVersion=2.0.0
INFO 2015-10-29 18:56:40,860 Controller.py:350 - Registration response from ip-10-0-1-168.ec2.internal was FAILED

Anyways, I ran into this today, saw the bug, and fixed it in 9876342.

Add support for Ambari 2.0, other versions.

Hey guys!

Ambari 2.0 has been out for awhile now. There are some pretty significant changes, so I think it's probably a good idea to support multiple versions.

I have a forked version supporting 2.0 I've been running for about a week now. I'll put together a pull request if there is interest in that.

Regards,
Joe Reid

Clarify docs - How to use blueprints recipe

Hey JP,

I am automating a single-node install in Kitchen. I've never used the blueprints recipe before and I'm not fully clear on how to do that from the README.md.

README.md - https://github.com/jp/ambari#blueprints-attributes
Recipe - https://github.com/jp/ambari/blob/master/recipes/blueprints.rb

I'm conversant in Ambari blueprints and have been using them in some capacity for the last year or so. We've just never automated this step through Chef before.

I'm looking at the code and the docs and I am not sure how to use the following two attributes correctly:

node['ambari']['blueprints']['blueprint_json'] = JSON description of the Blueprint. See a single-node example here : https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-ExampleBlueprint
node['ambari']['blueprints']['cluster_json'] = JSON template for the Cluster. See a single-node example here: https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-ExampleClusterCreationTemplate

Do those attributes need to be defined as file paths within the cookbooks? Are we creating these .json files as cookbook templates, then converting to .json in the recipe? I guess I'm looking for an example of how this works.

So let's say I have the following cluster_creation.json and single_node_blueprint.json. Where would I put those files within the cookbook, and what would I configure as these two attributes?

Happy to submit a patch for this once I understand it better.

Regards,
Joe Reid

setup_package_manager.rb misses case for rhel7

For ambari v2.1 the attribute rhel_7_repo is set already but the case is missing so the repo is not selected making the recipe fail.

Fix:

case node['platform']
when 'redhat', 'centos', 'amazon', 'scientific'
case node['platform_version'].to_i
when 5
yum_repo = node['ambari']['rhel_5_repo']
when 6
yum_repo = node['ambari']['rhel_6_repo']
when 7
yum_repo = node['ambari']['rhel_7_repo']
end
remote_file '/etc/yum.repos.d/ambari.repo' do
source yum_repo
not_if { ::File.exist?('/etc/yum.repos.d/ambari.repo') }
end

External Database Config

Is there a way to pass in external database configuration for ambari-server to use instead of defaulting to the embedded postgres? If not, this would be in incredibly useful feature

Exported blueprint won't upload

I've exported my ambari config to json. When running the ambari::blueprints chef recipe and importing, curl fails to post the blueprint with error: "execute[Init Blueprints] action runsh: 1: Syntax error: "(" unexpected"

From the json, I have copied from the brace after "blueprint_json": to the corresponding closing brace and attached here: json_file.txt

I can't upload .json files to this website, so had to rename to .txt. If you open, remove line-breaks in text using [http://www.textfixer.com/tools/remove-line-breaks.php], and at shell prompt type: curl --user admin:<password> -H 'X-Requested-By:ambari-cookbook' --data '<paste_text>' <your-ambari-server>:8080/api/v1/blueprints/my_blueprint Similar error occurs: -bash: syntax error near unexpected token `('

Interestingly, if you curl post this json as a file rather than text it works fine: at shell prompt type: curl --user admin:<password> -H 'X-Requested-By:ambari-cookbook' --data @<json_file> <your-ambari-server>:8080/api/v1/blueprints/my_blueprint

Is this a bug? If so, saving the ['ambari']['blueprints']['blueprint_json'] value to a file and curl-posting this instead of the text to ambari seems the solution.

My ambari config not particularly complex, so I expect others will export and experience this issue.

Ambari Agent 2.4 fails to start in Centos7

Hello all!

I am using Centos 7, almost a pristine minimal installation, with few changes only for hostname and NTP server.
I have Python 2.7 installed, as it is required by Ambari on this system.

I have successfully deployed an Ambari Server using this cookbook. However when I try to deploy the agents using the other recipe I find that chef fails to execute the resource 'service[ambari-agent]', the error I pasted below is shown and then thousands of lines where the system tries to chown for root:root everything under /proc directory, leading to a system where almost anything can work.

` * service[ambari-agent] action start

================================================================================
Error executing action `start` on resource 'service[ambari-agent]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '255'
---- Begin output of /sbin/service ambari-agent start ----
STDOUT: Verifying Python version compatibility...
Using python  /usr/bin/python
Checking for previously running Ambari Agent...
Starting ambari-agent
Verifying ambari-agent process status...
ERROR: ambari-agent start failed. For more details, see /ambari-agent.out:
====================
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 387, in <module>
    main(heartbeat_stop_callback)
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 363, in main
    controller = Controller(config, server_hostname, heartbeat_stop_callback)
  File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 107, in __init__
    self.heartbeat_idle_interval_min = int(self.config.get('heartbeat', 'idle_interval_min')) if self.config.get('heartbeat', 'idle_interval_min') else self.netutil.HEARTBEAT_IDLE_INTERVAL_DEFAULT_MIN_SEC
  File "/usr/lib/python2.6/site-packages/ambari_agent/AmbariConfig.py", line 172, in get
    raise err
ConfigParser.NoOptionError: No option 'idle_interval_min' in section: 'heartbeat'
====================
Agent out at: /ambari-agent.out
Agent log at: /ambari-agent.log
STDERR: mkdir: cannot create directory '': No such file or directory`

Any Idea why this is happening?