AMZ linux and chef 13 ohai hints issue

[chef-boneyard]

AMZ linux and chef 13 ohai hints issue

[tas50]

I'm a bit confused on where exactly this failure is coming from, but it looks like you're using the aws:ec2_hints recipe. The good news is you don't need that anymore. You can just remove that from your runlist entirely. Chef will autodetect AWS without issue now.

[Stromweld]

So I removed the aws::ec2_hints recipe from our base cookbook, but when our ohai-nativex cookbook tries to create an awsplugin for ohai and sets ['aws']['region'] attribute we get the same error:

[2017-04-19T16:33:59+00:00] WARN: Plugin Network: unable to detect ipaddress
resolving cookbooks for run list: ["role-base-amz-linux-nativex"]�[0m
Synchronizing Cookbooks:�[0m
- chef-sugar (3.4.0)�[0m
- nano (1.0.3)�[0m
- line (0.6.3)�[0m
- motd (0.6.4)�[0m
- hostsfile (2.4.5)�[0m
- dstat (0.1.1)�[0m
- sshd (1.3.0)�[0m
- bash-completion (0.1.0)�[0m
- compat_resource (12.16.3)�[0m
- vim (2.0.2)�[0m
- yum-epel (2.1.1)�[0m
- logrotate (2.1.0)�[0m
- sudo (3.3.1)�[0m
- now (1.0.0)�[0m
- ssh_authorized_keys (0.3.0)�[0m
- snmp (4.0.0)�[0m
- nfs (2.3.1)�[0m
- seven_zip (2.0.2)�[0m
- chef_handler (2.1.1)�[0m
- mingw (2.0.0)�[0m
- sysctl (0.8.1)�[0m
- collectd-ng (2.2.0)�[0m
- aws-tagging-nativex (1.1.4)�[0m
- ephemeral_lvm-nativex (1.0.6)�[0m
- aws (5.0.0)�[0m
- ohai (5.0.3)�[0m
- lvm (4.0.6)�[0m
- chef_client_updater (1.0.2)�[0m
- ssh-authorized-keys-nativex (1.0.3)�[0m
- ntp-nativex (1.1.3)�[0m
- mlocate-nativex (1.0.3)�[0m
- windows (3.0.5)�[0m
- autofs-nativex (1.1.4)�[0m
- iptables (4.1.0)�[0m
- ohai-nativex (1.0.5)�[0m
- cron (4.1.1)�[0m
- ephemeral_lvm (3.0.1)�[0m
- graphite-nativex (1.1.6)�[0m
- postfix-nativex (1.0.3)�[0m
- solarwinds-nativex (1.5.3)�[0m
- timezone-ii (0.2.0)�[0m
- ad-nativex (3.3.6)�[0m
- autopatch-nativex (1.3.6)�[0m
- chef-client (8.0.1)�[0m
- auto-patch (0.1.10)�[0m
- build-essential (8.0.1)�[0m
- timezone-nativex (1.0.5)�[0m
- hostname-nativex (1.0.8)�[0m
- role-base-amz-linux-nativex (1.2.4)�[0m
Installing Cookbook Gems:�[0m
Compiling Cookbooks...�[0m
/var/chef/cache/cookbooks/hostsfile/resources/entry.rb:35: warning: constant ::Fixnum is deprecated
Recipe: ohai-nativex::awsplugin�[0m
* ohai_hint[aws] action create
* directory[/etc/chef/ohai/hints] action create (up to date)
* file[/etc/chef/ohai/hints/aws.json] action create
�[32m- create new file /etc/chef/ohai/hints/aws.json�[0m�[0m
================================================================================�[0m
�[31mError executing action `create` on resource 'file[/etc/chef/ohai/hints/aws.json]'�[0m
================================================================================�[0m

�[0m NoMethodError�[0m
-------------�[0m
undefined method `split' for nil:NilClass�[0m

�[0m Cookbook Trace:�[0m
---------------�[0m
/var/chef/cache/cookbooks/ohai/resources/hint.rb:53:in `block in after_created'
�[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `each'
�[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `after_created'
�[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/awsplugin.rb:10:in `from_file'
�[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/default.rb:10:in `from_file'
�[0m /var/chef/cache/cookbooks/role-base-amz-linux-nativex/recipes/default.rb:151:in `from_file'�[0m

�[0m Resource Declaration:�[0m
---------------------�[0m
# In /var/chef/cache/cookbooks/ohai/resources/hint.rb
�[0m
�[0m 11: file ohai_hint_path do
�[0m 12: action :create
�[0m 13: content build_content
�[0m 14: end
�[0m 15: end
�[0m
�[0m Compiled Resource:�[0m
------------------�[0m
# Declared in /var/chef/cache/cookbooks/ohai/resources/hint.rb:11:in `block in class_from_file'
�[0m
�[0m file("/etc/chef/ohai/hints/aws.json") do
�[0m action [:create]
�[0m default_guard_interpreter :default
�[0m declared_type :file
�[0m cookbook_name "ohai-nativex"
�[0m path "/etc/chef/ohai/hints/aws.json"
�[0m owner nil
�[0m group nil
�[0m mode nil
�[0m verifications []
�[0m end
�[0m
�[0m System Info:�[0m
------------�[0m
chef_version=13.0.118
�[0m platform=amazon
�[0m platform_version=2016.09
�[0m ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
�[0m program_name=chef-client worker: ppid=32501;start=16:33:58;
�[0m executable=/opt/chef/bin/chef-client�[0m

�[0m �[0m
================================================================================�[0m
�[31mError executing action `create` on resource 'ohai_hint[aws]'�[0m
================================================================================�[0m

�[0m NoMethodError�[0m
-------------�[0m
file[/etc/chef/ohai/hints/aws.json] (/var/chef/cache/cookbooks/ohai/resources/hint.rb line 11) had an error: NoMethodError: undefined method `split' for nil:NilClass�[0m

�[0m Cookbook Trace:�[0m
---------------�[0m
/var/chef/cache/cookbooks/ohai/resources/hint.rb:53:in `block in after_created'
�[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `each'
�[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `after_created'
�[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/awsplugin.rb:10:in `from_file'
�[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/default.rb:10:in `from_file'
�[0m /var/chef/cache/cookbooks/role-base-amz-linux-nativex/recipes/default.rb:151:in `from_file'�[0m

�[0m Resource Declaration:�[0m
---------------------�[0m
# In /var/chef/cache/cookbooks/ohai-nativex/recipes/awsplugin.rb
�[0m
�[0m 10: ohai_hint 'aws'
�[0m 11:
�[0m
�[0m Compiled Resource:�[0m
------------------�[0m
# Declared in /var/chef/cache/cookbooks/ohai-nativex/recipes/awsplugin.rb:10:in `from_file'
�[0m
�[0m ohai_hint("aws") do
�[0m action [:create]
�[0m default_guard_interpreter :default
�[0m declared_type :ohai_hint
�[0m cookbook_name "ohai-nativex"
�[0m recipe_name "awsplugin"
�[0m hint_name "aws"
�[0m end
�[0m
�[0m System Info:�[0m
------------�[0m
chef_version=13.0.118
�[0m platform=amazon
�[0m platform_version=2016.09
�[0m ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
�[0m program_name=chef-client worker: ppid=32501;start=16:33:58;
�[0m executable=/opt/chef/bin/chef-client�[0m

�[0m �[0m
================================================================================�[0m
�[31mRecipe Compile Error in /var/chef/cache/cookbooks/role-base-amz-linux-nativex/recipes/default.rb�[0m
================================================================================�[0m

�[0m NoMethodError�[0m
-------------�[0m
ohai_hint[aws] (ohai-nativex::awsplugin line 10) had an error: NoMethodError: file[/etc/chef/ohai/hints/aws.json] (/var/chef/cache/cookbooks/ohai/resources/hint.rb line 11) had an error: NoMethodError: undefined method `split' for nil:NilClass�[0m

�[0m Cookbook Trace:�[0m
---------------�[0m
/var/chef/cache/cookbooks/ohai/resources/hint.rb:53:in `block in after_created'
�[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `each'
�[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `after_created'
�[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/awsplugin.rb:10:in `from_file'
�[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/default.rb:10:in `from_file'
�[0m /var/chef/cache/cookbooks/role-base-amz-linux-nativex/recipes/default.rb:151:in `from_file'�[0m

�[0m Relevant File Content:�[0m
----------------------�[0m
/var/chef/cache/cookbooks/ohai/resources/hint.rb:
�[0m
�[0m 46: end
�[0m 47: end
�[0m 48:
�[0m 49: # this resource forces itself to run at compile_time
�[0m 50: def after_created
�[0m 51: return unless compile_time
�[0m 52: Array(action).each do |action|
�[0m 53>> run_action(action)
�[0m 54: end
�[0m 55: end
�[0m 56: �[0m

�[0m System Info:�[0m
------------�[0m
chef_version=13.0.118
�[0m platform=amazon
�[0m platform_version=2016.09
�[0m ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
�[0m program_name=chef-client worker: ppid=32501;start=16:33:58;
�[0m executable=/opt/chef/bin/chef-client�[0m

�[0m �[0m
Running handlers:�[0m
[2017-04-19T16:34:01+00:00] ERROR: Running exception handlers
Running handlers complete
�[0m[2017-04-19T16:34:01+00:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 03 seconds�[0m
[2017-04-19T16:34:01+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2017-04-19T16:34:01+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2017-04-19T16:34:01+00:00] ERROR: ohai_hint[aws] (ohai-nativex::awsplugin line 10) had an error: NoMethodError: file[/etc/chef/ohai/hints/aws.json] (/var/chef/cache/cookbooks/ohai/resources/hint.rb line 11) had an error: NoMethodError: undefined method `split' for nil:NilClass
[2017-04-19T16:34:01+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1).```

[Stromweld]

Here's our awsplugin recipe:

ohai_hint 'aws'

ohai 'reload_aws' do
  plugin 'aws'
  action :nothing
end

ohai_plugin 'aws' do
  notifies :reload, 'ohai[reload_aws]', :immediately
end

and here is the plugin file:

Ohai.plugin(:AwsRegion) do
  provides "aws"
  depends "ec2"

  def collect_region
    aws Mash.new
    aws[:region] = ec2[:placement_availability_zone].gsub(/[a-z]$/,'') 
  end

  collect_data do
    aws collect_region[:region]
  end
end

[Stromweld]

Another note, this all worked with chef-client 12.19.36 but after the chef-client 13 was released new amz linux nodes started failing with issues around ohai and it not collecting attributes on first chef-client run when deployed with autoscale script. When knifing a server with same ami image it all works just fine.

[Stromweld]

also for reference here is the autoscale script used to install chef and register node with AWS OpsWorks for chef-automate:

#!/bin/bash

# required settings
NODE_NAME="$(curl --silent --show-error --retry 3 http://169.254.169.254/latest/meta-data/instance-id)" # this uses the EC2 instance ID as the node name
NODE_AZ="$(curl --silent --show-error --retry 3 http://169.254.169.254/latest/meta-data/placement/availability-zone)"
NODE_REGION=${NODE_AZ::-1} # E.g., remove trailing 'b' from us-west-2b
CHEF_SERVER_REGION="us-west-2" # use one of us-east-1, us-west-1, eu-west-1
CHEF_SERVER_NAME="chef-automate" # The name of your Chef Server
CHEF_SERVER_ENDPOINT="chef-automate-example.us-west-2.opsworks-cm.io" # The FQDN of your Chef Server

# optional
CHEF_ORGANIZATION="default"    # AWS OpsWorks for Chef Server always creates the organization "default"
NODE_ENVIRONMENT="base-amz-linux-${APP_ENV}"            # E.g. development, staging, onebox ...
CHEF_CLIENT_VERSION="" # latest if empty

# recommended: upload the chef-client cookbook from the chef supermarket  https://supermarket.chef.io/cookbooks/chef-client
# Use this to apply sensible default settings for your chef-client config like logrotate and running as a service
# you can add more cookbooks in the run list, based on your needs
RUN_LIST="role[base-amz-linux]" # e.g. "recipe[chef-client],recipe[apache2]"

# ---------------------------
set -e -x

CHEF_CA_PATH="/etc/chef/opsworks-cm-ca-2016-root.pem"
CHEF_DIR="/etc/chef"
CHEF_COMMAND="chef-client"
ASSOCIATION_PATH="${CHEF_DIR}/already_associated_when_this_file_present"

aws_cli() {
  aws opsworks-cm --region "${CHEF_SERVER_REGION}" --output text "$@" --server-name "${CHEF_SERVER_NAME}"
}

associate_node() {
    if [ ! -f ${ASSOCIATION_PATH} ]; then
        client_key="${CHEF_DIR}/client.pem"
        mkdir ${CHEF_DIR}
        ( umask 077; openssl genrsa -out "${client_key}" 2048 )

        aws_cli associate-node \
            --node-name "${NODE_NAME}" \
            --engine-attributes \
            "Name=CHEF_ORGANIZATION,Value=${CHEF_ORGANIZATION}" \
            "Name=CHEF_NODE_PUBLIC_KEY,Value='$(openssl rsa -in "${client_key}" -pubout)'"
    fi
}

write_chef_config() {
    config_path="${CHEF_DIR}/client.rb"
    if [ ! -f ${config_path} ]; then
        echo "Writing Chef config..."
        (
            echo "chef_server_url   'https://${CHEF_SERVER_ENDPOINT}/organizations/${CHEF_ORGANIZATION}'"
            echo "node_name         '${NODE_NAME}'"
            echo "ssl_ca_file       '${CHEF_CA_PATH}'"
        ) >> ${config_path}
    fi

    key_path="${CHEF_DIR}/encrypted_data_bag_secret"
    if [ ! -f ${key_path} ]; then
        echo "****Scrubbed****" \
            > ${key_path}
    fi
}

install_chef_client() {
    command -v ${CHEF_COMMAND} >/dev/null 2>&1 || {
    if [ ! $? -eq 0 ]; then
        echo "Installing Chef client..."
        # see: https://docs.chef.io/install_omnibus.html
        curl --silent --show-error --retry 3 --location https://omnitruck.chef.io/install.sh | bash -s -- -v "${CHEF_CLIENT_VERSION}"
        echo "Installed"
    fi
    }
}

install_trusted_certs() {
    if [ ! -f ${CHEF_CA_PATH} ]; then
        echo "Downloading trusted certs"
        curl --silent --show-error --retry 3 --location --output "${CHEF_CA_PATH}" \
            "https://opsworks-cm-${NODE_REGION}-prod-default-assets.s3.amazonaws.com/misc/opsworks-cm-ca-2016-root.pem"
    fi
}

wait_node_associated() {
    if [ ! -f ${ASSOCIATION_PATH} ]; then
        echo "Awaiting association..."
        aws_cli wait node-associated --node-association-status-token "$1"
        echo "Associatied"
        touch ${ASSOCIATION_PATH}
    fi
}

echo "Requesting association" # Can't be in function because it ends up in the output/${node_association_status_token}
node_association_status_token="$(associate_node)"
install_chef_client
write_chef_config
install_trusted_certs
wait_node_associated "${node_association_status_token}"

${CHEF_COMMAND} -r "${RUN_LIST}" -E "${NODE_ENVIRONMENT}"

[tas50]

I'll dig into this further, but you actually don't need this plugin anymore either. Region is an attribute under ec2 now in Chef 13.