sous-chefs / aws Goto Github PK
View Code? Open in Web Editor NEWDevelopment repository for the aws cookbook
Home Page: https://supermarket.chef.io/cookbooks/aws
License: Apache License 2.0
Development repository for the aws cookbook
Home Page: https://supermarket.chef.io/cookbooks/aws
License: Apache License 2.0
Would it be possible to add an attribute to ebs_volume
that sets the deleteOnTermination
flag for volumes created/attached to an instance? Currently we have to go in and manually clean up volumes after instances are pulled out of service.
Hello!
One thing that I noticed, in the updates that I'm making:
Use of class variables like this, say in ec2.rb
line 45, is causing some variable data to be duplicated across interface instances across resources.
def ec2
@@ec2 ||= create_aws_interface(::Aws::EC2::Client)
end
There are far-reaching implications for this, of course - basically if you want to have different resource attributes across the same service that have to do with connectivity data, you are by and large out of luck. It could also be leading to code where items are specified in one resource and were missed in another.
I stumbled upon this as I am working on a PR that, among other things, allows one to run the cookbook using local credentials and to specify region. This facilitates the cookbook to be run in local mode outside of an instance profile, working off your ~/.aws/credentials
file. Since there doesn't seem to be a way to get region out of API though (aside from environment), I have added a resource attribute for that (otherwise the cookbook falls back to instance profile, which doesn't work as I'm not running this within an EC2 instance). I noticed that I hadn't specified it within all resources, and when changed the IAM resources to use instanced-based variables, the resources that didn't have region specified predictably failed trying to grab instance data from http://169.254.169.254/
.
What would the potential impact be to switch all of these items to instance varaibles? Switching the IAM resources to use an instance-based interface (by just altering it in my libraries/iam.rb
) didn't really seem to cause issues, I was able to add the resources without issues.
Another part of my PR is the ability to assign resources to be deployed via assuming IAM roles thru STS (my company has multiple accounts and I want to manage resources centrally) so this I am predicting this being a bit more of an issue.
If some of this is not making sense, which I'm thinking it might not, feel free to look at the commits being made in the https://github.com/paybyphone/aws fork.
Thanks,
--Chris
I am seeing aws_s3_file re-download S3 resources even when the resource hasn't changed. It looks like the s3_file provider signs the URL and then hands it off to remote_file. Since the signed URL is different each time, the CacheControlData key where the request metadata is stored will necessarily change on each Chef run.
I'm writing a recipe that includes the aws default recipe, and then uses the aws_ebs_volume
resource. At runtime, I get the following error:
uninitialized constant Chef::Provider::Package::Rubygems::CurrentGemEnvironment::Bundler
https://gist.github.com/jcderose/0f589b36aad50c05d737
Here's my recipe:
include_recipe "opsworks_initial_setup"
include_recipe 'aws'
instance_az = node[:opsworks][:instance][:availability_zone]
device_id = "/dev/xvdx"
directory '/srv-enc' do
mode '0755'
end
aws_ebs_volume 'EncryptedVolume' do
size 8
device device_id
availability_zone instance_az
volume_type "gp2"
encrypted true
action [:create, :attach]
end
Here's my metadata.rb:
name 'setup-ebs-encryption'
maintainer 'XXXX'
maintainer_email 'XXXX'
license 'All rights reserved'
description 'Installs/Configures setup-ebs-encryption'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version '0.1.0'
depends 'opsworks_initial_setup'
depends 'aws'
Here's my Berksfile:
https://gist.github.com/jcderose/1f054f7714571c25d7b1
I'd like to see a retry parameter so that the Chef run doesn't abort when hitting Aws::EC2::Errors::RequestLimitExceeded
I keep getting an error and I have looked around and cant see what I am doing wrong...
Here is my resource...
include_recipe "aws"
aws_s3_file "/tmp/file_I_want.tar.gz" do
bucket "config-files"
remote_path "file_I_want.tar.gz"
aws_access_key_id node["aws-key-id"]
aws_secret_access_key node["aws-secret"]
end
* aws_s3_file[/tmp/file_I_want.tar.gz] action create
================================================================================
Error executing action `create` on resource 'aws_s3_file[/tmp/file_I_want.tar.gz]'
================================================================================
Errno::ETIMEDOUT
----------------
Connection timed out - connect(2) for "169.254.169.254" port 80
Cookbook Trace:
---------------
/tmp/kitchen/cache/cookbooks/aws/libraries/ec2.rb:83:in `query_instance_availability_zone'
/tmp/kitchen/cache/cookbooks/aws/libraries/ec2.rb:51:in `instance_availability_zone'
/tmp/kitchen/cache/cookbooks/aws/libraries/ec2.rb:63:in `create_aws_interface'
/tmp/kitchen/cache/cookbooks/aws/libraries/s3.rb:9:in `s3'
/tmp/kitchen/cache/cookbooks/aws/providers/s3_file.rb:33:in `do_s3_file'
/tmp/kitchen/cache/cookbooks/aws/providers/s3_file.rb:9:in `block in class_from_file'
Resource Declaration:
Frankfurt requests require you to use V4 api signing which the rightscale aws lib doesn't support.
http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
If your bucket has a dot in the name it returns an ssl error:
OpenSSL::SSL::SSLError
----------------------
hostname "bucket.with.dots.s3.amazonaws.com" does not match the server certificate
See this stack overflow:
http://stackoverflow.com/questions/18340551/amazon-s3-hostname-does-not-match-the-server-certificate-opensslsslsslerr
What is the correct method for specifying a proxy when using this cookbook to attach and ebs volume? http_proxy environment variable doesn't seem to be working, nor does setting it in the chef config file.
I don't think that the version of ruby that is embedded with chef supports those methods anyway.
# grep -i http_proxy /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb
# Note that net/http does not use the HTTP_PROXY environment variable.
# If you want to use a proxy, you must set it explicitly.
Getting the following error when using aws cookbook
NameError: uninitialized constant Chef::Provider::AwsS3File::RightAws
aws_s3_file "demo.py" do
bucket "#{demo_bucket_name}"
remote_path "bootstrap/worker/demo.py"
aws_access_key_id node['aws']['aws_access_key_id']
aws_secret_access_key node['aws']['aws_secret_access_key']
end
I can see that the resource is using the correct data when executing the recipe.
For full details, see chef-boneyard/chef-provisioning-aws#68 .
Hi,
I'd like to use this cookbook to describe (query) AWS tags, but there's no "describe" action.
Actions:
add - Add tags to a resource.
update - Add or modify existing tags on a resource -- this is the default action.
remove - Remove tags from a resource, but only if the specified values match the existing ones.
force_remove - Remove tags from a resource, regardless of their values.
I'm trying to use the secondary_ip
resource, and it is telling me that interface_id
was not found.
This is what happens when I run rubocop on a clean clone of master (064f17e) using the included .rubocop.yml configuration file:
$ rubocop
Inspecting 36 files
.C.WC.CWC.CCWC......................
Offenses:
metadata.rb:9:11: C: Put one space between the method name and the first argument.
source_url "https://github.com/opscode-cookbooks/aws"
^^^^^^^
metadata.rb:9:18: C: Prefer single-quoted strings when you don't need string interpolation or special symbols.
source_url "https://github.com/opscode-cookbooks/aws"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
metadata.rb:10:11: C: Put one space between the method name and the first argument.
issues_url "https://github.com/opscode-cookbooks/aws/issues"
^^^^^^^
metadata.rb:10:18: C: Prefer single-quoted strings when you don't need string interpolation or special symbols.
issues_url "https://github.com/opscode-cookbooks/aws/issues"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
libraries/ec2.rb:31:9: W: end at 31, 8 is not aligned with if at 27, 20
end
^^^
libraries/ec2.rb:43:9: C: Replace class var @@ec2 with a class instance var.
@@ec2 ||= create_aws_interface(::Aws::EC2::Client)
^^^^^
libraries/ec2.rb:47:9: C: Replace class var @@instance_id with a class instance var.
@@instance_id ||= query_instance_id
^^^^^^^^^^^^^
libraries/ec2.rb:51:9: C: Replace class var @@instance_availability_zone with a class instance var.
@@instance_availability_zone ||= query_instance_availability_zone
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
libraries/ec2.rb:76:83: W: Useless assignment to variable - options.
instance_id = open('http://169.254.169.254/latest/meta-data/instance-id', options = { proxy: false }) { |f| f.gets }
^^^^^^^
libraries/ec2.rb:83:106: W: Useless assignment to variable - options.
availability_zone = open('http://169.254.169.254/latest/meta-data/placement/availability-zone/', options = { proxy: false }) { |f| f.gets }
^^^^^^^
libraries/ec2.rb:97:115: W: Useless assignment to variable - options.
ip_addresses = open("http://169.254.169.254/latest/meta-data/network/interfaces/macs/#{mac}/local-ipv4s", options = { proxy: false }) { |f| f.read.split("\n") }
^^^^^^^
libraries/ec2.rb:104:110: W: Useless assignment to variable - options.
eni_id = open("http://169.254.169.254/latest/meta-data/network/interfaces/macs/#{mac}/interface-id", options = { proxy: false }) { |f| f.gets }
^^^^^^^
libraries/elb.rb:9:9: C: Replace class var @@elb with a class instance var.
@@elb ||= create_aws_interface(::Aws::ElasticLoadBalancing::Client)
^^^^^
libraries/s3.rb:9:9: C: Replace class var @@s3 with a class instance var.
@@s3 ||= create_aws_interface(::Aws::S3::Client)
^^^^
providers/ebs_raid.rb:55:7: W: Use Kernel#loop with break rather than begin/end/until(or while).
end while ::File.exist?(base_device)
^^^^^
providers/ebs_raid.rb:67:7: W: Use Kernel#loop with break rather than begin/end/until(or while).
end while ::File.exist?(dir)
^^^^^
providers/ebs_raid.rb:91:65: C: Use %r around regular expression.
node.set[:aws][:raid][mount_point][:raid_dev] = md_device.sub(/\/dev\//, '')
^^^^^^^^^
providers/ebs_raid.rb:108:3: C: Favor modifier unless usage when having a single-line body. Another good alternative is the usage of control flow &&/||.
unless ::File.exist?(mount_point)
^^^^^^
providers/ebs_raid.rb:113:3: C: Favor modifier if usage when having a single-line body. Another good alternative is the usage of control flow &&/||.
if !md_device || md_device == ''
^^
providers/ebs_raid.rb:150:21: C: Avoid parameter lists longer than 5 parameters.
def locate_and_mount(mount_point, mount_point_owner, mount_point_group, mount_point_mode, filesystem, filesystem_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/ebs_raid.rb:187:3: C: Annotation keywords like TODO should be all upper case, followed by a colon, and a space, then a note describing the problem.
# TODO fix this kludge: ideally we'd pull in the device information from the ebs_volume
^^^^^
providers/ebs_raid.rb:259:17: C: Avoid parameter lists longer than 5 parameters.
def mount_device(_raid_dev, mount_point, mount_point_owner, mount_point_group, mount_point_mode, filesystem, filesystem_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/ebs_raid.rb:319:22: C: Avoid parameter lists longer than 5 parameters.
def create_raid_disks(mount_point, mount_point_owner, mount_point_group, mount_point_mode, num_disks, disk_size,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/ebs_raid.rb:391:11: C: Indent when as deep as case.
when 'ext4'
^^^^
providers/ebs_raid.rb:394:15: C: Annotation keywords like TODO should be all upper case, followed by a colon, and a space, then a note describing the problem.
# TODO fill in details on how to format other filesystems here
^^^^^
providers/ebs_volume.rb:95:19: C: Use array literal [] instead of Array.new.
old_snapshots = Array.new
^^^^^^^^^
providers/ebs_volume.rb:157:18: C: Avoid parameter lists longer than 5 parameters.
def create_volume(snapshot_id, size, availability_zone, timeout, volume_type, piops, encrypted, kms_key_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/elastic_lb.rb:6:5: C: Do not use unless with else. Rewrite these with the positive case first.
unless target_lb[:instances].include?(instance_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/instance_monitoring.rb:31:5: C: Rename is_monitoring_enabled to monitoring_enabled?.
def is_monitoring_enabled
^^^^^^^^^^^^^^^^^^^^^
providers/resource_tag.rb:4:3: C: Do not use unless with else. Rewrite these with the positive case first.
unless @new_resource.resource_id
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/resource_tag.rb:11:5: C: Do not use unless with else. Rewrite these with the positive case first.
unless @current_resource.tags.keys.include?(k)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/resource_tag.rb:23:3: C: Do not use unless with else. Rewrite these with the positive case first.
unless @new_resource.resource_id
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/resource_tag.rb:30:3: C: Do not use unless with else. Rewrite these with the positive case first.
unless updated_tags.eql?(@current_resource.tags)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/resource_tag.rb:34:61: W: Ambiguous regexp literal. Parenthesize the method arguments if it's surely a regexp literal, or add a whitespace to the right of the / if it should be a division.
updated_tags.delete_if { |key, _value| key.to_s.match /^aws/ }
^
providers/resource_tag.rb:43:3: C: Do not use unless with else. Rewrite these with the positive case first.
unless @new_resource.resource_id
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/resource_tag.rb:51:18: C: Use next to skip iteration.
tags_to_delete.each do |key|
^^^^
providers/resource_tag.rb:62:3: C: Do not use unless with else. Rewrite these with the positive case first.
unless @new_resource.resource_id
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/resource_tag.rb:81:3: C: Do not use unless with else. Rewrite these with the positive case first.
unless @new_resource.resource_id
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
providers/resource_tag.rb:90:5: C: Block argument expression is not on the same line as the block start.
|tag| @current_resource.tags[tag[:key]] = tag[:value]
^^^^^
providers/s3_file.rb:31:20: C: Use %r around regular expression.
remote_path.sub!(/^\/*/, '')
^^^^^^
providers/s3_file.rb:38:23: C: Use %r around regular expression.
source s3url.gsub(/https:\/\/([\w\.\-]*)\.{1}s3.amazonaws.com:443/, 'https://s3.amazonaws.com:443/\1') # Fix for ssl cert issue
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
36 files inspected, 41 offenses detected
$ rubocop --verbose-version
0.32.0 (using Parser 2.3.0.pre.2, running on ruby 2.0.0 x86_64-darwin12.5.0)
There might have been a breaking change in the aws-sdk with regards to the aws_s3_file
resource. If I run a recipe with 2.6.4 on a node that is not hosted in AWS, it does not converge with the following:
Recipe: go-remote-control::install
* aws_s3_file[/var/cache/rcserver-1.1.0] action create
================================================================================
Error executing action `create` on resource 'aws_s3_file[/var/cache/rcserver-1.1.0]'
================================================================================
Errno::ETIMEDOUT
----------------
Connection timed out - connect(2) for "169.254.169.254" port 80
Cookbook Trace:
---------------
/tmp/kitchen/cache/cookbooks/aws/libraries/ec2.rb:83:in `query_instance_availability_zone'
/tmp/kitchen/cache/cookbooks/aws/libraries/ec2.rb:51:in `instance_availability_zone'
/tmp/kitchen/cache/cookbooks/aws/libraries/ec2.rb:63:in `create_aws_interface'
/tmp/kitchen/cache/cookbooks/aws/libraries/s3.rb:9:in `s3'
/tmp/kitchen/cache/cookbooks/aws/providers/s3_file.rb:33:in `do_s3_file'
/tmp/kitchen/cache/cookbooks/aws/providers/s3_file.rb:9:in `block in class_from_file'
Resource Declaration:
---------------------
# In /tmp/kitchen/cache/cookbooks/go-remote-control/recipes/install.rb
19: aws_s3_file artifact_path do
20: bucket "dnsimple-packages"
21: remote_path "travis-builds/go-remote-control/#{artifact}"
22: aws_access_key_id access_key
23: aws_secret_access_key secrey_key
24: action (Chef::Config[:solo] ? :create_if_missing : :create)
25: not_if { File.exist?(artifact_path) }
26: end
27:
Compiled Resource:
------------------
# Declared in /tmp/kitchen/cache/cookbooks/go-remote-control/recipes/install.rb:19:in `from_file'
As you can see, a small detail in the stack trace above, it appears a lookup is happening somewhere that is trying to use the internal 169.254.169.254
metadata endpoint. If I roll back to the older 2.5.0 version of this cookbook using the same exact provider, it works normally without issue.
Basically what the description says. Newly created volumes become part of the node object, why not store associated snapshots there as well? This would make them much easier to tag for later identification.
An attempt to remove an existing tag with the tag's explicit value fails:
* aws_resource_tag[i-0e4f5d02] action remove
================================================================================
Error executing action `remove` on resource 'aws_resource_tag[i-0e4f5d02]'
================================================================================
ArgumentError
-------------
unexpected value at params[:tags][0]["Tier"]
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/aws/providers/resource_tag.rb:54:in `block (3 levels) in class_from_file'
/var/chef/cache/cookbooks/aws/providers/resource_tag.rb:53:in `block (2 levels) in class_from_file'
/var/chef/cache/cookbooks/aws/providers/resource_tag.rb:51:in `each'
/var/chef/cache/cookbooks/aws/providers/resource_tag.rb:51:in `block in class_from_file'
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/jorhett/recipes/tags.rb:47:in `block in from_file'
aws_resource_tag("i-0e445d02") do
action [:remove]
retries 0
retry_delay 2
cookbook_name "jorhett"
recipe_name "tags"
tags {"Tier"=>"front"}
not_if { #code block }
end
hello, i just downloaded the latest version (2.5.0)
I'm trying to create and attach a 1TB gp2 volume with 3000 provisioned IOPS and can't seem to get past the following error:
[2015-01-16T17:44:08-05:00] ERROR: aws_ebs_volume[data_ebs_volume](cookbook::createEBSdatavolume line 25) had an error: RuntimeError: IOPS param without piops volume type.
[2015-01-16T17:44:08-05:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
here is output from chef.
Resource Declaration:
In /var/chef/cache/cookbooks/cookbook/recipes/createEBSdatavolume.rb
25: aws_ebs_volume "data_ebs_volume" do
26: provider "aws_ebs_volume"
27: #availability_zone "us-east-1a"
28: aws_access_key aws['access_key']
29: aws_secret_access_key aws['secret_key']
30: size ebs_size
31: piops piops
32: volume_type 'gp2'
33: device "#{device_id}"
34: action [ :create, :attach ]
35: end
36:
Compiled Resource:
Declared in /var/chef/cache/cookbooks/cookbook/recipes/createEBSdatavolume.rb:25:in `from_file'
aws_ebs_volume("data_ebs_volume") do
provider Chef::Provider::AwsEbsVolume
action [:create, :attach]
retries 0
retry_delay 2
cookbook_name "wordstream-postgresql"
recipe_name "createEBSdatavolume"
aws_access_key "XXXXXXXXXXXXXXXXXXX"
aws_secret_access_key "XXXXXXXXXXXXXXXXXXXXXX"
size 1024
volume_type "gp2"
device "/dev/sdc"
piops 3000
timeout 180
end
Is gp2 not a valid volume_type? if it is valid, i'm I missing a key component?
thanks
Austin
Hi,
I have a recipe that uses the ebs_volume LWRP but on some of my nodes, the the volume_id attribute exists on some nodes, but on others it doesn't and I get the following error. All the machines in question have been provisioned the same way, using knife ec2 and are of the same role.
Please be advised that I have change the attribute name because of the way I'm loading the LWRP. Again, it works on some, but not on others. My question is, how do I get the attribute to populate on every node?
================================================================================
Error executing action update
on resource 'aws_resource_tag[my ebs volume]'
================================================================================
Chef::Exceptions::ValidationFailed
----------------------------------
Option resource_id's value my ebs volume does not match regular expression /(i|snap|vol)-[a-zA-Z0-9]+/
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/aws/providers/resource_tag.rb:83:in `load_current_resource'
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/vungle/recipes/ebs_volume.rb
62: aws_resource_tag 'my ebs volume' do
63: aws_access_key node[:vungle][:aws][:aws_access_key_id]
64: aws_secret_access_key node[:vungle][:aws][:aws_secret_access_key]
65: resource_id lazy { node[:aws][:ebs_volume][:data_volume][:volume_id] }
66: tags({"Name" => node[:machinename]})
67: end
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/vungle/recipes/ebs_volume.rb:62:in `from_file'
aws_resource_tag("my ebs volume") do
action :update
retries 0
retry_delay 2
guard_interpreter :default
cookbook_name "vungle"
recipe_name "ebs_volume"
aws_access_key "fake_key"
aws_secret_access_key "secret access"
resource_id #<Chef::DelayedEvaluator:0x00000003f1be10@/var/chef/cache/cookbooks/vungle/recipes/ebs_volume.rb:65>
tags {"Name"=>"machine01"}
end
I'm trying to add instances to an ELB, but it doesn't seem to work in a VPC.
================================================================================
Error executing action `register` on resource 'aws_elastic_lb[test-mysql]'
RightAws::AwsError
ValidationError: ELB in VPC is not supported in this version of API. Please try 2011-11-15 or newer.
The documentation shows a sample policy to be able to create tags. The sample policy is missing the "ec2:DescribeTags" action permission, which is needed in order to use the "aws_resource_tag" resource without having a UnauthorizedOperation error.
By the way, you should update the repository guidelines for contributing, which says "If you would like to contribute, please open a ticket in JIRA", while JIRA says "We've moved issue tracking from JIRA to Github Issues". Very confusing... ;)
I came across this blog post: http://clarkdave.net/2013/04/managing-ebs-volumes-with-chef/
After using the aws_ebs_volume LWRP there's a ruby_block to do a sleep loop until the device is ready. Is that a necessary step? If so, any reason not have it baked into the aws cookbook?
(I asked this on the mailing list but didn't get any nibbles so I thought I'd try here)
Hi,
Using the aws cookbook, I am trying to create a volume and attaching it using the provider, ebs_volume.rb. While doing this, I am getting the error "Volume no longer exists".
[2014-06-26T07:43:17-04:00] ERROR: aws_ebs_volume[new_ebs_volume](aws::default line 30) had an error: RuntimeError: Volume no longer exists
[2014-06-26T07:43:17-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
The fix provided on resolved issue is already there in the provider, but not helping.
Let me know if you need any further information to resolve this.
Early help is much appreciated as I have very less time to complete this task. Thank you.
I believe this has been broken since 2a183fd.
The aws-sdk
gem introduced response paging in v2 and this requires the user to extract the underlying collections (via method invocation) from the response pages. The current implementation does not account for that API shift.
I have tested this use aws-sdk
v2.0.22 (attribute version) and v2.0.38 (latest at the time of this writing).
I have a static website that gets deployed on S3/CloudFront. What I would like to see are resources for managing this configuration (just because you're not using nodes, doesn't mean it's not configuration management), where we could make stuff like:
aws_cloudfront_distribution do
aliases ['example.com']
origins [{ domain_name: '...', http_port: '...', ... }]
error_pages [...]
action :create
end
aws_iam_user 'my-iam-deploy-user' do
policies [...]
action :create
end
aws_s3_bucket 'where-my-stuff-goes' do
policy({...})
website_configuration({...})
end
and run that recipe on deploy to ensure all the things that are needed for the site to be deployed are present and configured correctly.
It looks like this cookbook currently uses right-aws, which supports the resources that are already here, but not much else. Doing something like this might require using Fog or the AWS SDK gem.
Getting this error with the latest:
NameError: no member 'zone' in struct
C:\chef\b45ead9638cca100d4f0d0f39ff0ca8d\cookbooks\aws\providers\ebs_volume.rb:152:in `[]'
Looks like the the struct returned by AWS is now :availablity_zone
, not :zone
Anybody else seeing this? I can create a pull request.
Error : aws_ebs_volume[sdi1](/var/chef/cache/cookbooks/aws/providers/ebs_raid.rb line 337) had an error: Aws::EC2::Errors::InvalidParameterValue: Value (/dev/sdi1) for parameter device is invalid. /dev/sdi1 is not a valid EBS device name.
Issue: /dev/sdi1 is invalid, /dev/sdi is not.
Details:
With mount_point set, ebs_raid takes the default from aws_ebs_volume:
aws_ebs_volume disk_dev_path do
device "/dev/#{disk_dev_path}"
devices[disk_dev_path] = 'pending'
Chef::Log.info("creating ebs volume for device #{disk_dev_path} with size #{disk_size}")
Hello,
We stand up dynamic clusters that have config files we want to place on S3, so that they can be read and registered by another service.
I see the aws_s3_file resource for pulling files from S3. But what about pushing files from a node to a bucket?
Regards,
Joe Reid
Chef::Log.info("Format device found: #{md_device}")
case filesystem
when "ext4"
system("mke2fs -t #{filesystem} -F #{md_device}")
when "xfs"
system("mkfs.xfs #{md_device}")
else
#TODO fill in details on how to format other filesystems here
Chef::Log.info("Can't format filesystem #{filesystem}")
end
Would it be possible to add IAM role support to the aws_s3_file provider? It currently requires aws_access_key_id and aws_secret_access_key be set.
Thanks for consideration.
Hey,
the error is produced by this cookbook code
aws_ebs_raid "mysql_raid10_volume" do
disk_count 4
disk_size 250
level 10
filesystem 'ext4'
disk_type 'standard'
end
it gets compiled to:
aws_ebs_volume("sdi1") do
provider Chef::Provider::AwsEbsVolume
action [:create, :attach]
retries 0
retry_delay 2
guard_interpreter :default
cookbook_name "mysql"
size 250
volume_type "standard"
piops 0
device "/dev/sdi1"
timeout 180
end
If i run the compiled code in the same cookbook, the volume gets build and attached. However, the ebs_raid.rb provider seems to be the problem.
Has anybody seen this before, or can reproduce this?
Any advice on how to further troubleshoot the issue?
The elastic_lb LWRP runs a check to see if the node is currently in the ELB, this check does not currently work.
Lines:
providers/elastic_lb:6
providers/elastic_lb:18
providers/elastic_lb:20
Current Behavior:
When actions de/register run, the checks listed above in lines 6 and 18 always fail. So it will always try to add the node to the ELB with register and will never remove a node with deregister. converge_by will always claim that the action ran successfully.
The deregister_instances_with_load_balancer method does not exist either. The correct method is deregister_instances_from_load_balancer
So if the deregister action were to run, it would fail when trying to run this method on line 20.
EBS Volumes in the AWS Console have a "name" attribute, but the ebs_volume resource doesn't allow it to be set. This makes it hard ot keep track of the volumes in our account.
The "name" attribute should be added to the ebs_volume resource.
Running version 2.6.2 with Chef 11.16.4 on Ubuntu 14.04.1
================================================================================
Recipe Compile Error in /var/chef/cache/cookbooks/go-remote-control/recipes/default.rb
================================================================================
ArgumentError
-------------
wrong number of arguments (1 for 0)
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/aws/recipes/default.rb:22:in `block in from_file'
/var/chef/cache/cookbooks/aws/recipes/default.rb:20:in `from_file'
/var/chef/cache/cookbooks/go-remote-control/recipes/install.rb:18:in `from_file'
/var/chef/cache/cookbooks/go-remote-control/recipes/default.rb:10:in `from_file'
Relevant File Content:
----------------------
/var/chef/cache/cookbooks/aws/recipes/default.rb:
15: # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16: # See the License for the specific language governing permissions and
17: # limitations under the License.
18: #
19:
20: chef_gem 'aws-sdk' do
21: version node['aws']['aws_sdk_version']
22>> compile_time true if respond_to?(:compile_time)
23: action :install
24: end
25:
26: require 'aws-sdk'
27:
EBS Raid creation is failing with:
Error "Aws::EC2::Errors::MissingParameter The request must contain the parameter size/snapshot"
This is due to:
provider/ebs_raid.rb :349
snapshot_id creating_from_snapshot ? snapshots[i - 1] : ''
I guess AWS SDK doesn't like '' as an empty snapshot_id. I changed the line to:
snapshot_id creating_from_snapshot ? snapshots[i - 1] : nil
nil is working fine in my fork.
Also,
EBS Volume was updated last week to support Encrypted volumes, but EBS Raid lacks the passthrough to allow creation of Encrypted volumes when creating them as part of a Raid.
I'm attempting to use the aws_ebs_volume resource to attach an ebs volume to an opsworks instance at boot. However, when the custom recipe runs I get the following message:
================================================================================
Error executing action `create` on resource 'aws_ebs_volume[db_ebs_volume]'
================================================================================
RuntimeError
------------
Volume no longer exists
Cookbook Trace:
---------------
/var/lib/aws/opsworks/cache/cookbooks/aws/providers/ebs_volume.rb:189:in `block in create_volume'
/var/lib/aws/opsworks/cache/cookbooks/aws/providers/ebs_volume.rb:177:in `create_volume'
/var/lib/aws/opsworks/cache/cookbooks/aws/providers/ebs_volume.rb:38:in `block (2 levels) in class_from_file'
/var/lib/aws/opsworks/cache/cookbooks/aws/providers/ebs_volume.rb:37:in `block in class_from_file'
Resource Declaration:
---------------------
# In /var/lib/aws/opsworks/cache/cookbooks/mongodb/recipes/ebs_backups.rb
60: aws_ebs_volume "db_ebs_volume" do
61: size 7
62: device "/dev/sdi"
63: action [ :create, :attach ]
64: end
Compiled Resource:
------------------
# Declared in /var/lib/aws/opsworks/cache/cookbooks/mongodb/recipes/ebs_backups.rb:60:in `from_file'
aws_ebs_volume("db_ebs_volume") do
action [:create, :attach]
retries 0
retry_delay 2
cookbook_name "mongodb"
recipe_name "ebs_backups"
size 7
device "/dev/sdi"
timeout 180
volume_type "standard"
piops 0
end
[2014-06-09T16:51:52+00:00] INFO: Running queued delayed notifications before re-raising exception
[2014-06-09T16:51:52+00:00] INFO: template[/etc/default/mongodb] sending restart action to service[mongodb] (delayed)
[2014-06-09T16:51:52+00:00] INFO: Processing service[mongodb] action restart (mongodb::default line 127)
[2014-06-09T16:52:04+00:00] INFO: service[mongodb] restarted
[2014-06-09T16:52:04+00:00] ERROR: Running exception handlers
[2014-06-09T16:52:04+00:00] ERROR: Exception handlers complete
[2014-06-09T16:52:04+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache/chef-stacktrace.out
[2014-06-09T16:52:04+00:00] ERROR: aws_ebs_volume[db_ebs_volume] (mongodb::ebs_backups line 60) had an error: RuntimeError: Volume no longer exists
[2014-06-09T16:52:05+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Steps to reproduce:
include_recipe "aws"
aws_ebs_volume "db_ebs_volume" do
size 7
device "/dev/sdi"
action [ :create, :attach ]
end
I believe these recipe's are supposed to work with chef-solo but I haven't been able to use this cookbook to create a volume and attach it to my opsworks instance. Is this a bug? Am I doing something wrong? Any help appreciated! Thank you!
Got the following error when trying to use aws cookbook in ec2-central-1 region.
ec2-54-93-80-34.eu-central-1.compute.amazonaws.com [2015-01-21T12:12:19+00:00] WARN: Rightscale::HttpConnection : re-raising same error: https://eu-central-1.ec2.amazonaws.com:443 temporarily unavailable: (SocketError: getaddrinfo: Name or service not known) -- error count: 4, error age: 0
Looks like the endpoint is different for eu-central-1 vs other regions.
$ host ec2.eu-central-1.amazonaws.com
ec2.eu-central-1.amazonaws.com has address 54.239.54.44
$ host eu-central-1.ec2.amazonaws.com
Host eu-central-1.ec2.amazonaws.com not found: 3(NXDOMAIN)
$ host eu-west-1.ec2.amazonaws.com
eu-west-1.ec2.amazonaws.com has address 178.236.6.52
$ host ec2.eu-west-1.amazonaws.com
ec2.eu-west-1.amazonaws.com is an alias for eu-west-1.ec2.amazonaws.com.
eu-west-1.ec2.amazonaws.com has address 178.236.4.54
I wrote this up as part of #129. The other part of #129 has been closed, so I'm opening a new issue just on this part.
PR #114 brought in the ability to create Encrypted EBS volumes, however it didn't provide a passthrough for volumes created as part of a raid to be created with encryption.
I wrote PR #130 to fix this right after PR #114 was merged, but it included a general fix for EBS Raids that was sliced out into its own PR #156, which was recently merged. So I am submitting a new PR with just the remaining Encryption fix for volumes created in a Raid.
Hi. I've added a testing integration testing skeleton to this cookbook with this commit:
Can somebody please add some recipes in that fixture cookbook that exercise the resource in this cookbook?
The following is output from a system trying to converge an aws_s3_file
provider resource from a recipe like this. The chef.sean.horn.test.bucket
bucket was originally created in the us-west-2 region, while the instance trying to converge the recipe lies in the eu-west-1 region. Currently the aws_s3_file
will not follow redirects, although it should, to meet the spec at http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro
Also, bucket names can have dots, so we should prefer path-based, rather than virtual-host-based URLs to avoid ssl issues due to unmatched certs.
testjunk/recipes/default.rb
include_recipe "aws"
aws = data_bag_item("aws", "main")
aws_s3_file "/tmp/foo" do
bucket "chef.sean.horn.test.bucket"
remote_path "yepit.json"
aws_access_key_id aws['aws_access_key_id']
aws_secret_access_key aws['aws_secret_access_key']
end
[2015-03-24T20:05:25+00:00] DEBUG: Instance's availability zone is eu-west-1c
* remote_file[/tmp/foo] action create[2015-03-24T20:05:25+00:00] INFO: Processing remote_file[/tmp/foo] action create (/var/chef/cache/cookbooks/aws/providers/s3_file.rb line 36)
[2015-03-24T20:05:25+00:00] DEBUG: remote_file[/tmp/foo] checking for changes
[2015-03-24T20:05:25+00:00] DEBUG: Cache control headers: {}
[2015-03-24T20:05:25+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request
[2015-03-24T20:05:25+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request
[2015-03-24T20:05:25+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_request
[2015-03-24T20:05:25+00:00] DEBUG: Initiating GET to https://s3-eu-west-1.amazonaws.com/chef.sean.horn.test.bucket/yepit.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJUYAZ6NNDKZCUEJA%2F20150324%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20150324T200525Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=532a8e121b9d05d3229f90d593053d944c2253699be877b9547366024c49efef
[2015-03-24T20:05:25+00:00] DEBUG: ---- HTTP Request Header Data: ----
[2015-03-24T20:05:25+00:00] DEBUG: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
[2015-03-24T20:05:25+00:00] DEBUG: ---- End HTTP Request Header Data ----
[2015-03-24T20:05:25+00:00] DEBUG: ---- HTTP Status and Header Data: ----
[2015-03-24T20:05:25+00:00] DEBUG: HTTP 1.1 301 Moved Permanently
[2015-03-24T20:05:25+00:00] DEBUG: x-amz-request-id: D66DC67D58AB6B7F
[2015-03-24T20:05:25+00:00] DEBUG: x-amz-id-2: zmy89OOz1JD+2sWP9egkhuXaoRscw51zkUtkXmlB0YMuBYjJUagNBlPHPo5i5lWeWU8EwBohNCQ=
[2015-03-24T20:05:25+00:00] DEBUG: content-type: application/xml
[2015-03-24T20:05:25+00:00] DEBUG: transfer-encoding: chunked
[2015-03-24T20:05:25+00:00] DEBUG: date: Tue, 24 Mar 2015 20:05:25 GMT
[2015-03-24T20:05:25+00:00] DEBUG: server: AmazonS3
[2015-03-24T20:05:25+00:00] DEBUG: connection: close
[2015-03-24T20:05:25+00:00] DEBUG: ---- End HTTP Status/Header Data ----
[2015-03-24T20:05:25+00:00] DEBUG: ---- HTTP Response Body ----
[2015-03-24T20:05:25+00:00] DEBUG: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>chef.sean.horn.test.bucket</Bucket><Endpoint>chef.sean.horn.test.bucket.s3-us-west-2.amazonaws.com</Endpoint><RequestId>D66DC67D58AB6B7F</RequestId><HostId>zmy89OOz1JD+2sWP9egkhuXaoRscw51zkUtkXmlB0YMuBYjJUagNBlPHPo5i5lWeWU8EwBohNCQ=</HostId></Error>
It would be great to be able to pass in an encryption_key
parameter to an aws_s3_file
resource to retrieve encrypted files. The AWS Ruby SDK has great support for client side encryption/decryption of S3 objects (http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Encryption/Client.html). Unfortunately, because of the way that aws_s3_file
creates files (inline remote_file with a presigned url as the source), it can't support using Aws::S3::Encryption::Client
. So, since this will require a pretty serious rewrite, I wanted to see if there was community support for the idea before proceeding. Thoughts?
Per the AWS docs:
"Hardware virtual machine (HVM) AMIs (such as the base Windows and Cluster Compute images) do not support the use of trailing numbers on device names (xvd[a-p][1-15])."
The RAID recipe compiles the disk by making device names like sdi1, sdi2, etc., so I get an error like this when trying to create the RAID:
"InvalidParameterValue: Value (/dev/sdi1) for parameter device is invalid. /dev/sdi1 is not a valid EBS device name."
This cookbook works with Linux but doesn't seem to work properly with Windows. I've tried creating EBS volumes and EC2 tags, and no matter what I do the Chef client run always produces the following SSL error:
Seahorse::Client::NetworkingError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
I've tried updating the CA bundle certificate, Ruby, Ruby gems, and I've also tried updating aws-sdk to 2.0.48 and nothing seems to fix the issue.
Has anyone else been able to update make this cookbook work on Windows servers?
it would be nice to set EBS volume encryption.
Source URL Issues URL allow for supermarket to automatically show the "show issues and show source" when looking around at cookbooks. Without these in the metadata, these have to be handcrafted.
Expected format
source_url ""
issues_url ""
Contributing has some issues referring to retired wiki.
I only see this error in the chef-client logs. Running manually via "chef-client" does not reproduce this error. However, firing off a run via /etc/init.d/chef-client run does produce the error, visible in the logs.
[2014-10-03T20:45:22+00:00] ERROR: Running exception handlers
[2014-10-03T20:45:22+00:00] ERROR: Creating JSON exception report
[2014-10-03T20:45:23+00:00] ERROR: Exception handlers complete
[2014-10-03T20:45:23+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2014-10-03T20:45:23+00:00] ERROR: aws_ebs_volume[data_volume](drbd-ebs::ha line 36) had an error: RuntimeError: Volume with id vol-a8fda8ed is registered with the node but does not exist in EC2. To clear this error, remove the ['aws']['ebs_volume']['data_volume']['volume_id'] entry from this node's data.
[2014-10-03T20:45:23+00:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
chef-client -v
Chef: 11.14.0.alpha.4
root@use1d-admin101a:/var/log/chef# curl http://169.254.169.254/latest/meta-data/block-device-mapping/ebs19/ && echo
xvdb
root@use1d-admin101a:/var/log/chef# mount
/dev/xvda on / type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)
/dev/drbd0 on /data type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)
root@use1d-admin101a:/var/log/chef# cat /etc/drbd.d/pair.res
resource pair {
device /dev/drbd0;
disk /dev/xvdb;
}
root@use1d-admin101a:/var/log/chef# cat /proc/drbd
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:688215440 nr:0 dw:744654652 dr:2126176838 al:15406293 bm:141869 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
the changes in 2.6.1 from [https://github.com/opscode-cookbooks/aws/commit/2fc04a4f5e45494da6e27a5246789d8b17797cf2] Intended to suppress the warnings from the newer compile_time resource on chef_gem break the cookbook if you are not using chef 12.1 (which isn't out yet anyway).
This fixes #106
pinging @someara who has made a few of these changes recently ;)
================================================================================
Error executing action `attach` on resource 'aws_ebs_volume[sdi1]'
================================================================================
Aws::EC2::Errors::InvalidParameterValue
---------------------------------------
Value (/dev/sdi1) for parameter device is invalid. /dev/sdi1 is not a valid EBS device name.
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:212:in `attach_volume'
/var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:70:in `block (2 levels) in class_from_file'
/var/chef/cache/cookbooks/aws/providers/ebs_volume.rb:68:in `block in class_from_file'
Today our deployment start failing and the reason seems to be the recently released 2.6.0 version.
The error in aws opsworks log is this:
================================================================================
Error executing action `update` on resource 'aws_resource_tag[i-540be0b9]'
================================================================================
NameError
---------
uninitialized constant Aws
Cookbook Trace:
---------------
/var/lib/aws/opsworks/cache.stage2/cookbooks/aws/libraries/ec2.rb:43:in `ec2'
/var/lib/aws/opsworks/cache.stage2/cookbooks/aws/providers/resource_tag.rb:89:in `load_current_resource'
I see the recent change in the line mentioned in the error text.
Reverting to version 2.5.0 fixed the issue with our deployment.
Update:
Just for the case here is a custom recipe code:
include_recipe "aws"
tags_data = {
"MakeSnapshot" => "True"
}
aws_resource_tag node['ec2']['instance_id'] do
tags(tags_data)
action :update
end
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.