fog/fog-aws

Module for the 'fog' gem to support Amazon Web Services http://aws.amazon.com/

fog-aws's Issues

IAM authentication not compatible with GovCloud

In https://github.com/fog/fog-aws/blob/master/lib/fog/aws/iam.rb#L233, the IAM implementation assumes the region us us-east-1 for all requets.

#global services that have no region are signed with the us-east-1 region
@signer = Fog::AWS::SignatureV4.new( @aws_access_key_id, @aws_secret_access_key,'us-east-1','iam')

This is not true for AWS GovCloud, the government-specific AWS region.

If the host is set as iam.us-gov.amazonaws.com, a request for Fog::AWS::IAM#roles will return:

Credential should be scoped to a valid region, not 'us-east-1'.

Instead, I'd suggest that the IAM constructor accept a region argument (to mirror other constructors), which defaults to us-east-1, to allow for better GovCloud support.

AWS security group tests have become unstable

Original issue at: fog/fog#2932

I've been so busy travelling to the fog summit I missed that tests/aws/requests/compute/security_group_tests.rb has been failing randomly on different Travis settings for a while.

https://travis-ci.org/fog/fog/jobs/25614603#L1105 is one of many failures.

It seems unstable enough that every build is failing and what I have noticed (but didn't look at further) is lots of people going "Hey it's not my changes!" about something so this hasn't been looked at.

Needs fixing or I'll disable the test.

CircleCI failing

I get the following output on cirlceci

export RAILS_ENV="test"
export RACK_ENV="test"
bundle exec rake db:create db:schema:load --trace

** Invoke db:create (first_time)
** Invoke db:load_config (first_time)
** Execute db:load_config
** Execute db:create
** Invoke db:schema:load (first_time)
** Invoke environment (first_time)
** Execute environment
rake aborted!
TypeError: no implicit conversion of Bignum into String
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/fog-aws-0.1.1/lib/fog/aws/signaturev4.rb:12:in `+'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/fog-aws-0.1.1/lib/fog/aws/signaturev4.rb:12:in `initialize'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:509:in `new'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:509:in `setup_credentials'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/fog-aws-0.1.1/lib/fog/aws/storage.rb:492:in `initialize'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/fog-core-1.29.0/lib/fog/core/service.rb:115:in `new'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/fog-core-1.29.0/lib/fog/core/service.rb:115:in `new'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/fog-core-1.29.0/lib/fog/storage.rb:25:in `new'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:83:in `eager_load_fog'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:96:in `fog_credentials='
/home/ubuntu/hermes_dragon/config/initializers/carrierwave.rb:2:in `block in <top (required)>'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:118:in `configure'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/carrierwave-0.10.0/lib/carrierwave.rb:14:in `configure'
/home/ubuntu/hermes_dragon/config/initializers/carrierwave.rb:1:in `<top (required)>'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:268:in `load'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:268:in `block in load'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:240:in `load_dependency'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:268:in `load'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/engine.rb:652:in `block in load_config_initializer'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/notifications.rb:166:in `instrument'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/engine.rb:651:in `load_config_initializer'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/engine.rb:615:in `each'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/engine.rb:615:in `block in <class:Engine>'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/initializable.rb:30:in `instance_exec'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/initializable.rb:30:in `run'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/initializable.rb:55:in `block in run_initializers'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:226:in `block in tsort_each'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:348:in `block (2 levels) in each_strongly_connected_component'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:420:in `block (2 levels) in each_strongly_connected_component_from'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:429:in `each_strongly_connected_component_from'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:419:in `block in each_strongly_connected_component_from'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/initializable.rb:44:in `each'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/initializable.rb:44:in `tsort_each_child'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:413:in `call'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:413:in `each_strongly_connected_component_from'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:347:in `block in each_strongly_connected_component'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:345:in `each'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:345:in `call'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:345:in `each_strongly_connected_component'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:224:in `tsort_each'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/tsort.rb:203:in `tsort_each'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/initializable.rb:54:in `run_initializers'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/application.rb:352:in `initialize!'
/home/ubuntu/hermes_dragon/config/environment.rb:5:in `<top (required)>'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:274:in `require'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:274:in `block in require'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:240:in `load_dependency'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:274:in `require'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/application.rb:328:in `require_environment!'
/home/ubuntu/hermes_dragon/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/application.rb:443:in `block in run_tasks_blocks'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:240:in `call'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:240:in `block in execute'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:235:in `each'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:235:in `execute'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:179:in `block in invoke_with_call_chain'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:172:in `invoke_with_call_chain'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:201:in `block in invoke_prerequisites'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:199:in `each'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:199:in `invoke_prerequisites'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:178:in `block in invoke_with_call_chain'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:172:in `invoke_with_call_chain'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/task.rb:165:in `invoke'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/application.rb:150:in `invoke_task'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/application.rb:106:in `block (2 levels) in top_level'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/application.rb:106:in `each'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/application.rb:106:in `block in top_level'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/application.rb:115:in `run_with_threads'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/application.rb:100:in `top_level'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/application.rb:78:in `block in run'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/application.rb:176:in `standard_exception_handling'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rake/application.rb:75:in `run'
/home/ubuntu/.rvm/rubies/ruby-2.2.0/bin/rake:33:in `<main>'

export RAILS_ENV="test"
export RACK_ENV="test"
bundle exec rake db:create db:schema:load --trace
 returned exit code 1

Tasks: TOP => db:schema:load => environment Action failed: bundle exec rake db:create db:schema:load --trace

let me know if there's anything else I can share to narrow down the issue.

Enabling termination protection

I am trying to enable termination protection of instance while launching. I am using {:disable_api_termination => true } as a parameter to servers.create method. But that does not enable the termination protection. Am I doing something wrong or it is not supported?

Creating VPC instances in AWS

While using Fog::Compute[:aws].servers.bootstrap I cannot create an EC2 instance inside a VPC. To create instance one has to specify security_group_id and subnet_id. bootstrap method does not allow this.

AWS Credentials required when using IAM Profile

Hi,

i am trying to access S3 from an EC2 Instance with Fog using only the IAM Profile but not the AWS Credentials (secret_key_id and secret_access_key are not available in the environment). However the credentials seem to be required even when using IAM. Should this not be possible? I expected that Fog would fetch the credentials for me when using IAM.

Thanks for your support!

Doesn't work after upgrading to 0.1.2

I'm getting error:

rake aborted!
NameError: uninitialized constant Fog::Service
ruby/2.2.0/gems/fog-aws-0.1.2/lib/fog/aws/storage.rb:3:in `<module:Storage>'
ruby/2.2.0/gems/fog-aws-0.1.2/lib/fog/aws/storage.rb:2:in `<module:Fog>'
ruby/2.2.0/gems/fog-aws-0.1.2/lib/fog/aws/storage.rb:1:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.2.0@global/gems/bundler-1.7.9/lib/bundler/runtime.rb:76:in `require'
/usr/local/rvm/gems/ruby-2.2.0@global/gems/bundler-1.7.9/lib/bundler/runtime.rb:76:in `block (2 levels) in require'
/usr/local/rvm/gems/ruby-2.2.0@global/gems/bundler-1.7.9/lib/bundler/runtime.rb:72:in `each'
/usr/local/rvm/gems/ruby-2.2.0@global/gems/bundler-1.7.9/lib/bundler/runtime.rb:72:in `block in require'
/usr/local/rvm/gems/ruby-2.2.0@global/gems/bundler-1.7.9/lib/bundler/runtime.rb:61:in `each'
/usr/local/rvm/gems/ruby-2.2.0@global/gems/bundler-1.7.9/lib/bundler/runtime.rb:61:in `require'
/usr/local/rvm/gems/ruby-2.2.0@global/gems/bundler-1.7.9/lib/bundler.rb:134:in `require'
releases/20150408132737/config/application.rb:7:in `<top (required)>'
releases/20150408132737/Rakefile:4:in `require'
releases/20150408132737/Rakefile:4:in `<top (required)>'
(See full trace by running task with --trace)
---- End output of bash -c 'source /etc/profile.d/rvm.sh && bundle exec rake assets:precompile' ----

I'm using https://github.com/rumblelabs/asset_sync gem - not sure if this issue should be addressed there or here.

Still empty content-encoding when it is not set

We are experiencing empty content-type with v0.4.0 when using with middleman-s3_sync needed to downgrade all the way to 0.1.2 to fix it.

Cannot saving auto scaling group

I'm having trouble updating an auto scaling group. It seems to be trying to assign a serialized version of the tags list as a tag, and hitting the tag length limit.

The error I'm getting is:

Value '{"ResourceId"=>"xxxxxxxxxxx-AutoScalingGroup-9O3T0B4Z0E26", "PropagateAtLaunch"=>true, "Value"=>"xxxxxxxxxxx-Z5ZZTKPWA7WO", "Key"=>"aws:cloudformation:stack-name", "ResourceType"=>"auto-scaling-group"}' at 'tags.6.member.key' failed to satisfy constraint: Member must have length less than or equal to 128

Add support for EC2 Container Service (ECS)

Hi,

I'm opening this issue to avoid effort duplication just in case somebody else plans or is already working on this.

I need support for ECS, so I'm implementing this in fog-aws.

Cheers,

NoMethodError - undefined method `signature_parameters' for nil:NilClass

We are having problems when uploading files to aws s3 storage using carrierwave and fog with the setting aws_signature_version set to 2.

Specifically when we try to read the url of the downloaded file e.g.

class Attachment < ActiveRecord::Base
  mount_uploader :file, CarrierWave::Uploader::Base
end
attachment.file.url

Carrierwave config

  CarrierWave.configure do |config|
    config.storage = :fog
    config.fog_credentials = {
      :provider => 'AWS',
      :aws_access_key_id => '',
      :aws_secret_access_key => '',
      :aws_signature_version => 2, 
      :host => '',
    }
    config.fog_public = false
    config.fog_attributes = { 'Cache-Control' => "max-age=#{365.day.to_i}" }
  end

Following is the backtrace:

NoMethodError - undefined method `signature_parameters' for nil:NilClass:
  fog-aws (0.0.6) lib/fog/aws/storage.rb:160:in `signed_url'
  fog-aws (0.0.6) lib/fog/aws/requests/storage/get_object_url.rb:12:in `get_object_url'
  fog-aws (0.0.6) lib/fog/aws/models/storage/files.rb:82:in `get_url'
  fog-aws (0.0.6) lib/fog/aws/models/storage/file.rb:227:in `url'
  carrierwave (0.10.0) lib/carrierwave/storage/fog.rb:145:in `authenticated_url'
  carrierwave (0.10.0) lib/carrierwave/storage/fog.rb:325:in `url'
  carrierwave (0.10.0) lib/carrierwave/uploader/url.rb:20:in `url'
  carrierwave (0.10.0) lib/carrierwave/uploader/versions.rb:207:in `url'
  carrierwave (0.10.0) lib/carrierwave/uploader/default_url.rb:8:in `url'

Is anyone going to add support for AWS ElasticTranscoder

Original issue at: fog/fog#2875

Or maybe it's done already somewhere. Thanks for the great work on this Gem!

Getting SignatureDoesNotMatch error with eu-central-1

When using fog via paperclip with the following configuration:

config.paperclip_defaults = {
  :storage => :fog,
  :fog_credentials => {
    :provider => 'AWS',
    :aws_access_key_id => ENV['AWS_ACCESS_KEY_ID'],
    :aws_secret_access_key => ENV['AWS_SECRET_ACCESS_KEY'],
    :region => 'eu-central-1'
  },
  :fog_directory => ENV['FOG_DIRECTORY']
}

Access to S3 fails with the following error:

Excon::Errors::Forbidden: Expected(200) <=> Actual(403 Forbidden)
SignatureDoesNotMatch - The request signature we calculated does not match the signature you provided. Check your key and signing method.

Logging directly with the awscli tools using the same credentials and setting the same region works. I double checked the keys. Also, aws s3api get-bucket-location --bucket mybucket returns eu-central-1.

Change architecture attribute in AWS::Compute::Server model

Original issue at: fog/fog#3289

change :architecture from attr_accessor to attribute in lib/fog/aws/models/compute/server.rb

How to determine the disableApiTermination attribute value

How do I determine whether API termination is disbaled or enabled using fog. The disable_api_termination atribute of the fog instance object seems to be nil always.

reading s3 upload progress

Original issue at fog/fog#1187

I'm using fog via carrierwave to allow users of my app to push videos to s3. The arrangement right now is a proxying one where they first push the file to my server, then I do some processing and push the output to s3 via fog. The actual upload happens via creating a Fog::Storage::AWS::File instance. It works great, but as the uploads can take several minutes I would like to be able to let the client poll me for upload progress so I can inform the user approximately when he'll be able to view his video. I can't figure out via documentation or looking at the source for the Fog::Storage::AWS::File model how to do this (query upload progress). Would appreciate guidance. To be clear, I've figured out how to report progress of the uploads to my server, but I need to know how to poll for (or otherwise be given) the progress of the upload (via fog) from my server to s3.

Is there an equivalent to describe-instance-status?

I'm looking for an equivalent to describe-instance-status - status returns "running" when the instance is up, but I would like to wait until status checks have passed before proceeding. It doesn't look like there is a way to get this information currently, but maybe I'm missing it?

AWS - distributionConfig.enabled' failed to satisfy constraint: Member must not be null

I have a Script to create a CDN Environment which does:

cdn = Fog::CDN.new({
:provider => 'AWS',
:aws_access_key_id => 'MY_KEY',
:aws_secret_access_key => 'MY_SECRET'
})

cdn.post_distribution({
'CustomOrigin' => {
'DNSName' => 'cdn.foo.bar',
'HTTPPort' => '80',
'OriginProtocolPolicy' => 'match-viewer',
'DefaultRootObject' => '/',
}

The error I get is:
/home/ezb/.rvm/gems/ruby-1.9.3-p547@fogDev/gems/excon-0.38.0/lib/excon/middlewares/expects.rb:10:in `response_call': Expected(201) <=> Actual(400 Bad Request) (Excon::Errors::BadRequest)
response => #"\nSenderMalformedXML1 validation error detected: Value null at 'distributionConfig.enabled' failed to satisfy constraint: Member must not be null16ef24ab-ac80-11e4-8ff3-dd12c6d41832", :headers=>{"x-amzn-RequestId"=>"16ef24ab-ac80-11e4-8ff3-dd12c6d41832", "Content-Type"=>"text/xml", "Content-Length"=>"371", "Date"=>"Wed, 04 Feb 2015 15:11:26 GMT"}, :status=>400, :remote_ip=>"176.32.98.148", :local_port=>57905, :local_address=>"10.100.6.203"}, @Body="\nSenderMalformedXML1 validation error detected: Value null at 'distributionConfig.enabled' failed to satisfy constraint: Member must not be null16ef24ab-ac80-11e4-8ff3-dd12c6d41832", @headers={"x-amzn-RequestId"=>"16ef24ab-ac80-11e4-8ff3-dd12c6d41832", "Content-Type"=>"text/xml", "Content-Length"=>"371", "Date"=>"Wed, 04 Feb 2015 15:11:26 GMT"}, @status=400, @remote_ip="176.32.98.148", @local_port=57905, @local_address="10.100.6.203">

Now, the Fog Library says that the distributionconfig enabled is OPTIONAL but it seems that, for AWS anyways, that is HAS to be set at the end:

cdn = Fog::CDN.new({
:provider => 'AWS',
:aws_access_key_id => 'MY_KEY',
:aws_secret_access_key => 'MY_SECRET'
})

cdn.post_distribution({
'CustomOrigin' => {
'DNSName' => 'cdn.foo.bar',
'HTTPPort' => '80',
'OriginProtocolPolicy' => 'match-viewer',
'DefaultRootObject' => '/',
},
'Enabled' => 'true'

It took awhile to figure it out WHY it wasn't working but I stumbled upon this gist:
https://gist.github.com/morten/3139f3ca80571f441f13

Anyways, not sure if this is a bug but I figure I'd bring it to your attention.

Heroku error

I googled this error and got some very old posts. I'm assuming this may be a valid bug due to the newness of this gem. uninitialized constant CarrierWave::Storage::Fog (NameError)

2015-03-25T09:27:31.876246+00:00 app[web.1]: Detected 512 MB available memory, 512 MB limit per process (WEB_MEMORY)
2015-03-25T09:27:31.876264+00:00 app[web.1]: Recommending WEB_CONCURRENCY=1
2015-03-25T09:27:32.048613+00:00 heroku[worker.1]: State changed from starting to up
2015-03-25T09:27:32.515704+00:00 app[web.1]: I, [2015-03-25T09:27:32.515592 #3]  INFO -- : Refreshing Gem list
2015-03-25T09:27:33.325050+00:00 app[worker.1]: Detected 512 MB available memory, 512 MB limit per process (WEB_MEMORY)
2015-03-25T09:27:33.326484+00:00 app[worker.1]: Recommending WEB_CONCURRENCY=1
2015-03-25T09:27:35.240294+00:00 app[web.1]: /app/vendor/bundle/ruby/2.2.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:73:in `eval': uninitialized constant CarrierWave::Storage::Fog (NameError)
2015-03-25T09:27:35.240300+00:00 app[web.1]:    from /app/vendor/bundle/ruby/2.2.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:73:in `eval'
2015-03-25T09:27:35.240320+00:00 app[web.1]:    from /app/vendor/bundle/ruby/2.2.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:73:in `storage'
2015-03-25T09:27:35.240322+00:00 app[web.1]:    from /app/app/uploaders/room_icon_uploader.rb:12:in `<class:RoomIconUploader>'
2015-03-25T09:27:35.240323+00:00 app[web.1]:    from /app/app/uploaders/room_icon_uploader.rb:3:in `<top (required)>'
2015-03-25T09:27:35.240325+00:00 app[web.1]:    from /app/app/models/room.rb:2:in `<class:Room>'
2015-03-25T09:27:35.240326+00:00 app[web.1]:    from /app/app/models/room.rb:1:in `<top (required)>'
2015-03-25T09:27:35.240327+00:00 app[web.1]:    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/engine.rb:472:in `block (2 levels) in eager_load!'
2015-03-25T09:27:35.240331+00:00 app[web.1]:    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/engine.rb:471:in `each'
2015-03-25T09:27:35.240332+00:00 app[web.1]:    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/engine.rb:471:in `block in eager_load!'
2015-03-25T09:27:35.240334+00:00 app[web.1]:    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/engine.rb:469:in `each'
2015-03-25T09:27:35.240335+00:00 app[web.1]:    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/engine.rb:469:in `eager_load!'
2015-03-25T09:27:35.240336+00:00 app[web.1]:    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/engine.rb:346:in `eager_load!'
2015-03-25T09:27:35.240338+00:00 app[web.1]:    from /app/vendor/bundle/ruby/2.2.0/gems/railties-4.2.0/lib/rails/application/finisher.rb:56:in `each'

more code is available at https://gist.github.com/morenoh149/a303bb9e823ffc77ac37

AWS Launch Configuration missing Ebs.Volume_Type

Original issue at: fog/fog#3021

The AWS Autoscaling Launch Configuration object is missing the volume type attribute for creating and describing. It should be part of the block_device_mappings.

Fog::Compute::AWS::Error InvalidParameterValue => secondary-addresses

Using fog to assign a private IP address to an AWS eni, I'm getting this error:

Fog::Compute::AWS::Error
------------------------
InvalidParameterValue => secondary-addresses

This is to assign a secondary private IP as a VIP for the instance.
This previously worked for me (Ubuntu 12.04 OS) on and eni - but fails now (Ubuntu 14.04) on an eni/instance created using the same recipe.

Recipe lines to create the address are shown below; and I can't see why it's even complaining that secondary-addresses is invalid as a param as it's not stipulated.

# Fetch our MAC address
mac = File.read('/sys/class/net/eth0/address').strip
# Use the MAC to get our AWS interface ID
eth0_interface_id = Mixlib::ShellOut.new("curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/#{mac}/interface-id").run_command.stdout

# Create a new AWS compute connection
connection = Fog::Compute.new(
  :provider               => :aws,
  :aws_access_key_id      => node['my_cookbook']['chef_ha_access_key_id'],
  :aws_secret_access_key  => node['my_cookbook']['chef_ha_secret_access_key'],
  :region                 => node['my_cookbook']['region'],
  :endpoint               => "https://ec2.#{node['my_cookbook']['region']}.amazonaws.com/"
)

# Here's where we attach the IP address
connection.assign_private_ip_addresses(eth0_interface_id, 'PrivateIpAddresses' => node['my_cookbook']['backend_vip']['ip_address'], 'AllowReassignment' => true )

SSL Error on S3 connection

Original issue at: fog/fog#3075

I'm getting an error with S3 at the moment. If I set up a connection (using my key + secret, eu-west-1) such that connection is an instance of Fog::Storage::AWS::Real, then do

puts connection.directories.to_a, I get an array of directory objects, as expected.

However, if I try:

p connection.directories.to_a or puts connection.directories

I get the error details in the below trace, after a long delay.

Any idea why this is happening?

/home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/ssl_socket.rb:105:in `connect_nonblock': SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A (OpenSSL::SSL::SSLError) (Excon::Errors::SocketError)
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/ssl_socket.rb:105:in `block in initialize'
    from /usr/lib/ruby/2.1.0/timeout.rb:91:in `block in timeout'
    from /usr/lib/ruby/2.1.0/timeout.rb:35:in `block in catch'
    from /usr/lib/ruby/2.1.0/timeout.rb:35:in `catch'
    from /usr/lib/ruby/2.1.0/timeout.rb:35:in `catch'
    from /usr/lib/ruby/2.1.0/timeout.rb:106:in `timeout'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/ssl_socket.rb:101:in `initialize'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/connection.rb:414:in `new'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/connection.rb:414:in `socket'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/connection.rb:126:in `request_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/mock.rb:44:in `request_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/instrumentor.rb:22:in `request_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/base.rb:15:in `request_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/base.rb:15:in `request_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/base.rb:15:in `request_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/connection.rb:269:in `request'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/idempotent.rb:12:in `error_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/connection.rb:292:in `rescue in request'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/connection.rb:229:in `request'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/idempotent.rb:12:in `error_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/connection.rb:292:in `rescue in request'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/connection.rb:229:in `request'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/idempotent.rb:12:in `error_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/middlewares/base.rb:10:in `error_call'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/connection.rb:292:in `rescue in request'
    from /home/iain/.gem/ruby/2.1.0/gems/excon-0.38.0/lib/excon/connection.rb:229:in `request'
    from /home/iain/.gem/ruby/2.1.0/gems/fog-1.23.0/lib/fog/xml/sax_parser_connection.rb:35:in `request'
    from /home/iain/.gem/ruby/2.1.0/gems/fog-1.23.0/lib/fog/xml/connection.rb:17:in `request'
    from /home/iain/.gem/ruby/2.1.0/gems/fog-1.23.0/lib/fog/aws/storage.rb:535:in `request'
    from /home/iain/.gem/ruby/2.1.0/gems/fog-1.23.0/lib/fog/aws/requests/storage/get_service.rb:21:in `get_service'
    from /home/iain/.gem/ruby/2.1.0/gems/fog-1.23.0/lib/fog/aws/models/storage/directories.rb:11:in `all'
    from /home/iain/.gem/ruby/2.1.0/gems/fog-core-1.23.0/lib/fog/core/collection.rb:139:in `lazy_load'
    from /home/iain/.gem/ruby/2.1.0/gems/fog-core-1.23.0/lib/fog/core/collection.rb:15:in `empty?'
    from /home/iain/.gem/ruby/2.1.0/gems/fog-core-1.23.0/lib/fog/core/collection.rb:84:in `block in inspect'
    from /home/iain/.gem/ruby/2.1.0/gems/formatador-0.2.5/lib/formatador.rb:92:in `indent'
    from /home/iain/.gem/ruby/2.1.0/gems/fog-core-1.23.0/lib/fog/core/collection.rb:77:in `inspect'

InvalidClientTokenId => The security token included in the request is invalid

Original issue at: fog/fog#3387

Using fog/1.26.0 fog-core/1.27.2

>> iam = Fog::AWS::IAM.new(Fog.credentials)
#<Fog::AWS::IAM::Real:70166177964140 @use_iam_profile=nil @connection_options={:debug_response=>true, :headers=>{"User-Agent"=>"fog/1.26.0 fog-core/1.27.2"}, :persistent=>false} @instrumentor=nil @instrumentor_name="fog.aws.iam" @host="iam.amazonaws.com" @path="/" @persistent=false @port=443 @scheme="https" @connection=#<Fog::XML::Connection:0x007fa1ac39dd48 @excon=#<Excon::Connection:7fa1ac397bc8 @data={:chunk_size=>1048576, :ciphers=>"HIGH:!SSLv2:!aNULL:!eNULL:!3DES", :connect_timeout=>60, :debug_request=>false, :debug_response=>true, :headers=>{"User-Agent"=>"fog/1.26.0 fog-core/1.27.2"}, :idempotent=>false, :instrumentor_name=>"excon", :middlewares=>[Excon::Middleware::ResponseParser, Excon::Middleware::Expects, Excon::Middleware::Idempotent, Excon::Middleware::Instrumentor, Excon::Middleware::Mock], :mock=>false, :nonblock=>true, :omit_default_port=>false, :persistent=>false, :read_timeout=>60, :retry_limit=>4, :ssl_verify_peer=>true, :tcp_nodelay=>false, :thread_safe_sockets=>true, :uri_parser=>URI, :versions=>"excon/0.42.1 (x86_64-darwin14.0) ruby/2.1.5", :write_timeout=>60, :host=>"iam.amazonaws.com", :path=>"/", :port=>443, :query=>nil, :scheme=>"https"} @socket_key="https://iam.amazonaws.com:443">> @aws_session_token=nil @aws_credentials_expire_at=nil @signer=#<Fog::AWS::SignatureV4:0x007fa1ac397790 @region="us-east-1", @service="iam", @aws_access_key_id="XXX", @hmac=#<Fog::HMAC:0x007fa1ac3976f0 @key="XXX", @digest=#<OpenSSL::Digest: XXX>, @signer=#<Proc:0x007fa1ac397650@/Users/drnic/.rvm/gems/ruby-2.1.5/gems/fog-core-1.27.2/lib/fog/core/hmac.rb:28 (lambda)>>>>

>> iam.list_account_aliases
Fog::AWS::IAM::Error: InvalidClientTokenId => The security token included in the request is invalid.
    from /Users/drnic/.rvm/gems/ruby-2.1.5/gems/excon-0.42.1/lib/excon/middlewares/expects.rb:6:in `response_call'
    from /Users/drnic/.rvm/gems/ruby-2.1.5/gems/excon-0.42.1/lib/excon/middlewares/response_parser.rb:8:in `response_call'

Invalid XML Character in S3 Response

Original issue at: fog/fog#3315

I'm trying to get all the versions in a directory, but the code is encountering xmlParseCharRef: invalid xmlChar value 26 (Nokogiri::XML::SyntaxError) when I call the versions function on the directory. Looking at the raw XML, it looks like there is some invalid characters in the response...a couple keys have those diamond-with-question-marks characters in them, which I'm guessing is what Nokogiri is dying on. Here's a sample traceback:

"/myapp/shared/bundle/ruby/2.1.0/gems/nokogiri-1.5.11/lib/nokogiri/xml/sax/push_parser.rb:47:in `native_write'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/nokogiri-1.5.11/lib/nokogiri/xml/sax/push_parser.rb:47:in `write'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/fog-1.23.0/lib/fog/xml/sax_parser_connection.rb:31:in `block in request'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/response.rb:87:in `call'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/response.rb:87:in `parse'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/middlewares/response_parser.rb:6:in `response_call'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/connection.rb:363:in `response'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/connection.rb:233:in `request'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/middlewares/idempotent.rb:22:in `error_call'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/middlewares/base.rb:10:in `error_call'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/middlewares/base.rb:10:in `error_call'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/connection.rb:253:in `rescue in request'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/connection.rb:201:in `request'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/middlewares/idempotent.rb:22:in `error_call'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/middlewares/base.rb:10:in `error_call'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/middlewares/base.rb:10:in `error_call'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/connection.rb:253:in `rescue in request'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/connection.rb:201:in `request'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/middlewares/idempotent.rb:22:in `error_call'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/middlewares/base.rb:10:in `error_call'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/middlewares/base.rb:10:in `error_call'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/connection.rb:253:in `rescue in request'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/excon-0.39.5/lib/excon/connection.rb:201:in `request'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/fog-1.23.0/lib/fog/xml/sax_parser_connection.rb:35:in `request'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/fog-1.23.0/lib/fog/xml/connection.rb:17:in `request'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/fog-1.23.0/lib/fog/aws/storage.rb:535:in `request'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/fog-1.23.0/lib/fog/aws/requests/storage/get_bucket_object_versions.rb:54:in `get_bucket_object_versions'", 
"/myapp/shared/bundle/ruby/2.1.0/gems/fog-1.23.0/lib/fog/aws/models/storage/versions.rb:17:in `all'"

Is there a way for me to get around this through Fog, maybe a way to have Fog replace invalid characters during the parsing?

fog-aws not working with dynamoDB Local

This what i see on the terminal when i try to create a table using dynamodb local

2.1.5 :046 > dynamo.create_table("people",
2.1.5 :047 >       {HashKeyElement: {AttributeName: "username", AttributeType: "S"}},
2.1.5 :048 >       {ReadCapacityUnits: 5, WriteCapacityUnits: 5})
Excon::Errors::BadRequest: Expected(200) <=> Actual(400 Bad Request)
excon.error.response
  :body          => "{\"__type\":\"com.amazonaws.dynamodb.v20120810#InvalidAction\",\"Message\":\"DynamoDB Local does not support v1 API.\"}"
  :headers       => {
    "Content-Length"   => "111"
    "Content-Type"     => "application/x-amz-json-1.0"
    "Server"           => "Jetty(8.1.12.v20130726)"
    "x-amzn-RequestId" => "c8d74d2d-a3fa-4b01-abeb-2aacacc2ef49"
  }
  :local_address => "::1"
  :local_port    => 53801
  :reason_phrase => "Bad Request"
  :remote_ip     => "127.0.0.1"
  :status        => 400
  :status_line   => "HTTP/1.1 400 Bad Request\r\n"

Is there a work around for this? or am i missing something? Please let me know.

AWS SQS AddPermission API missing

Original issue at: fog/fog#3375

Pretty straightforward: the AddPermission API from AWS SQS (http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_AddPermission.html) is not available in Fog::AWS::SQS.

Is this intentional (i.e., there's some issue with it) or inadvertent? If the latter, please add.

(EDIT: RemovePermission is also missing... might as well add that one too)

AWS AutoScaling group min_size & max_size getting set to 0

Original issue at: fog/fog#1521

When creating an auto scaling group, the min_size and max_size parameters are getting overridden to 0. I've tried looking through the code, but the way the mappings/aliases/etc is very strange to me, so I'm not sure where the error is occurring.

The way I'm trying to create the group is:

as = Fog::AWS::AutoScaling.new(...)
group = as.groups.create(..., :min_size => 1, :max_size => 2, ...)

I've put in code in the Fog::AWS::AutoScaling::Group#save method and at that point, both max_size and min_size are 0.

(This is with the 1.9.0 gem)

Clarify versioning on README

It would be nice to know if this gem is going to follow semantic versioning like Fog, or other scheme. The reason why I ask is because I didn't know what kind of pinning should I use when including it in my gemspecs.

Thanks!

SSLv3 deprecation: action required?

Normally I try to figure out these things on my own, but SSL etc. is terribly confusing, I'm not a network guy, and with all the different gems involved here, I'm not even sure where to start.

We just received an email from Amazon stating that "AWS will only support versions of the more modern Transport Layer Security (TLS) rather than SSLv3" as of April 30. Since we use Fog to access our S3 buckets, I'm not sure whether this implies that we should take any action, and if so, what exactly we are supposed to do.

I would be very thankful if somebody could shed some light on this issue.

AWS China region

Original issue at: fog/fog#3214

I don't think I see cn-north-1 references in fog yet.

It was announced Dec 2013 http://aws.amazon.com/blogs/aws/coming-soon-new-china-beijing-region/

Services:

Amazon Elastic Compute Cloud (Amazon EC2)
Amazon Elastic Block Store (Amazon EBS)
Amazon Simple Storage Service (Amazon S3)
Amazon Relational Database Service (Amazon RDS)
Amazon DynamoDB
Amazon Elastic MapReduce (Amazon EMR)
Amazon Virtual Private Cloud (Amazon VPC)
Amazon CloudWatch
AWS CloudFormation
Amazon Simple Queue Service (Amazon SQS)
Amazon Simple Notification Service (Amazon SNS)
Auto Scaling
Elastic Load Balancing
Amazon Glacier
Amazon Simple Workflow (SWF)
AWS Identity and Access Management (IAM)
Amazon ElastiCache
AWS Storage Gateway
AWS Management Console
AWS Premium Support

head_url signed

I'm using presigned head requests, currently using signed_url but would like to use a new method on files head_url , similar to get_url. Would a pull request with this be welcome?

Bring AWS CloudFront API Models/Requests up to date

Original issue at: fog/fog#3115

Heaps of missing functionality, specifically around the use of Price Classes, Custom/Dedicated SSL certificates, etc.

Current API version is 3~ years behind, has some authentication breakage when you try and just change the version to 2014-05-31:

 response => #<Excon::Response:0x007f991daab9b0 @data={:body=>"<?xml version=\"1.0\"?>\n<ErrorResponse xmlns=\"http://cloudfront.amazonaws.com/doc/2014-01-31/\"><Error><Type>Sender</Type><Code>IncompleteSignature</Code><Message>Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter.

Ruby warnings Comparable & Return nil

I receive the following warnings when require Fog with Ruby 2.2.1:

vendor/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/fog-core-1.29.0/lib/fog/core/service.rb:166: warning: Comparable#== will no more rescue exceptions of #<=> in the next release.
vendor/ruby-2.2.1/lib/ruby/gems/2.2.0/gems/fog-core-1.29.0/lib/fog/core/service.rb:166: warning: Return nil in #<=> if the comparison is inappropriate or avoid such comparison.

I am on Fog master, commit f16fae7b0b40af3212facbf317a76628e353fcc5

add missing attributes to aws describe_reserved_instances parser

Original issue at: fog/fog#1746

In particular it looks like tagSet, instanceTenancy, currencyCode, recurringCharges:amount and recurringCharges:frequency. See: http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-ItemType-DescribeReservedInstancesResponseSetItemType.html

Also, since tagSet has tags it is throwing off the parser when tags are present to make it appear as though there are one or more (empty) instances in the return value.

delete_on_termination=true attribute on new volume is not set on create

original issue at : fog/fog#1737

I am using a config hash to create a new EBS volume and attach it to an existing server like so:

      new_ebs_config = {
                        'availability_zone'     => 'us-east-1d,
                        'size'                  => 20,
                        'device'                => "/dev/sdj",
                        'delete_on_termination' => true,
                        'type' => 'standard'
                        'server' => instance_id
      }
      new_volume = @ec2.volumes.create(new_ebs_config)

The delete_on_termination attribute is not being set on this new volume after it has been attached to the server. I have been using

          @ec2.modify_instance_attribute(
            @instance_id, {
              'BlockDeviceMapping.1.DeviceName'              => config['device'],
              'BlockDeviceMapping.1.Ebs.DeleteOnTermination' => config['delete_on_termination'].to_s
            }
          )

It would be nice to have it set after attach is called if server is part of the passed in config.

Add support for AWS Lambda

I'm working on this, I already submitted an early PR (#123) to discuss some implementation details.

Fog doesn't support storage_type or gp2 for RDS?

Hello,

It looks like Fog doesn't support the storage_type (StorageType) parameter for RDS creation, which means there's no way to create an RDS instance that uses General Purpose SSD (gp2). It does have the 'iops' parameter for a Provisioned IOps RDS instance (which assumes a storage_type of 'io1'). But I don't see a way to specify gp2.

Am I misunderstanding something? Is this a missing feature or is it intentional?

Here are the relevant docs in Amazon:

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateInstance.html

http://docs.aws.amazon.com/AmazonRDS/latest/CommandLineReference/CLIReference-cmd-CreateDBInstance.html

Repeatable signed urls for the same expiry

Hi - one thing we've noticed when V4 signatures are active is that calling Fog::Storage::AWS::File.url (on an instance, not the class) with the same expiry time results in a new, unique signature/url each time.

With V2 it's the same output for an identical expiry. Because of this we're not giving users the ability to cache the files served with a V4 signature.

Would you have any suggestions to get the same URL when called multiple times with the same expiry param on V4 signed urls?

Setting region of AWS::Compute after initialization

Original issue at: fog/fog#1825

It would be nice to be able to change the aws region AFTER the connection object is created, however this does not seem to be possible. The host and excon connection still point to ec2.us-east-1.amazonaws.com even though the region has been set to 'us-west-1'.

1.9.2p290 :024 > ec2_connection = Fog::Compute::AWS.new(:aws_access_key_id => 'AKIAI....', :aws_secret_access_key => 'wrun.............')
=> #<Fog::Compute::AWS::Real:70116107567500 @use_iam_profile=nil @aws_access_key_id="AKIAIK.........." @aws_credentials_expire_at=nil @connection_options={} @region="us-east-1" @Instrumentor=nil @instrumentor_name="fog.aws.compute" @Version="2012-12-01" @endpoint=nil @host="ec2.us-east-1.amazonaws.com" @path="/" @Persistent=false @PORT=443 @scheme="https" @connection=#<Fog::Connection:0x007f8a5b5d17f8 @excon=#<Excon::Connection:7f8a5b5d12d0 @DaTa={:chunk_size=>1048576, :connect_timeout=>60, :headers=>{"User-Agent"=>"fog/1.9.0"}, :idempotent=>false, :instrumentor_name=>"excon", :middlewares=>[Excon::Middleware::Expects, Excon::Middleware::Idempotent, Excon::Middleware::Instrumentor, Excon::Middleware::Mock], :mock=>false, :nonblock=>true, :read_timeout=>60, :retry_limit=>4, :ssl_ca_file=>"/Users/atistler/.rvm/gems/ruby-1.9.2-p290@cloudstack/gems/excon-0.21.0/data/cacert.pem", :ssl_verify_peer=>false, :uri_parser=>URI, :write_timeout=>60, :host=>"ec2.us-east-1.amazonaws.com", :path=>"/", :port=>"443", :query=>nil, :scheme=>"https", :user=>nil, :password=>nil} @socket_key="https://ec2.us-east-1.amazonaws.com:443">, @Persistent=false>>
1.9.2p290 :025 > ec2_connection.region = 'us-west-1'
=> "us-west-1"
1.9.2p290 :026 > ec2_connection
=> #<Fog::Compute::AWS::Real:70116107567500 @use_iam_profile=nil @aws_access_key_id="AKIAIKTPL6FEOAZCNTYQ" @aws_credentials_expire_at=nil @connection_options={} @region="us-west-1" @Instrumentor=nil @instrumentor_name="fog.aws.compute" @Version="2012-12-01" @endpoint=nil @host="ec2.us-east-1.amazonaws.com" @path="/" @Persistent=false @PORT=443 @scheme="https" @connection=#<Fog::Connection:0x007f8a5b5d17f8 @excon=#<Excon::Connection:7f8a5b5d12d0 @DaTa={:chunk_size=>1048576, :connect_timeout=>60, :headers=>{"User-Agent"=>"fog/1.9.0"}, :idempotent=>false, :instrumentor_name=>"excon", :middlewares=>[Excon::Middleware::Expects, Excon::Middleware::Idempotent, Excon::Middleware::Instrumentor, Excon::Middleware::Mock], :mock=>false, :nonblock=>true, :read_timeout=>60, :retry_limit=>4, :ssl_ca_file=>"/Users/atistler/.rvm/gems/ruby-1.9.2-p290@cloudstack/gems/excon-0.21.0/data/cacert.pem", :ssl_verify_peer=>false, :uri_parser=>URI, :write_timeout=>60, :host=>"ec2.us-east-1.amazonaws.com", :path=>"/", :port=>"443", :query=>nil, :scheme=>"https", :user=>nil, :password=>nil} @socket_key="https://ec2.us-east-1.amazonaws.com:443">, @Persistent=false>>

Why are the credential keys not generalized?

Original issue at: fog/fog#3165

We're currently implementing fog in one of our applications and were surprised to find that all of the credential keys for each service are specific to that service.

I would have thought that I'd be able to set something like:

STORAGE_PROVIDER=AWS
STORAGE_USERNAME=foo
STORAGE_PASSWORD=bar

And in my code do something like:

    @connection ||= Fog::Storage.new(
      provider: ENV['STORAGE_PROVIDER'],
      username: ENV['STORAGE_USERNAME'],
      password: ENV['STORAGE_PASSWORD'])

Considering the point of fog is that you can abstract away your cloud providers, then theoretically changing it shouldn't require a code change.

Please let me know if I'm misunderstanding.

Thanks for all the hard work on this!

AWS auto scaling: availability zones are not a required parameter

Original issue at: fog/fog#1503

Currently AWS::AutoScaling::Group has availability_zones as a required parameter. Per the AWS API documentation, availability zones are not a required parameter when creating an auto scaling group. If you provide VPCZoneIdentifier (subnet IDs), the zone(s) is optional.

auto_scaling_group.instances does not return only instances for that group

Original issue at: fog/fog#2019

here's a gist that shows the problem:
https://gist.github.com/akatz/798561a1f893b8a6750f

but essentially calling instances on an autoscaling group returns all auto scaling instances. I def don't expect that.

Relevant line here:
https://github.com/fog/fog/blob/master/lib/fog/aws/models/auto_scaling/group.rb#L77-L79

user_data is still base64 encoded in Real launch_configurations

I think the Mock is working correctly. When I pull a launch_configuration from AWS, it comes back as base64. Is this a known issue? I'm hacking around it right now.
Thanks,
-Ben

Add support for Amazon Kinesis

Original issue at: fog/fog#3086

Amazon Kinesis

"NoMethodError: undefined method `body' for #<Fog::DNS::AWS::Error:0x007f6c673e1720>"

The following line of code is not expecting a Fog::DNS::AWS::Error to be returned, so it's blindly calling .body on the result of service.change_resource_record_sets:

fog-aws/lib/fog/aws/models/dns/record.rb

Line 44 in 5903a78

data = service.change_resource_record_sets(zone.id, [options]).body

data = service.change_resource_record_sets(zone.id, [options]).body

My code is doing the following:

record = Fog::DNS['AWS'].zones.get( "......" ).records.new({
  name: "foo.foobar.com",
  value: "ec2-123-123-123-123.compute-1.amazonaws.com",
  ttl: 600,
  type: 'CNAME'
})
record.save # this line raises the error

Support AWS Support API

Original issue at: fog/fog#2656

Amazon provides an API for managing support tickets. It'd be awesome for Fog to provide these APIs. http://docs.aws.amazon.com/awssupport/latest/APIReference/Welcome.html

I volunteer to write the support for the AWS Support API. I'm submitting the ticket as a placeholder in case anybody else is working on this as well - or in case the Fog maintainers object before I put too much time into it.

Fog-aws not working with Hitachi

Good morning.
I have been trying to use fog to connect to the S3-compatible endpoint of an Hitachi instance and I get the following error (this comes from an irb session):

2.1.5 :014 > connection.directories
Excon::Errors::Found: Expected(200) <=> Actual(302 Found)
excon.error.response
  :body          => ""
  :headers       => {
    "Content-Length" => "0"
    "Content-Type"   => "text/xml"
    "Date"           => "Fri, 22 May 2015 09:19:10 GMT"
    "Location"       => "https://mybucket.hitachiinstance.net/rest"
    "Server"         => "HCP V7.1.0.10"
  }
  :local_address => "10.0.0.2"
  :local_port    => 47938
  :reason_phrase => "Found"
  :remote_ip     => "10.5.0.30"
  :status        => 302
  :status_line   => "HTTP/1.1 302 Found\r\n"

        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/expects.rb:6:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/response_parser.rb:8:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:372:in `response'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:236:in `request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/idempotent.rb:26:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:73:[41/1875]
_call'                                                                                          [0/1876]
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:256:in `rescue in 
request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:204:in `request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/idempotent.rb:26:in 
`error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error
_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error
_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:256:in `rescue in 
request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:204:in `request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/idempotent.rb:26:in 
`error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error
_call'
... 7 levels...
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-aws-0.2.2/lib/fog/aws/requests/storage/get_service.
rb:21:in `get_service'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-aws-0.2.2/lib/fog/aws/models/storage/directories.rb
:10:in `all'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:113:in `lazy
_load'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:17:in `map'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:85:in `inspect_ob
ject'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:72:in `block in n
ested_objects_string'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/formatador-0.2.5/lib/formatador.rb:92:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:41:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:72:in `nested_obj
ects_string'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:54:in `object_str
ing'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:14:in `block in f
ormat'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/formatador-0.2.5/lib/formatador.rb:92:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:41:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:14:in `format'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:73:in `inspe
ct'
        from /usr/local/rvm/rubies/ruby-2.1.5/bin/irb:11:in `<main>'2.1.5

It seemed that Excon was not following a 302 redirection. Thus, I did this:

Excon.defaults[:middlewares].insert(0,Excon::Middleware::RedirectFollower)

So that Excon::Middleware::RedirectFollower is placed before Excon::Middleware::Expects.

Then, the 302 response became a 403 forbidden, although I was providing good credentials (I triple-checked this). The full stack trace is:

2.1.5 :090 > connection.directories
Excon::Errors::Forbidden: Expected(200) <=> Actual(403 Forbidden)
excon.error.response
  :body          => ""
  :headers       => {
    "Content-Length" => "0"
    "Date"           => "Fri, 22 May 2015 09:53:34 GMT"
    "Expires"        => "Thu, 01 Jan 1970 00:00:00 GMT"
    "Server"         => "Jetty(7.6.0.v20120127)"
    "Set-Cookie"     => "JSESSIONID=1n23cfhzrnb074sniv7jnfkxm;Path=/"
  }
  :local_address => "10.0.0.2"
  :local_port    => 48045
  :reason_phrase => "Forbidden"
  :remote_ip     => "10.5.0.30"
  :status        => 403
  :status_line   => "HTTP/1.1 403 Forbidden\r\n"

        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/expects.rb:6:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/redirect_follower.rb:48:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/response_parser.rb:8:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/redirect_follower.rb:51:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:372:in `response'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:236:in `request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/redirect_follower.rb:45:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/response_parser.rb:8:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/redirect_follower.rb:51:in `response_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:372:in `response'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/connection.rb:236:in `request'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/idempotent.rb:26:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error_call'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/excon-0.45.3/lib/excon/middlewares/base.rb:10:in `error_call'
... 20 levels...
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-aws-0.2.2/lib/fog/aws/requests/storage/get_service.rb:21:in `get_service'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-aws-0.2.2/lib/fog/aws/models/storage/directories.rb:10:in `all'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:113:in `lazy_load'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:17:in `map'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:85:in `inspect_object'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:72:in `block in nested_objects_string'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/formatador-0.2.5/lib/formatador.rb:92:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:41:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:72:in `nested_objects_string'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:54:in `object_string'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:14:in `block in format'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/formatador-0.2.5/lib/formatador.rb:92:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:41:in `indent'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/formatador.rb:14:in `format'
        from /usr/local/rvm/gems/ruby-2.1.5/gems/fog-core-1.30.0/lib/fog/core/collection.rb:73:in `inspect'
        from /usr/local/rvm/rubies/ruby-2.1.5/bin/irb:11:in `<main>'2.1.5 :091 >

Can you help me with this, please?

Thinking more in the future, is there any plan to natively support 302 redirections at fog out-of-the-box? I say this because, in my irb session, I can change the Excon defaults with no problem. But I need to do this kind of connection from a more complex program, which uses fog and only accepts a fog connection hash (and does not allow me setting anything else).

missed files

@lanej a couple things just came in under the wire before the switch. Could you pull these in? See: https://github.com/fog/fog/tree/master/lib/fog/aws (the requests/parsers). Might be related test changes to grab too. Sorry I missed that before. Thanks!

No usage instructions in Readme

AWS CloudFormation ListStacks options

Original issue at fog/fog#2287

I've been using the API for CloudFormation, and I noticed that in list_stacks, to give the StackStatusFilter option, instead of specifying an array of members for the option such as
{"StackStatusFilter"=>["CREATE_IN_PROGRESS", "CREATE_COMPLETE"...]}
I have to do something like

{"StackStatusFilter.member.1"=>"CREATE_FAILED", "StackStatusFilter.member.2"=>"CREATE_IN_PROGRESS"}

Is this intentional? I know AWS APIs require StackStatusFilter.member.N, but in other API calls with Fog, instead of specifying .member for each option, I can just pass in an array (create_stack has examples of this in their options, with Parameters/Capabilities)