All DynamoDB requests time out

[ex-aws]

All DynamoDB requests time out

[jordi-chacon]

I have fixed the timeout problem by sending the requests to https://dynamodb.eu-west-1.amazonaws.com:443/ instead.

Now I have a different problem. My requests failed to authenticate:

{error,
  {<<\"UnrecognizedClientException\">>,
   <<\"The security token included in the request is invalid.\">>}}

Debugging data:

url:
"https://dynamodb.eu-west-1.amazonaws.com:443/"

headers:
[{"Authorization",
  "AWS4-HMAC-SHA256 Credential=A....A/20150628/eu-west-1/dynamodb/aws4_request,SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-target,Signature=98........................c1"},
 {"host", "dynamodb.eu-west-1.amazonaws.com"},
 {"x-amz-date", "20150628T071505Z"},
 {"x-amz-target", "DynamoDB_20120810.CreateTable"},
 {"content-type", "application/x-amz-json-1.0"},
 {"x-amz-content-sha256", ""}]

config:
%{access_key_id: "A.......A",
  host: "dynamodb.eu-west-1.amazonaws.com",
  http_client: HTTPoison,
  json_codec: Poison,
  port: "443",
  region: "eu-west-1",
  scheme: "https://",
  secret_access_key: "k.....c"}

service:
:dynamodb

I have retrieved the key and the secret by curling http://169.254.169.254/latest/meta-data/iam/security-credentials/my_role and they are the same as the ones in the config. So that looks correct.

Any hints on what could be the problem?

[benwilson512]

Hey sorry I'm just not getting to this. So the issue related to the ports is partially my fault, partially not. HTTPs uses port 443 not port 80, so it isn't surprising that 80 didn't work. You could actually just leave off specifying the port entirely because then the URL would be simply "https://dynamodb.eu-west-1.amazonaws.com" and that will route to the correct port.

Basically, if the port is a numerical 80 then I don't put it as part of the URL at all. This is why the config may say port 80 for https but still work anyway, it gets stripped when the URLs are generated. This is clearly confusing behavior and I plan on doing something about it.

Unfortunately I'm not sure why that's the error you're getting. Try specifying the port as a numerical 80 which will drop it from the URL. I wonder if including the port number is munging up the AWS request signing process.

[jordi-chacon]

Hi!

Thanks for your reply.

I have specified the port as numerical 80 in config.exs as well as schema "http://". The url sent to ExAws.Request is now "http://dynamodb.eu-west-1.amazonaws.com/". So no port there as you said. Still getting the same error though :/

Is there anything else that I could debug that could hint us what the problem is?

[benwilson512]

Hey, I didn't intend for you to change the schema. HTTPoison (the http client library) is smart enough to use the correct port when the request is actually sent via HTTP. I simply want the port out of the signing process. Try with a schema of "https://" and a port of numerical 80.

On a different angle, the error you're getting seems to have to do with authentication, but isn't about signing errors. This seems to indicate that there may be permission issues with the IAM credentials. Have you used the AWS IAM simulator to determine if that role is able to execute the requests you want?

[jordi-chacon]

The IAM simulator says the operation is allowed. All DynamoDB operations are allowed by my policy. Furthermore, I have sent such requests from both inside the EC2 instance and from inside an ECS container via awscli.

[benwilson512]

Awesome good to know, we'll figure it out. Any luck with [schema: "https://", port: 80] ?

[jordi-chacon]

Just tried it, same problem.

[benwilson512]

Ok. I haven't had a chance to test much on the EU region, I'll ssh into an instance this afternoon and see if I can reproduce and determine the issues.

[jordi-chacon]

Thanks for your support!

I have tested running my Elixir application in an Ubuntu Server EC2 instance instead of inside an ECS container and the problem remains.

[jordi-chacon]

Interesting news.

Just did another test where, before starting the Elixir application, I run

export AWS_ACCESS_KEY_ID=A...L
export AWS_SECRET_ACCESS_KEY=b.....a

using the key and secret of my admin account. This worked. The requests to dynamodb worked just fine!

I have also tried exporting those environment variables with the key and secret I get from:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/myrole

This doesn't work. These credentials are actually the ones fetched by your library if I don't export the environment variables. So the problem seems to be with the key and the secret used.

[benwilson512]

Well that's certainly a relief on my end. Only thing I can recommend is to fiddle around with with the IAM simulator until you can reproduce the issue with the role / keys. Best of luck!

[jordi-chacon]

This is very strange.

Without running aws configure, this works:

aws dynamodb create-table --region eu-west-1 --table-name test_table --attribute-definitions AttributeName=uid,AttributeType=S --key-schema AttributeName=uid,KeyType=HASH --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

If this works, shouldn't the requests coming from ex_aws also work?

[benwilson512]

Use set to determine if your environment variables are still set. aws will use those first.

[jordi-chacon]

I run this on a fresh new EC2 instance. I have not copied my admin credentials to this one.

[benwilson512]

So let me see if I can recap:

• aws dynamodb $stuff: works
• curl myrole + setting environment variables based on that result: doesn't work
• ExAws configured to use instance_role: doesn't work

The fact that the second option doesn't work is super weird. Let me see if I can reproduce.

[jordi-chacon]

The recap is correct.

My EC2 instance is running ubuntu-trusty-14.04-amd64-server-20150325 (ami-47a23a30).

This EC2 instance has a role with the "Managed Policy" "AmazonDynamoDBFullAccess".

[jordi-chacon]

I'm a n00b in AWS, so I might be misunderstanding this statement, but it say that we should include a session token, which we don't seem to do.

"When you make a call using temporary security credentials, the call must include a session token, which is returned along with those temporary credentials. AWS uses the session token to validate the temporary security credentials."

http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html

[benwilson512]

You may well be right there. This hasn't been a feature I've had a chance to test much I'm afraid. Investigating.

[benwilson512]

I've been able to reproduce. I believe you're correct about the cause. Fixing.

[jordi-chacon]

Cool!

Yes, I think we need to add a header called X-Amz-Security-Token with the value of the Token field in the security credentials.

[benwilson512]

This should be fixed on master now. Will do a release shortly. I've tested it myself, but please confirm that it looks ok on your end as well.

[jordi-chacon]

It works!!! 😃
Thank you so much for your dedication Ben.