Shared SQS Queue Deploy Failure Due to IAM Permission Issues

[djolereject]

Shared SQS Queue Deploy Failure Due to IAM Permission Issues

[djolereject]

And you guessed it correctly! I went to the failed stack ARN with your helpful link and saw it there:

Resource handler returned message: "User: arn:aws:iam::228342851134:user/{my user} is not authorized to perform: sqs:createqueue on resource: arn:aws:sqs:eu-central-1:228342851134:demo-dev-List-2AXDJ1D221D1-Waitlist-QzhRPuni2qNn because no identity-based policy allows the sqs:createqueue action ...

I added AmazonSQSFullAccess to the user and all went smoothly.
Thanks a lot! I don't think I would've found this on my own. Something about all those users/permissions/services is just too much to keep in working memory for me.

Thank you once more!

[djolereject]

Just to expand, since I noticed I didn't mention it, but this is in my config/application.rb:

    config.managed_iam_policy = %w[AWSLambda_FullAccess AmazonSQSFullAccess]

as I understand it should be enough to allow creating everything needed?

[tongueroo]

Unsure. Wondering about this:

05:15:28AM CREATE_FAILED AWS::CloudFormation::Stack List Embedded stack arn:aws:cloudformation:eu-central-1:... was not successfully created: The following resource(s) failed to create: [Waitlist]. 

Can you see what the error on the nested List stack is. Here are some tips https://docs.rubyonjets.com/docs/debug/cloudformation/

Note, I also put together a demo test project https://github.com/tongueroo/demo-jets-sqs-queue It deployed successfully on a brand new cloud9 machine.

Maybe try deploying the test project to debug also. Thinking the CloudFormation error in the List stack will be most helpful though.

Deploy Logs

tung:~/environment/demo (main) $ jets deploy
Deploying stack demo-dev ...
Building CloudFormation templates
Built CloudFormation templates at /tmp/jets/demo/templates
Deploying CloudFormation stack with jets app!
Waiting for stack to complete
06:11:03PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack demo-dev User Initiated
06:11:06PM CREATE_IN_PROGRESS AWS::S3::Bucket S3Bucket 
06:11:09PM CREATE_IN_PROGRESS AWS::S3::Bucket S3Bucket Resource creation Initiated
06:11:36PM CREATE_COMPLETE AWS::S3::Bucket S3Bucket 
06:11:38PM CREATE_COMPLETE AWS::CloudFormation::Stack demo-dev 
Stack success status: CREATE_COMPLETE
Time took: 35s
The /tmp/jets/demo/cache folder exists. Incrementally re-building the jets using the cache.  To clear the cache: rm -rf /tmp/jets/demo/cache
=> jets assets:precompile
=> Copying current project directory to temporary build area: /tmp/jets/demo/stage/code
=> Reconfiguring webpacker development settings for AWS Lambda.
=> Bundling: running bundle install in cache area: /tmp/jets/demo/cache.
=> cd /tmp/jets/demo/cache && env bundle install
Using rake 13.1.0
Using concurrent-ruby 1.2.2
Using i18n 1.14.1
Using minitest 5.20.0
Using tzinfo 2.0.6
Using activesupport 7.0.8
Using builder 3.2.4
Using erubi 1.12.0
Using racc 1.7.3
Using nokogiri 1.15.5 (x86_64-linux)
Using rails-dom-testing 2.2.0
Using crass 1.0.6
Using loofah 2.22.0
Using rails-html-sanitizer 1.6.0
Using actionview 7.0.8
Using rack 2.2.8
Using rack-test 2.1.0
Using actionpack 7.0.8
Using globalid 1.2.1
Using activejob 7.0.8
Using mini_mime 1.1.5
Using date 3.3.4
Using timeout 0.4.1
Using net-protocol 0.2.2
Using net-imap 0.4.8
Using net-pop 0.1.2
Using net-smtp 0.4.0
Using mail 2.8.1
Using actionmailer 7.0.8
Using activemodel 7.0.8
Using activerecord 7.0.8
Using aws-eventstream 1.3.0
Using aws-partitions 1.867.0
Using aws-sigv4 1.8.0
Using jmespath 1.6.2
Using aws-sdk-core 3.190.0
Using aws_config 0.1.1
Using memoist 0.16.2
Using rainbow 3.1.1
Using thor 1.3.0
Using zeitwerk 2.6.12
Using aws-mfa-secure 0.4.4
Using aws-sdk-cloudwatchlogs 1.76.0
Using aws-logs 0.5.1
Using aws-sdk-apigateway 1.90.0
Using aws-sdk-cloudformation 1.97.0
Using aws-sdk-dynamodb 1.98.0
Using aws-sdk-kinesis 1.54.0
Using aws-sdk-kms 1.74.0
Using aws-sdk-lambda 1.113.0
Using aws-sdk-s3 1.141.0
Using aws-sdk-sns 1.70.0
Using aws-sdk-sqs 1.69.0
Using aws-sdk-ssm 1.162.0
Using bundler 2.4.10
Using cfn-status 0.4.6
Using cfn_camelizer 0.4.9
Using cfn_response 0.2.0
Using text-table 1.2.4
Using cli-format 0.5.0
Using dotenv 2.8.1
Using dsl_evaluator 0.3.1
Using gems 1.2.0
Using hashie 5.0.0
Using importmap 0.1.0
Using importmap-jets 0.1.1
Using jets-api 0.1.5
Using jets-git 0.2.0
Using jets-html-sanitizer 1.0.4
Using rexml 3.2.6
Using kramdown 2.4.0
Using method_source 1.0.0
Using railties 7.0.8
Using recursive-open-struct 1.1.3
Using jets 5.0.2
Using mysql2 0.5.5
Using sprockets 4.2.1
Using sprockets-jets 0.1.0
Bundle complete! 9 Gemfile dependencies, 78 gems now installed.
Gems in the groups 'development' and 'test' were not installed.
Bundled gems are installed into `./vendor/gems`
Bundle install completed
Tidying project: removing ignored files to reduce package size.
=> rsync -a --links /tmp/jets/demo/stage/code/vendor/gems/ruby/3.2.0/ /tmp/jets/demo/stage/opt/ruby/gems/3.2.0/
=> Replacing compiled gems with AWS Lambda Linux compiled versions: /tmp/jets/demo/stage/opt
Checking project for compiled gems...
=> Generating shims in the handlers folder.
Checking s3://demo-dev-s3bucket-wlp7oi01imol/jets/code/opt-17e2d665.zip
=> Creating zip file for /tmp/jets/demo/stage/opt
=> cd /tmp/jets/demo/stage/opt && chmod -R 755 . && zip --symlinks -rq opt.zip .
Zip file created at: /tmp/jets/demo/stage/zips/opt-17e2d665.zip (18.3 MB)
Checking s3://demo-dev-s3bucket-wlp7oi01imol/jets/code/code-24e03de7.zip
=> Creating zip file for /tmp/jets/demo/stage/code
=> cd /tmp/jets/demo/stage/code && chmod -R 755 . && zip --symlinks -rq code.zip .
Zip file created at: /tmp/jets/demo/stage/zips/code-24e03de7.zip (23.5 KB)
Building CloudFormation templates
Built CloudFormation templates at /tmp/jets/demo/templates
Uploading CloudFormation templates to S3.
Uploading code zip files to S3.
Uploading /tmp/jets/demo/stage/zips/code-24e03de7.zip (23.5 KB) to S3
Uploaded to s3://demo-dev-s3bucket-wlp7oi01imol/jets/code/code-24e03de7.zip
Time to upload code to s3: 0s
Uploading /tmp/jets/demo/stage/zips/opt-17e2d665.zip (18.3 MB) to S3
Uploaded to s3://demo-dev-s3bucket-wlp7oi01imol/jets/code/opt-17e2d665.zip
Time to upload code to s3: 0s
Checking for modified public assets and uploading to S3.
Time for public assets to s3: 0s
Deploying CloudFormation stack with jets app!
Waiting for stack to complete
06:11:53PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack demo-dev User Initiated
06:11:57PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway 
06:11:57PM CREATE_IN_PROGRESS AWS::IAM::Role IamRole 
06:11:57PM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer 
06:11:58PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway Resource creation Initiated
06:11:58PM CREATE_IN_PROGRESS AWS::IAM::Role IamRole Resource creation Initiated
06:12:06PM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer Resource creation Initiated
06:12:07PM CREATE_COMPLETE AWS::Lambda::LayerVersion GemLayer 
06:12:08PM CREATE_COMPLETE AWS::CloudFormation::Stack ApiGateway 
06:12:09PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiResources1 
06:12:10PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiResources1 Resource creation Initiated
06:12:15PM CREATE_COMPLETE AWS::IAM::Role IamRole 
06:12:16PM CREATE_IN_PROGRESS AWS::IAM::Policy IamPolicy 
06:12:16PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack List 
06:12:16PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack PostmanJob 
06:12:16PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsController 
06:12:17PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack List Resource creation Initiated
06:12:17PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack PostmanJob Resource creation Initiated
06:12:17PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsController Resource creation Initiated
06:12:22PM CREATE_IN_PROGRESS AWS::IAM::Policy IamPolicy Resource creation Initiated
06:12:28PM CREATE_COMPLETE AWS::CloudFormation::Stack List 
06:12:29PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack HardJob 
06:12:31PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack HardJob Resource creation Initiated
06:12:32PM CREATE_COMPLETE AWS::CloudFormation::Stack ApiResources1 
06:12:37PM CREATE_COMPLETE AWS::IAM::Policy IamPolicy 
06:12:51PM CREATE_COMPLETE AWS::CloudFormation::Stack JetsController 
06:12:52PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiMethods1 
06:12:53PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiMethods1 Resource creation Initiated
06:13:04PM CREATE_COMPLETE AWS::CloudFormation::Stack ApiMethods1 
06:13:04PM CREATE_COMPLETE AWS::CloudFormation::Stack PostmanJob 
06:13:05PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiDeployment20231215181152 
06:13:06PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiDeployment20231215181152 Resource creation Initiated
06:13:17PM CREATE_COMPLETE AWS::CloudFormation::Stack ApiDeployment20231215181152 
06:13:48PM CREATE_COMPLETE AWS::CloudFormation::Stack HardJob 
06:13:49PM UPDATE_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack demo-dev 
06:13:50PM UPDATE_COMPLETE AWS::CloudFormation::Stack demo-dev 
Stack success status: UPDATE_COMPLETE
Time took: 1m 57s
Prewarming application.
API Gateway Endpoint: https://6z18xbrvnd.execute-api.us-west-2.amazonaws.com/dev/
Release version: 1
tung:~/environment/demo (main) $ 

Screenshot of CloudFormation stacks:

Shared SQS List screenshot:

Here's are also the lambda functions.

I also did some manual test by sending a message to the generated SQS queue.

$ export QUEUE_URL=https://sqs.us-west-2.amazonaws.com/112233445566/demo-dev-List-NDBP8IFB1M05-Waitlist-StxATDNj3X78
$ aws sqs send-message --queue-url $QUEUE_URL --message-body '{"test": "hello world"}'                                                                                   
{
    "MD5OfMessageBody": "3d635e69eb93fd184b47a31d460ca2b6",
    "MessageId": "3a9a99e1-a866-4e68-ac31-e413fb495988"
}
$ 

You can log the hard_job-fix in another terminal and see the event coming in

$ jets logs -f -n hard_job-fix
Tailing logs for /aws/lambda/demo-dev-hard_job-fix
Waiting for log group /aws/lambda/demo-dev-hard_job-fix to exist. Waiting 5 seconds.
2023-12-15 18:17:55 UTC INIT_START Runtime Version: ruby:3.2.v9 Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:b96ddb9b1905c3979339d7706a5f7cfda1d851593b1255eb0f15ff573c17fd28
2023-12-15 18:17:58 UTC START RequestId: be392533-83a5-5aae-b263-78c5192df684 Version: $LATEST
2023-12-15 18:17:58 UTC fix {"Records":[{"messageId":"3a9a99e1-a866-4e68-ac31-e413fb495988","receiptHandle":"AQEBzmNbOceOGqBkoYPEfF7ZUQ2MoXkrY1cHFADrUjDiw3t4ecP2ZOlhAOicrPONYzmC9EPkRRchxaGDH4mN/G5WfzyCmbUo/BQBFuvqpVhBrXpZJ8pNJPKkDfXSZbtpvMLdCofTHuAg6IhDU3K9l/EEQzlF9WSyOFnGfrsLY15yeGyJ4pfJ6R3+X6dt4uCy3fpU1hD2zwJoVrZVYz499Sr9b5vh+0zyR570idC2EQww+r6XkNR1YA2+rALsashA1XI/868ZPA2CGeaj+qAlPOoQUv0JBW22ulILUi0i8jzVu9sTcJAoyzz4JzAjq0uaSIa2WPShMKA7XN0ArM7bs3MJ6ngCnqHzOKR857i6dyN5BeVENH+2mQzHZwwUX0ImanOIWi1bxXEztzEr8SNo2l88r9gIk1Tw5QGmhGL1nw3lwg3nHwjH3hDZE9YED4YVqVJZ","body":"{\"test\": \"hello world\"}","attributes":{"ApproximateReceiveCount":"1","SentTimestamp":"1702664274758","SenderId":"AIDAJTCD6O457Q7BMTLYM","ApproximateFirstReceiveTimestamp":"1702664274764"},"messageAttributes":{},"md5OfBody":"3d635e69eb93fd184b47a31d460ca2b6","eventSource":"aws:sqs","eventSourceARN":"arn:aws:sqs:us-west-2:536766270177:demo-dev-List-NDBP8IFB1M05-Waitlist-StxATDNj3X78","awsRegion":"us-west-2"}]}
2023-12-15 18:17:58 UTC END RequestId: be392533-83a5-5aae-b263-78c5192df684
2023-12-15 18:17:58 UTC REPORT RequestId: be392533-83a5-5aae-b263-78c5192df684  Duration: 4.91 ms  Billed Duration: 5 ms    Memory Size: 1536 MB    Max Memory Used: 168 MB Init Duration: 3035.12 ms

Also note: I tested both eu-central-1 and us-west-2 to eliminate AWS_REGION as an issue.

[djolereject]

I have to say that I'm novice with CloudFormation so maybe I'm missing something, but here is the screenshot of all Stacks that are related to this project:

I'll look into the documentation you provided so maybe I'll get some better understanding of CloudFormation which will allow me to show results in more details and ask better question.

[tongueroo]

All good. Can you click on the mailroom-dev-JetsController-* stack and see what that error is when you get a chance? It usually has details that explains why it rolled back. Thanks!

[djolereject]

Here it is:

[tongueroo]

Interesting. In this case, there's no useful info. Bummer. Can you create project that reproduces the issue and put it on github when you get a chance?

Also, can you try deploying the demo project https://github.com/tongueroo/demo-jets-sqs-queue Wondering what’s different.

Also, just released Jets 5.0.3 to fix #690 Unsure if it's related. Maybe give that a shot too.

cd project
bundle update
bundle info jets # confirm jets 5.0.3
bundle exec jets deploy

[tongueroo]

Took another look, was looking too fast last time. In the first error logs. This stands out.

05:15:28AM CREATE_FAILED AWS::CloudFormation::Stack List Embedded stack arn:aws:cloudformation:eu-central-1:... was not successfully created: The following resource(s) failed to create: [Waitlist]. 
...
05:15:39AM UPDATE_ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack mailroom-dev The following resource(s) failed to create: [PostmanJob, List]. The following resource(s) failed to update: [JetsController]. 

The JetsController stack is only rolling back because of the Waitlist stack error. So believe the root error is in the Waitlist stack.

When you get a chance, can you click on the Waitlist stack and see what that error says. It should provide more details.

Thanks!

[djolereject]

1. I just looked at the CloudFormation and there is no Waitlist stack on its own. You can see the first image in this thread, those are my only stacks. I tried to find the first red line and it might be this one:

2. Checked Jets 5.0.3 and received the same issue.

3. I tried your demo and got the same situation:

05:15:08AM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack demo-dev User Initiated
05:15:11AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway 
05:15:11AM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer 
05:15:11AM CREATE_IN_PROGRESS AWS::IAM::Role IamRole 
05:15:12AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway Resource creation Initiated
05:15:13AM CREATE_IN_PROGRESS AWS::IAM::Role IamRole Resource creation Initiated
05:15:20AM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer Resource creation Initiated
05:15:21AM CREATE_COMPLETE AWS::Lambda::LayerVersion GemLayer 
05:15:23AM CREATE_COMPLETE AWS::CloudFormation::Stack ApiGateway 
05:15:24AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiResources1 
05:15:25AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiResources1 Resource creation Initiated
05:15:29AM CREATE_COMPLETE AWS::IAM::Role IamRole 
05:15:30AM CREATE_IN_PROGRESS AWS::IAM::Policy IamPolicy 
05:15:30AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack List 
05:15:30AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsController 
05:15:30AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack PostmanJob 
05:15:31AM CREATE_IN_PROGRESS AWS::IAM::Policy IamPolicy Resource creation Initiated
05:15:31AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsController Resource creation Initiated
05:15:31AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack PostmanJob Resource creation Initiated
05:15:31AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack List Resource creation Initiated
05:15:35AM CREATE_COMPLETE AWS::CloudFormation::Stack ApiResources1 
05:15:42AM CREATE_FAILED AWS::CloudFormation::Stack List Embedded stack arn:aws:cloudformation:eu-central-1:228342851134:stack/demo-dev-List-2AXDJ1D221D1/eac76b40-9c92-11ee-94a4-02719741ea87 was not successfully created: The following resource(s) failed to create: [Waitlist]. 
05:15:42AM CREATE_FAILED AWS::CloudFormation::Stack JetsController Resource creation cancelled
05:15:42AM CREATE_FAILED AWS::CloudFormation::Stack PostmanJob Resource creation cancelled
05:15:42AM CREATE_FAILED AWS::IAM::Policy IamPolicy Resource creation cancelled
05:15:43AM UPDATE_ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack demo-dev The following resource(s) failed to create: [PostmanJob, JetsController, IamPolicy, List]. 
05:15:45AM UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack demo-dev 
05:15:46AM DELETE_IN_PROGRESS AWS::IAM::Policy IamPolicy 
05:15:46AM DELETE_IN_PROGRESS AWS::CloudFormation::Stack ApiResources1 
05:15:46AM DELETE_IN_PROGRESS AWS::CloudFormation::Stack List 
05:15:46AM DELETE_IN_PROGRESS AWS::CloudFormation::Stack PostmanJob 
05:15:46AM DELETE_IN_PROGRESS AWS::CloudFormation::Stack JetsController 
05:15:46AM DELETE_COMPLETE AWS::IAM::Policy IamPolicy 
05:15:56AM DELETE_COMPLETE AWS::CloudFormation::Stack ApiResources1 
05:16:07AM DELETE_COMPLETE AWS::CloudFormation::Stack PostmanJob 
d05:18:00AM DELETE_COMPLETE AWS::CloudFormation::Stack List 
05:18:24AM DELETE_COMPLETE AWS::CloudFormation::Stack JetsController 
05:18:24AM DELETE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer 
05:18:24AM DELETE_IN_PROGRESS AWS::IAM::Role IamRole 
05:18:24AM DELETE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway 
05:18:25AM DELETE_COMPLETE AWS::Lambda::LayerVersion GemLayer 
05:18:35AM DELETE_COMPLETE AWS::CloudFormation::Stack ApiGateway 
05:18:37AM DELETE_COMPLETE AWS::IAM::Role IamRole 
05:18:37AM UPDATE_ROLLBACK_COMPLETE AWS::CloudFormation::Stack demo-dev 
Stack rolled back: UPDATE_ROLLBACK_COMPLETE

demo problem seems to me to be the same, here is the only log from demo:

[tongueroo]

Opps should had mentioned. So the WaitList stack will "disappear". This is because when a nested stack fails to create for the first time, it'll rollback and delete itself. So you won't see it if you refresh too late.

CloudFormation behaves this way so you don't have orphan resources and tries to keep things tidy. It makes debugging trickier.

There are a few ways to get to the failed deleted stack:

1. Keep refreshing while it's deploying so you can see it being created. Once it's created, click on it so and leave it open in your browser.
2. You can grab the failed stack ARN. I see it above. It's arn:aws:cloudformation:eu-central-1:228342851134:stack/demo-dev-List-2AXDJ1D221D1/eac76b40-9c92-11ee-94a4-02719741ea87. You can then go to the "deleted stack" url directly. Build it up to this: https://eu-central-1.console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/events?filteringText=&filteringStatus=active&viewNested=true&stackId=arn:aws:cloudformation:eu-central-1:228342851134:stack/demo-dev-List-2AXDJ1D221D1/eac76b40-9c92-11ee-94a4-02719741ea87 You have to replace the stackId query string parameter with the failed deleted stack arn value. Don't worry about url escaping the value, modern browsers do this automatically.
3. Create a simpler class with something that'll likely work. Deploy that first so the nested stack is successfully created. Then, add the complete code in. Deploy again. It'll fail, but the nested stack won't "disappear" since it already existed.

I'm wondering:

• If it's an account-level-specific issue. Completely spitballing here. Wondering if there's a SCP, Service control policy, at the AWS organization level or something like that that prevents the resource from being I created I. Seeing the error in that WaitList stack's event log would be useful.
• Or the IAM deploy user being used does not have IAM permission. Docs: https://docs.rubyonjets.com/docs/extras/minimal-deploy-iam/ There's a difference between the IAM deploy user vs IAM permissions that the app takes on: https://docs.rubyonjets.com/docs/extras/minimal-deploy-iam/#lambda-function-vs-user-deploy-iam-policies

[tongueroo]

Made some improvements in #696 This should help for future users.