Help Wanted - Troubleshooting Tips, Tricks, and Strategies

[awsdocs]

Help Wanted - Troubleshooting Tips, Tricks, and Strategies

[josh-demuth]

this may be more of a documentation update request vs. an issue with the process itself.

i'm using node within a custom rotation lambda and ran into issues where only the 'createSecret' step was being triggered. it seemed to hang due to how i was creating the AWSPENDING label on my secret. if i created any other label beside AWSPENDING all four steps would be called (but w/o the functionality i needed) so i figured there was something wrong with my create function.

however, my create fucntion was very straightforward - create a random key and create a new stage called AWSPENDING with the new key.

i was using the documentation from the javascript SDK to do this (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SecretsManager.html#putSecretValue-property).

i used the following recommendation about usage of ClientRequestToken...

ClientRequestToken — (String)
(Optional) Specifies a unique identifier for the new version of the secret.

Note: If you use the AWS CLI or one of the AWS SDK to call this operation, then you can leave this parameter empty. The CLI or SDK generates a random UUID for you and includes that in the request...

every call i was making i left this parameter blank and it would technically work and create a new index with the AWSPENDING label (as confirmed using the CLI) but it would never initiate the next calls to the rotate function.

i randomly decided to include this ClientRequestToken parameter (using info from the 'event' object)...and...it finally worked!

so...i think another note in the SDK docs or somewhere in the Secrets Manager documentation it should state that if creating a custom lambda rotate function and trying to facilitate the AWSPENDING initial create you MUST pass the ClientRequestToken value that is passed when invoking the lambda or it will cause the rotate to stop/hang.

overall, this offering seems really great and i can't wait to move on with our development...thanks for asking for this type of feedback!

[bisdavid]

@josh-demuth - The docs are correct in that for an SDK call, the parameter is technically optional. However, if you use the Secrets Manager calls from inside a Lambda rotation function, then that scenario does indeed require that you pass the ClientRequestToken that is sent to the Lambda as part of the Lambda Event parameter to each Secrets Manager API that accepts the parameter. That ensures that each rotation step works on the same version of the secret and doesn't accidentally create a new version with each step. I'll look for opportunities to clarify this requirement in the docs.

I hope that helps clarify the issue! Thanks for reporting this!

Dave

[bisdavid]

@josh-demuth - I'll pass this feedback on to the API Gateway team. Thanks for sharing.

[billymac98]

Hi Dave,

I recently went through the tutorial and ran into an issue with the default settings not quite working right.

• What were the symptoms of the problem?

In https://docs.aws.amazon.com/secretsmanager/latest/userguide/tutorials_db-rotate-master.html
the rotation failed with the following error in the cloud watch log:
"Unable to clone user, username length with _clone appended would exceed 16 characters"

• Were the exception/error messages helpful or less than helpful? Did they help lead you to the solution, confuse you, or were they just white noise that didn't help or hurt? What specific message(s) did you get and how did they help or not help?

The message was clear and helpful.

• What did you do to troubleshoot? What did you try that didn't help? What finally did lead you to the correct root cause?

Reviewed the lambda code to change the suffix.

• What did you finally do to correct the problem?

I edited the Lambda for rotation to use the clone_suffix of "_2" instead of "_clone" near the bottom of the lambda script created for the rotation.

So not a major issue, but the tutorial doesn't work completely as written in my testing and this seemed to be the best place to provide the feedback.

Thanks,
Billy

[hwatts]

Not sure if it warrants inclusion into the troubleshooting section - it may have been a unique problem to me and was ultimately my error, but I had to raise a support ticket and it took a few days for the support team to get to the bottom of the issues, so:

What were the symptoms of the problem?
Manual rotation of the secret worked correctly, but scheduled rotation stopped after the first invocation (the lambda functions were never called again by Secrets Manager). There was no log output or errors in Cloudwatch logs for the lambda function, or in Secrets Manager.

What did you do to troubleshoot? What did you try that didn't help? What finally did lead you to the correct root cause?
Added debug lines to the code, double checked that the 4 handlers all worked correctly (which they did) and that it followed the pattern described here: https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-one-user-one-password.html
Had to raise a support ticket to get to the root cause.

What did you do to correct the problem?
The problem was caused by the lambda functions calling the Secrets Manager API asynchronously and not blocking until it received a result in the handler. Although the async calls did complete and the secrets got updated correctly, Secrets Manager checks its internal state immediately after calling the function and if there's a mismatch, silently disables rotation. In this case, the secret was updated a few milliseconds after the state was checked.

It was caseId: 5045958181 if you have access and want to check any of the details...

[bisdavid]

@hwatts Thanks for replying! I'll take this and investigate and consider adding it to our troubleshooting topics. I appreciate the contribution!

[stevenpovlitz]

Is adding Secrets Manager to Amplify in your backlog? If not, it would be helpful in the Amplify documentation to explicitly mention that Secrets Manager is not included, and point people to the JS-SDK.

[bisdavid]

@stevenpovlitz - I passed your request on to the team that owns Amplify. Thanks for the feature suggestion!

[josh-demuth]

@bisdavid - I wanted to bring up something else we've come across that would be helpful to either get your feedback and/or let the community know about.

Our use case is to use Secrets Manager to rotate API Gateway API keys that are attached to Usage Plans.

We've run into an issue where there is latency during the Set step when we create the new API key and attach it a usage plan (using createApiKey and createUsagePlanKey in Node). We get a completed promise and everything looks good in Set but when the Test step kicks off we get 403 errors occasionally. The Test step starts almost immediately after our Set finishes and it seems to take between 3-10 seconds for the new API key to fully take hold.

Our workaround for this is to sleep about 8 seconds before starting the tests and then loop every second until we get a 200 status. We need to limit erroneous 400 errors as much as possible since it will trigger our CloudWatch alarms.

Ideally we wouldn't get a complete status from the createUsagePlanKey step until it was ready to use.

This definitely is not a Secrets Manager issue but I thought you may like to know about this since it seems like a pretty standard use case for leveraging Secrets Manager. I'd be happy to provide more details if needed.

Edit: also wanted to say that I saw the comment here but still wanted to point this out. Overall, our workaround is functional and does not add much overhead...

[dianeo]

@josh-demuth - I'm the doc writer for API Gateway. I passed your comment on to the API Gateway team and was told that there's always a brief delay between when a new API keys is created and when it becomes fully active/usable; and there's no workaround for this.

Can you tell us why you're rotating API keys? I'll work with the engineering team to understand your use case and update the docs with some best practices. Thanks!

[josh-demuth]

hi @dianeo - apologies for the delay. due to requirements from our security team we are rotating our api keys on a scheduled basis. as to 'why' we are doing this it is to ensure that if our keys get compromised they won't be for long. secrets manager is a perfect service for us since it handles the 4 steps of the rotation process and allows us to focus on actually rotating the keys instead of orchestrating the rotation process.

the way we are getting around the propagation delay is to sleep during our 'test' step. we just have a simple loop that tries to call our api with the new key. it will loop for X seconds and keep trying until successful. we've seen that sleeping for < 10 seconds works 98% of the time but sometimes it would have been better to sleep about 20 seconds.

since we are using our main API URL it is pretty much impossible as-is to tell that the 403 errors we are receiving when it loops and fails is from the rotation process vs. from an actual request. this has triggered our CloudWatch alarms a few times. Other than that, the process seems to be working great.

[bisdavid]

Thanks for the feedback, Billy! I'm actually no longer the writer for Secrets Manager. I forwarded your comments to the current writer though, so Mark should be looking at this and coming up with a fix. Thanks for reporting the issue!

[ispringer]

i randomly decided to include this ClientRequestToken parameter (using info from the 'event' object)...and...it finally worked!

so...i think another note in the SDK docs or somewhere in the Secrets Manager documentation it should state that if creating a custom lambda rotate function and trying to facilitate the AWSPENDING initial create you MUST pass the ClientRequestToken value that is passed when invoking the lambda or it will cause the rotate to stop/hang.

@josh-demuth, thanks so much for reporting your experience and the solution to the issue! I encountered the exact same issue and was pulling my hair out trying to figure out why it was hanging after createSecret. I agree the docs and Javadoc (still) could use some improvement in documenting the clientRequestToken requirement when putSecret is used to update the pending secret from a rotation Lambda.