[BUG] Use default AWS credentials chain

[caiorcferreira]

[BUG] Use default AWS credentials chain

[Jeffail]

Hey @caiorcferreira, I'm not sure I fully understand the solution, would we need to migrate to the v2 AWS API? As long as it's backwards compatible them I'm happy to accept a fix.

[caiorcferreira]

Sorry for the short description, let me expand on that:

Scenario
We use KIAM to authenticate pods with AWS. KIAM works by intercepting calls to IMDS and returning credentials for the role specified on the pod's annotation.

Currently, when trying to use benthos with the following configuration:

input:
      label: "input"
      aws_s3:
        region: us-east-1
        codec: all-bytes
        sqs:
          url: "https://sqs.us-east-1.amazonaws.com/0000000000/source-queue"
          key_path: Records.*.s3.object.key
          bucket_path: Records.*.s3.bucket.name
    output:
      label: "output"
      aws_s3:
        region: us-east-1
        bucket: "dest-bucket"
        path: ${!metadata("s3_key")}
        content_type: application/json
        content_encoding: gzip
        max_in_flight: 32

Some information was replaced for security

I'm getting the following error when reading from SQS: Failed to read message: AccessDenied: Access Denied\n\tstatus code: 403, request id: REAT4SFJ99ZDDBT8, host id: abdcde" @service=benthos label=input path=root.input

Troubleshooting

I have verified the roles permission by going into the pod and using the AWS CLI to validate it was getting the correct credentials with aws sts get-caller-identity and that the credentials had the necessary permission to read from SQS with aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/0000000000/source-queue.

Hence, the KIAM setup is working fine as with other applications we run in the same Kubernetes Cluster.

The major difference between benthos and our Go applications, as I was able to figure out, is how they initialize the SDK configuration. In fact, we use SDK V2 which have the github.com/aws/aws-sdk-go-v2/config.LoadDefaultConfig that I mentioned previously. I had not realized that benthos was still using V1 when I made the suggestion.

After all, I believe that the V1 SDK should work fine with KIAM, however I'm not sure why it's not. I tried to set from_ec2_role in order to force benthos to get credentials from IMDS, but it didn't work, I kept getting the same error message.

Also, looking back at the code from the EC2 IMDS credential provider, I see my previous statement was wrong, as the constant is a predefined path on the IMDS service, not a filesystem path.

as it seams AWS SDK looks for instance information in a predefined path that does not exist in the context of a Kubernetes container.

Conclusion

To answer your question directly, I don't the SDK V2 is in any way backwards compatible. The migration guide has a lot of topics to be covered.

As I said, now that I not tired from a day of troubleshooting I think SDK V1 should work with KIAM, but I'm without ideas about why it's still not.

Thanks for the kind and fast response, benthos is an incredible project and if it can work on our setup it will become a major part of our infrastructure.

[caiorcferreira]

After another of troubleshooting I was able to figure out what was happing. For future reference, what happened was:

Because of the error message Failed to read message: AccessDenied: Access Denied\n\tstatus code: 403, request id: REAT4SFJ99ZDDBT8, host id: abdcde" @service=benthos label=input path=root.input I though that the problem was consuming messages from SQS. Turns out I was wrong, as Failed to read message is a static message added by benthos here.

Benthos was failing because the objects in the input bucket didn't have the correct ACL and were still owned by the writer.

It would be nice to wrap error with the action being taken to improve clarity. Maybe later I open a PR with this.

Thanks