Comments (9)
If you don’t need to use an external Id with your assume role arn, you can install the aws cli on the server use the aws credentials to setup the aws default profile for the primary account. In your logstash config do not put an access key and secret key leave them out, just put the role arn in the config and a sessionname. The plugin will then use the default profile that you setup with cli to present to aws to get the assumed role in return.
The code for this is actually in logstash-mixin-aws not the this plugin, this plugin has a dependency on it. If you need to use external Id I have open pull request for an enhanced logstash-mixin-plugin that enhances assume role function to use an external Id as well.
from logstash-input-s3.
Thanks alot brother.... It worked!!!
from logstash-input-s3.
It would appear from the source code that the assumed role may only work if logstash is running on an AWS ec2 and your using the identity assigned to the instance and also not populating the access key and secret options, only providing a assumed role and session name.
The code requires changes to use a different identity for an assumed role and also would then work on a non AWS hosted server.
from logstash-input-s3.
Hi @cabberley , thanks for your comment.
When you are working into the EC2 instance, you don't need to assume the role of the instance, this is automatically implemented into the AWS SDK.
The common behaviour is to use assume-role when you are operating cross-account, where you use the actual credentials to call sts API and create temporary credentials into the second account.
We can see it into the docs Creating an AWS STS Access Token
Of course, could be a case where you are operating into the EC2 instance and needs to operate cross-account also.
For me, if you pass the AWS access_key_id
, secret_access_key
and role_arn
is because you are going to use the two firsts to call AWS STS API
to assume the role role_arn
and generate new credentials like show the Creating an AWS STS Access Token
from logstash-input-s3.
Hi @christiangda I may not have explained very well. Your comments are correct.
What I am trying to say is that the way the s3 plugin code has been written, if you you supply access_key_id, secret_access_key in the .conf file the code will never do the assumeRole with the role_arn you provide. It will only use role_arn and execute the Assumerole if the .conf file only has role_arn.
The code which is the problem for us is actually part of logstash-mixin-aws not this plugin
The logic in the code says
IF access_key_id and secret_access key is provided then use them to authenticate
ELSIF credentials are in a YML file read only access_key_id and secret_access key and authenticate
ELSIF role_ARN is provide then do assumerole and use the ec2 identity as the Identity authorized to use the ARN_ROLE.
END
Which means if you provide as you want all 3 values, it will never do the assume role.
I made my own version of the plugin which changed the logic to cater for your scenario
IF access_key_id and secret_access key is provided then use them to authenticate
ELSIF credentials are in a YML file read only access_key_id and secret_access key and authenticate
END
IF role_ARN is provide then do assumerole using the access_key_id and secret_access key provided in the .conf file.
END
Mine also caters for using external_id which is also a parameter that AssumeRole sometimes requires depending on how the identity has been setup. addiing external_id does require a few other code changes to Logstash-mixin-aws for it to work. But Doesn't require changes to the plugins that rely on logstash-mixin-aws. I use the s3 input plugin and the cloudwatch plugin which rely on this code.
from logstash-input-s3.
I am also facing the same issue. Can you please let me know how to solve this problem.
I have installed logstash in on-prem server.
from logstash-input-s3.
@cabberley, can you help me with this error. Just wanted to know when does this occur. Is it because of configuration issue or permissions at s3 bucket. And, I was able to list the objects through awscli.
from logstash-input-s3.
We are still facing this issue? Any updates on this ?
from logstash-input-s3.
One of the ways I tried fixing this issue was to export Access Keys as environment variables and then start logstash
from logstash-input-s3.
Related Issues (20)
- logstash s3 input enable use_accelerate_endpoint failed
- backup_to_dir backups rotation
- seahorse::client:: networking error HOT 2
- S3 input plugin is not reading AWS-KMS (CMK) encrypted bucket HOT 2
- Large files are very slow to read locally HOT 2
- Files being unprocessed with the same last modified timestamp HOT 6
- Could not fetch objects from "requester pays" enabled bucket HOT 1
- logstash-input-s3 3.6.0 restarting due to unrecoverable error with CipherSuites array issue HOT 1
- s3 input plugin not handling shutdown correctly, leading to duplicates once started again HOT 2
- Files being unprocessed because of cutoff time calculation part
- logstash-input-s3 Error: Net::OpenTimeout 没人解决就关闭问题吗?
- [Logstash 7.16.2] S3 input plugin replaces the region in endpnt url HOT 7
- [Docs] Document workaround when using s3 private link endpoints with us-east-1
- sincedb file not created, files from bucket not deleted HOT 2
- S3 input plugin does not work with IAM role WebIdentity HOT 3
- Plugin dont process objects correctly, dont delete or backup HOT 11
- Files with same last modified timestamp miss processing HOT 1
- Constant and frequent s3 plugin restart due to TCP connection failure
- java.util.concurrent.ScheduledThreadPoolExecutor related to S3 input resulting in all pipeline logstash failure. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logstash-input-s3.