Comments (14)
@hdj630 Same issue happens when running Cloudwatch agent on EKS Fargate. The metadata service isn't available so the agent decides that it is running On-prem and starts to look for credentials in .aws/credentials
even though it has an IAM role attached via a serviceAccount. Running the same pod on EC2 with the same serviceAccount works fine.
from amazon-cloudwatch-agent.
I have pushed an image to thisisqasim/cloudwatch-agent
that allows an env FORCE_MODE_EC2=True
which forces it to use ec2 mode and not look for credentials at .aws/credetials
from amazon-cloudwatch-agent.
@ThisIsQasim That's great! Your behaviour should be the default.
But really, the config system of amazon-cloudwatch-agent is just bizarre. Why must it act differently than every other AWS client app? It's trying to be "smart" and just being inflexible. It should just use the auth defaults of the aws library and be done with it.
from amazon-cloudwatch-agent.
I think itβs being done to validate the config for EC2 specific features e.g. {instance_id}
. It should however let the sdk handle the credentials chain and not manually handle it.
from amazon-cloudwatch-agent.
I have changed the flag for forcing EC2 mode from FORCE_MODE_EC2=True
to RUN_IN_AWS=True
as per instructions from the maintainers on #122
from amazon-cloudwatch-agent.
Just to be clear, your patch does not change the default behaviour, correct? Fargate instances will continue to be detected as OnPrem.
While I appreciate having a work-around, the patch seems strange to me. You've added an undocumented env var to fix behaviour that should be a straight-forward command-line option or automatic. Why not look at one of the existing variables like AWS_EXECUTION_ENV
? It can be either AWS_ECS_FARGATE or AWS_ECS_EC2`
from amazon-cloudwatch-agent.
You are right in assuming this to be a workaround and not the ideal solution. Ideally, it should detect Fargate automatically without extra flags.
However, as per my limited knowledge, there is no metadata service of any kind on EKS Fargate nor are any AWS specific Env Vars injected. This makes it very difficult to automatically determine if the agent is running on EKS Fargate or OnPrem k8s. So the only solution, again as per my limited knowledge, was to add a flag that forces EC2 mode.
Fargate on ECS is different but I have never used it. Feel free to do it properly and make a PR. The maintainers sound like helpful people and I am sure they'll take up your contributions.
from amazon-cloudwatch-agent.
This issue was marked stale due to lack of activity.
from amazon-cloudwatch-agent.
I would like to revisit this issue and look at our credentials provider implementation. The default credentials provider for the aws sdk is able to handle being on Fargate vs k8s native cluster and move through different modes of credentials in order. I believe we can also take this approach with the agent.
from amazon-cloudwatch-agent.
This issue was marked stale due to lack of activity.
from amazon-cloudwatch-agent.
This is still a pending issue
from amazon-cloudwatch-agent.
This issue was marked stale due to lack of activity.
from amazon-cloudwatch-agent.
I believe this still needs attention
from amazon-cloudwatch-agent.
Hey @rs-garrick, this would be a short notice from me. However, would you able to build the image from source by using make dockerized-build
and confirm this image works on your ECS Fargate. Moreover, for @ThisIsQasim, please help me in creating a separate issue even though its still addresses the same problem but on different platform. Thanks in advance for this
from amazon-cloudwatch-agent.
Related Issues (20)
- The CloudWatch Agent requires additional configuration to use IAM Roles for Service Accounts (IRSA) or Pod Identity HOT 2
- Unable to reload agent with error message unable to address what is wrong HOT 1
- Rotate amazon-cloudwatch-agent.log HOT 2
- XRay traces sent by cloudwatch-agent are missing cloudwatch_logs data HOT 2
- cannot get pod from kubelet, err: call to /pods endpoint failed: HOT 2
- amazon-cloudwatch-observability fails with open /root/.aws/credentials ignoring the IRSA credentials HOT 12
- Log agent not running as root sometimes fails to tail log files HOT 3
- Example for append_dimensions using a value that depends on Amazon EC2 metadata HOT 1
- Container is stopped. On-premise setup. HOT 2
- X-Ray: how to set index_all_attributes (or provide a list of attributes to index)? HOT 2
- [k8s / EKS] Support for Filtering Metrics by Namespace/Pod Name in CloudWatch Agent HOT 2
- https://github.com/aws/amazon-cloudwatch-agent/releases shows v1.300034.0 but that is not current HOT 1
- CloudWatch metrics collected from prometheus , contains undesired dimensions HOT 1
- Undesired metrics added HOT 1
- [K8E mode] CloudWatch Agent shutting down when configured to collect container insights metric on EC2 K8s setup
- I wish fetch-config would not delete the .json config file HOT 3
- Publish a JSON schema of the config HOT 7
- Add KMS Key to Cloudwatch Log Group HOT 1
- Support Credentials from IAM Roles Anywhere HOT 2
- Agent fails to detect EKS cluster created in Access Entries only authentication mode HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-cloudwatch-agent.