Comments (9)
@jessegoodier @AjayTripathy or others will likely be able to provide more detailed troubleshooting recommendations, but it looks like your prometheus isn't reachable. What's status of that pod?
from features-bugs.
@jessegoodier @AjayTripathy or others will likely be able to provide more detailed troubleshooting recommendations, but it looks like your prometheus isn't reachable. What's status of that pod?
Prometheus pod is running.
from features-bugs.
Do you have network policies that prevent communication between pods?
Also, is anything else running in this cluster that has a networking issue?
@githubeto
from features-bugs.
Do you have network policies that prevent communication between pods? Also, is anything else running in this cluster that has a networking issue? @githubeto
@jessegoodier
Cluster has Istio installed, but neither AuthorizationPolicy nor NetworkPolicy is applied.
There are also no other resources controlling inter-Pod communication.
Are there no detailed logs when the connection fails? Is it an issue with the debug level? I believe it should appear in the logs.
from features-bugs.
Do you have network policies that prevent communication between pods? Also, is anything else running in this cluster that has a networking issue? @githubeto
@jessegoodier Cluster has Istio installed, but neither AuthorizationPolicy nor NetworkPolicy is applied. There are also no other resources controlling inter-Pod communication. Are there no detailed logs when the connection fails? Is it an issue with the debug level? I believe it should appear in the logs.
You can try a curl from the frontend:
kubectl exec -i -t -n kubecost deployments/kubecost-cost-analyzer -c cost-analyzer-frontend -- curl http://kubecost-prometheus-server.kubecost
should get:
<a href="/graph">Found</a>
you can also try a curl to other pods, perhaps grafana?
curl http://kubecost-grafana.kubecost
from features-bugs.
Because Kubecost does not block traffic, I would not expect any logs, other than the communication failures you are seeing.
Do you have another cluster to test on to rule out other issues?
from features-bugs.
Do you have network policies that prevent communication between pods? Also, is anything else running in this cluster that has a networking issue? @githubeto
@jessegoodier Cluster has Istio installed, but neither AuthorizationPolicy nor NetworkPolicy is applied. There are also no other resources controlling inter-Pod communication. Are there no detailed logs when the connection fails? Is it an issue with the debug level? I believe it should appear in the logs.
You can try a curl from the frontend:
kubectl exec -i -t -n kubecost deployments/kubecost-cost-analyzer -c cost-analyzer-frontend -- curl http://kubecost-prometheus-server.kubecost
should get:
<a href="/graph">Found</a>
you can also try a curl to other pods, perhaps grafana?
curl http://kubecost-grafana.kubecost
@jessegoodier
The curl to kubecost-prometheus-server returned the correct response "Found".
As you can see from the Helm Chart, Grafana is not running, so it has not been checked.
There are no clusters without Istio, making it difficult to verify.
from features-bugs.
We do not have other reports of this.
I don't have any other ideas here. Very strange the test command works but the cost-model container cannot communicate.
from features-bugs.
While closely monitoring the browser access logs, I found an interesting log.
Does this error log correspond to the reason why the Full Diagnostics screen cannot be displayed?
This error log seems to be a 403 error (Rate limit) when accessing
https://api.github.com/repositories/178079595/releases or
https://api.github.com/repos/kubecost/cost-model/releases.
178079595/releases response:
{
"message": "API rate limit exceeded for xx.xx.xx.xx. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
"documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"
}
from features-bugs.
Related Issues (20)
- [Feature] Support `LOG_FORMAT` and `LOG_LEVEL` in cluster controller and network costs
- [Bug] network monitoring is empty HOT 5
- [Bug] cloud integration - spot datafeed - no datas HOT 2
- [Bug] Turndown schedule deleted a few seconds after being deployed HOT 2
- [Feature] Support Cilium - add parsing of conntrack information HOT 5
- [Feature] - Add ability to aggregate allocations by Provider ID HOT 1
- [Bug] Duckdb corruption after 2.2.5 upgrade HOT 7
- Request Right Sizing action created via Helm shows incorrect in UI HOT 13
- [Bug] cost-model image not available on aws ecr for release 2.3.2 HOT 1
- [Bug] Aggregator panics on QueryAssetCTE HOT 7
- [Bug] Cluster Controller stopped targeting namespace level annotations HOT 1
- [Bug] Alibaba Instance Price Error HOT 1
- [Bug] Allocation query fails due to a max concurrency semaphore error HOT 1
- You found an enterprise feature! HOT 1
- [Feature] Provide a configuration to restrict out of cluster cost dashboard HOT 1
- [Bug] Operation error Organizations: ListAccounts HOT 3
- [Bug] No data HOT 1
- [Feature] Support Kubernetes 1.30 in AWS Marketplace listing HOT 2
- [Bug] Crash Aggregator container HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from features-bugs.