[Bug] full diagnostics cannot access (No change from "Running Diagnostics...")

[githubeto]

[Bug] full diagnostics cannot access (No change from "Running Diagnostics...")

[dwbrown2]

@jessegoodier @AjayTripathy or others will likely be able to provide more detailed troubleshooting recommendations, but it looks like your prometheus isn't reachable. What's status of that pod?

[githubeto]

@jessegoodier @AjayTripathy or others will likely be able to provide more detailed troubleshooting recommendations, but it looks like your prometheus isn't reachable. What's status of that pod?

Prometheus pod is running.

[jessegoodier]

Do you have network policies that prevent communication between pods?
Also, is anything else running in this cluster that has a networking issue?
@githubeto

[githubeto]

Do you have network policies that prevent communication between pods? Also, is anything else running in this cluster that has a networking issue? @githubeto

@jessegoodier
Cluster has Istio installed, but neither AuthorizationPolicy nor NetworkPolicy is applied.
There are also no other resources controlling inter-Pod communication.
Are there no detailed logs when the connection fails? Is it an issue with the debug level? I believe it should appear in the logs.

[jessegoodier]

Do you have network policies that prevent communication between pods? Also, is anything else running in this cluster that has a networking issue? @githubeto

@jessegoodier Cluster has Istio installed, but neither AuthorizationPolicy nor NetworkPolicy is applied. There are also no other resources controlling inter-Pod communication. Are there no detailed logs when the connection fails? Is it an issue with the debug level? I believe it should appear in the logs.

You can try a curl from the frontend:

kubectl exec -i -t -n kubecost deployments/kubecost-cost-analyzer -c cost-analyzer-frontend -- curl http://kubecost-prometheus-server.kubecost

should get:
<a href="/graph">Found</a>

you can also try a curl to other pods, perhaps grafana?

curl http://kubecost-grafana.kubecost

[jessegoodier]

Because Kubecost does not block traffic, I would not expect any logs, other than the communication failures you are seeing.

Do you have another cluster to test on to rule out other issues?

[githubeto]

Do you have network policies that prevent communication between pods? Also, is anything else running in this cluster that has a networking issue? @githubeto

@jessegoodier Cluster has Istio installed, but neither AuthorizationPolicy nor NetworkPolicy is applied. There are also no other resources controlling inter-Pod communication. Are there no detailed logs when the connection fails? Is it an issue with the debug level? I believe it should appear in the logs.

You can try a curl from the frontend:

kubectl exec -i -t -n kubecost deployments/kubecost-cost-analyzer -c cost-analyzer-frontend -- curl http://kubecost-prometheus-server.kubecost

should get: <a href="/graph">Found</a>

you can also try a curl to other pods, perhaps grafana?

curl http://kubecost-grafana.kubecost

@jessegoodier
The curl to kubecost-prometheus-server returned the correct response "Found".
As you can see from the Helm Chart, Grafana is not running, so it has not been checked.

There are no clusters without Istio, making it difficult to verify.

[jessegoodier]

We do not have other reports of this.

I don't have any other ideas here. Very strange the test command works but the cost-model container cannot communicate.

[githubeto]

@jessegoodier @AjayTripathy

While closely monitoring the browser access logs, I found an interesting log.
Does this error log correspond to the reason why the Full Diagnostics screen cannot be displayed?

This error log seems to be a 403 error (Rate limit) when accessing
https://api.github.com/repositories/178079595/releases or
https://api.github.com/repos/kubecost/cost-model/releases.

178079595/releases response:

{
    "message": "API rate limit exceeded for xx.xx.xx.xx. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
    "documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"
}