Stackdriver fails to export metrics after living long enough

[ascherkus]

Stackdriver fails to export metrics after living long enough

[james-bebbington]

Apologies for taking so long to respond to this.

I believe the first error means you are sending multiple metrics with the exact same label keys to Cloud Monitoring within a certain period of time (10s). Are you running multiple instances of your application under the same "resource" that could produce the same set of metrics?

[ascherkus]

No worries! I think you tipped me off in the right direction... I run my applications as a go binary inside of a distroless docker image inside of Google Cloud Run, so I would have thought the labels should be different based on the Cloud Run instance that spins up my applications.

However, based on the documentation for DefaultMonitoringLabels it defaults to this defaults to a single label with key "opencensus_task" and value "go-<pid>@<hostname>"

Sure enough in my metrics the resulting label is go-1@localhost since (I believe?) distroless boots my applications as PID 1. I'll see if I can key this off based on some Cloud Run metadata (or similar) and see if that helps.

[ascherkus]

Yeah setting MonitoredResource combined with DefaultMonitoringLabels fixes things so bug was on my end. Thanks for the pointer!