Metrics related to persistent store operations

[launchdarkly]

Metrics related to persistent store operations

[eli-darkly]

It's an interesting idea, but I don't think it's feasible to try to design it in a single PR in this repository. All of the persistent store functionality comes from the Go SDK, and some of the metrics you're talking about could only be reported from within the database-specific store implementation component (e.g. there's no such thing as a universal connection pool that Relay could observe in any generic way) so this would probably require changes in the SDK component APIs.

[amitsaha]

I am missing something perhaps here - but ignoring the pool metrics (which I presume is a SDK implementation detail), can we not perform the metric calculation in the relay_feature_store.go file? Consider the Upsert() method which currently looks as:

sw.loggers.Debugf(`Received feature flag update: %s (version %d)`, key, item.Version)
updated, err := sw.store.Upsert(kind, key, item)

What if we surrounded the call to Upsert() with a timer for latency and a counter for non-nil errors?

[eli-darkly]

I am missing something perhaps here - but ignoring the pool metrics (which I presume is a SDK implementation detail)

That's exactly the thing I just mentioned above:

some of the metrics you're talking about could only be reported from within the database-specific store implementation component (e.g. there's no such thing as a universal connection pool that Relay could observe in any generic way)

So, if you want connection pool information, then this cannot be done only with Relay changes. If you don't want connection pool information after all, then that's another matter, and Relay changes might be sufficient.

[amitsaha]

So, if you want connection pool information, then this cannot be done only with Relay changes. If you don't want connection pool information after all, then that's another matter, and Relay changes might be sufficient.

Sounds 👍 , I will take a stab at the persistent store operations metrics first and follow up separately related to the pool metrics in the Go SDK project.

[eli-darkly]

You will not be able to add connection pool metrics in the Go SDK project; I think you may have underestimated the scope of such a change. Connection pooling is part of the individual database integrations— as I said, there is no universal concept of a connection pool that the SDK can deal with overall. That is part of the third-party Redis/Consul/DynamoDB clients that we use. We would have to 1. determine what (if any) pool-related metrics are actually available to us from those clients, 2. come up with a standardized way to represent them in the SDK, 3. design a new component interface in the SDK (since the existing PersistentDataStore interface has no method for this), and 4. update each of the integration libraries to implement that interface.

We're happy to note it as a feature request, but I think this one will be more feasible for us to implement in-house. At the very least, I strongly recommend that you look in detail at how persistent data stores are implemented in the SDK before pursuing this.

[eli-darkly]

I'm also not clear on what kind of metrics you're looking for in regard to connection pools. If you want to know how many connections are being made to your database instance, I would think it'd be easier to monitor that on the server side.

[amitsaha]

I'm also not clear on what kind of metrics you're looking for in regard to connection pools. If you want to know how many connections are being made to your database instance, I would think it'd be easier to monitor that on the server side.

That's a good point. I guess we can do that too and from what you mention in the previous comment, might be an easier starting point.