Investigate running vendor-specific tests

[eclipse]

Investigate running vendor-specific tests

[pilhuhn]

On 22 Aug 2017, at 14:30, Mike Croft wrote: > Yes, that is what OPTIONS /metrics/<scope> is for. Does the mention of sub-trees in the description cover things like `/metrics/vendor/group1/fooVal`? E.g. we make heavy use of Hazelcast in Payara, so we may want to expose some Hazelcast metrics under something like `/metrics/payara/hazelcast/${cacheName}` or something.

This is not yet really fleshed out. So far the assumption is that stuff is flat below the scope (app, vendor, base), but we talked about more hierarchy in the past.

Is there a requirement for all metrics to be "flat" (meaning no hierarchy below the 3rd tier) or would something like the above be permitted? I can't see anywhere in the spec where it defines the metadata for an endpoint which exposes no data directly.

While it is flat at the moment and I could see the above as `/metrics/payara/hazelcast-${cacheName}`, I understand the need for (some) hierarchy. I have been thinking for the integrations scope (post 1.0), that scopes get some sort of 'namespace' like e.g. integration:servlet:aServletMetric or integration:ft:numberBreakerTrigered This is not exactly the same of course (or better just moves the need for hierarchy down one level). Right now I would like to postpone this to post 1.0 if possible.

[mikecroft]

I've been thinking about this - my first reaction was just "no" because I think those sorts of tests are the responsibility of vendors and it would mean we would need to keep the TCK updated each time someone wants to implement the spec.

Could Metrics report metadata about itself? like available metrics and their types? We could then automate basic testing of vendor endpoints, i.e. testing for something rather than nothing or that the result is an Integer when we expect it to be. Not sure if we could maybe hook into the OpenAPI proposal for that (although I haven't seen any work on it since the repo was created)

[pilhuhn]

On 22 Aug 2017, at 13:03, Mike Croft wrote: I've been thinking about this - my first reaction was just Could Metrics report metadata about itself? like available

Yes, that is what OPTIONS /metrics/<scope> is for.

metrics and their types? We could then automate *basic* testing of vendor endpoints, i.e. testing for something rather than nothing or that the result is an Integer when we expect it to be. Not sure if we could maybe hook into

But yes, that could at least allow to test some data format.

the OpenAPI proposal for that (although I haven't seen any work on it since the repo was created)

I was thinking about OpenAPI in the past, but dropped it for now. The other option could be that the tests can point to an external configuration file, that lists "meta meta data", which can test the exported data + metadata. I am sort of following this in my PR for the TCK 5e7595b

…

-- [email protected] m:0179/207 4919 b:http://pilhuhn.blogspot.com

[mikecroft]

Yes, that is what OPTIONS /metrics/ is for.

Does the mention of sub-trees in the description cover things like /metrics/vendor/group1/fooVal? E.g. we make heavy use of Hazelcast in Payara, so we may want to expose some Hazelcast metrics under something like /metrics/payara/hazelcast/${cacheName} or something.

Is there a requirement for all metrics to be "flat" (meaning no hierarchy below the 3rd tier) or would something like the above be permitted? I can't see anywhere in the spec where it defines the metadata for an endpoint which exposes no data directly.